Bindable - General purpose for-comprehension at your service

I didn’t get what are the features and why can’t I just use Stream for every lazily evaluated enumerable?

All of your examples can be rewritten using Stream (or Enumerable in case of two streams’ product)

And this tuple syntax is unique across all Elixir’s macros and will be confusing for elixir developers. I suggest nested structure to imitate variadic function like

quote do
  bindable for x <- xs, y <- ys, z <- zs do
    x + y + z
  end
end

Here you just need to implement the bindable/2 macro and parse variadic for call as an AST

But my last suggestion would be to implement Stream.product function in the PR to the elixir-lang instead of creating custom macro with unfamiliar (to elixir devs) syntax.

This would solve the problem with for-style lazy evaluation as a product of two enumerables and it would be compatible with both elixir language design and Stream API

Anyway, great work!

Thanks for the feedback!

That is not about Streams at all. As I mentioned, Stream composition comes as a bonus.

Anyway, let’s talk Streams: Elixir’s for is a great facility to create a new lists (well, up to :reduce feature, using which you can obtain any value). And one of the great option of it is “contextual” filtration. And the thing is:

for a <- stream_of_as,
    b <- stream_of_bs,
    a < b,             # accessing values from another stream
    c <- stream_of_cs,
    a + b + c < 10 do
  a + b + c
end

can not be reproduced in general with Stream.product, as it works on independent streams:

Stream.product([
  Stream.filter(1..5, fn a -> a < 3 end),
  Stream.filter(5..1, fn b -> b > 3 end),    # only accessing values from the current stream
  Stream.take(1..5, 3)
])
|> Stream.filter(fn {a, b, c} -> a + b + c < 10 end)
|> Stream.map(fn {a, b, c} -> a + b + c end)

or if you have only two streams combinator:

Stream.product(
  Stream.filter(1..5, fn a -> a < 3 end),
  Stream.product(
    Stream.filter(5..1, fn b -> a > 3 end),
    Stream.take(1..5, 3)
  )
)
|> Stream.filter(fn {a, {b, c}} -> a + b + c < 10 end)
|> Stream.map(fn {a, {b, c}} -> a + b + c end) # looks scary :)

So I don’t think Stream.product would close the gap as a Stream combinator (at the end of the day, for is a way more readable option to construct new values).

The main point: for facility not only for lists.

If you have a data type, that “behaves flat-mappable”, you can use it inside for.

Clever trick, I like it!

(but we still need to use parentheses to explicitly bound the only bindable argument)

What about

Stream.product(as, bs)
|> Stream.filter(fn {a, b} -> a < b end)
|> Stream.product(c)
|> Stream.fitler(fn {{a, b}, c} -> a + b + c < 10 end)
|> Stream.map(fn {{a, b}, c} -> a + b + c end)

?

That’s mostly because you’re comping from Scala. Elixir developers use Streams for lazy collections all the time.

for is not for lists only. It accepts any enumerable, but produces any Collectable you’re specifiying in option into. While your for solution always produces a composition of something your custom protocols implement. I undestand that this is scala-like approach, and it looks really interesting to play with, and I definitely have nothing against this approach, but I am just noticing that

1. Elixir is not lazily evaluated
2. For lazy collections developers usually use Stream in Elixir
3. Elixir developers do not use lazy collections for resource management. The most common approach here is to implement monitoring process for resource management.

And with this in mind, your ideas will just be alien to the ecosystem, since they do not match regular development style. You can take a look at witchcraft and algae projects which brought algebraic typing ideas to Elixir. These libraries became a fun tools to play with, not a widely adopted practice for the community and the ecosystem.

Any implementation of algebraic types in Elixir comes across a well-know set of problems like

1. Extremely low performance, caused by the lack of compile-time type analysis based optimizations in the language. This means there will be protocol dispatch and map key access for every operation with this structure in runtime.
2. Steep learning curve, since Elixir devs are used to dynamic typing with composition of simple primitives
3. Pattern-matching, since there is no way to pattern match on structures whose internals are encapsulated and interactions are hidden behind the protocols.

While your library certainly gives an unusual (for Elixir ecosystem) approach to solve these problems with lazy evaluation and it is interesting to talk about, I can only suggest you to get a grasp of Elixir’s approach to these problems in return.

For example, for resource management in Elixir we usually use something like this

def at_resource(func) do
  Process.flag :trap_exit, true

  # Open the resource
  resource = open_resource()

  # Spawn the worker
  pid = spawn_link(fn -> func.(resource) end)
  receive do
    # Worker finished normally or exited abnormally
    {:EXIT, ^pid, reason} ->
      :ok

    # It takes too much time, which means that the worker is in
    # infinite loop or just stuck or unreachable
    after :timer.hours(1) ->
      Process.exit(pid, :kill)
  end

  # Close the resource as soon as the worker finishes or dies
  close_resource(resource)
end

This kind of approach is more fault tolerant because

1. It handles infinite loops during resource acquisition. If the function (func) using resource is taking more than 5 seconds, it will be forcibly terminated

2. It handles stack overflow and any other memory exhaustion issue, because as soon the working process hits the memory limit, it will be killed by the VM (or VM will crash, it’s configurable) and the supervising process will close the resource

3. It handles hardware outages (sic!!!), because we can spawn the working process on the separate node, and when the separate node has hardware outage, we will receive either exit signal or a timeout in receive

For more on resource management, I’d suggest reading libraries NimblePool and DBConnection since they’re doing exactly what I’ve described above.

Really appreciate such a deep and comprehensive reply. And I’m inlined with almost all of it. I’ll try to make a more refined statement to reason about:

There is nothing about lazy evaluation of anything in the library, as well as adopting some external abstraction in the first place (nor addressing performance aspects). It’s only DX. Initially I was addressing the problem of copy-pasting and refactoring the same code-approaches across our projects (e.g. time bounded synchronous expression evaluation or “singleton-resource” management). Stick with Elixir’s kernel (primary building blocks) as much as possible was a must. As a result I ended up poking around:

• Stream.resource/3, which is (from the-day-one) a way to “handle resources” of the “stream-nature” (so it’s like 2-in-1: Elixir’s standard library resource management + acquired resource content streaming);
• for as a “composition facility” for enumerables (as I mentioned before, I started with some kind of Resource.use_all!/2, until I realised, for is a more natural candidate as it does exactly the same for lists as a syntactic sugar).

And all this “composable hustle” was just to refactor these patterns into separate library, so they target one concept at a time in a general way, so no preconditions nor assumptions on the nature of resources was made (should it be a file, database connection pool or whatever).

And that is where our points of view diverge:

• you can view Stream.resource/3 as a stream in the first place, but with “resource management” baked in;
• being able to:

  Stream.resource(
    fn -> File.open!("sample") end,
    fn file ->
      case IO.read(file, :line) do
        data when is_binary(data) -> {[data], file}
        _ -> {:halt, file}
      end
    end,
    fn file -> File.close(file) end
  )
  

  but do-it-by-yourself for resources of “non-stream-nature” every time you need it (no matter how: using at_resource-approach or adopting Stream.resource/3);
• you can view for as an exclusive facility for enumerables;
• and at the same time being able to:

  for a <- as, b <- bs, a < b, c <- cs, a + b + c < 10, do: a + b + c
  

  for lists, but:

  Stream.product(as, bs)
  |> Stream.filter(fn {a, b} -> a < b end)
  |> Stream.product(c)
  |> Stream.fitler(fn {{a, b}, c} -> a + b + c < 10 end)
  |> Stream.map(fn {{a, b}, c} -> a + b + c end)
  

  for streams, although for also accepts streams (when you really need a Stream as a result).

So these libraries are nothing more than to view these points as a some sort of asymmetry:

• being able to use Elixir’s kernel “resource management” facility (Stream.resource/3) without “stream-nature” assumption on the acquired resource content (which could be implemented using at_resource-approach);
• being able to use for as a for-comprehension (just as its name suggests , I’m totally ok with Kernel.SpecialForms.for/1).

FWIW, with LazyFor v1.1.0 — Documentation one might do

users = [user: "john", admin: "meg", guest: "barbara"]
result = stream {type, name} when type != :guest <- users do
  String.upcase(name)
end
Enum.to_list(result)
#⇒ ["JOHN", "MEG"]

Kernel.SpecialForms.for/1 is translated to erlang comprehension not to flat_map/2+filter/2.