tomekowal
Best way to iterate a Stream
Background: I am writing a multiplayer mini-game with math puzzles. I need the puzzles to be random but reproducible given a seed. I decided to model them as an infinite Stream. The players will ask the stream for new puzzles. I did something like that but for one player in Elm long time ago.
http://tomekowal.github.io/elm-multiplication-game/
Now, the real question is: how do I get elements from the Stream one by one? I’d love an API similar to String.next_grapheme/1
String.next_grapheme("asdf")
{"a", "sdf"}
String.next_grapheme("")
nil
{1, stream} = Stream.next([1, 2, 3])
{2, stream} = Stream.next(stream)
{3, stream} = Stream.next(stream)
nil = Stream.next(stream)
Unfortunately, there is no such API neither in Stream nor in Enum modules. It seems that the question is fairly popular:
https://github.com/elixir-lang/elixir/issues/2515
And there are a couple of solutions to it:
https://github.com/tallakt/stream_split
https://github.com/hamiltop/streamz
https://github.com/Qqwy/elixir-iter
None of them is perfect. E.g. StreamSplit works on infinite Streams but breaks on lists.
Did you encounter such a problem? How do you code around it?
When I started, I was pretty sure something like that is in the standard library but I was wrong.
Is there a reason it is not there? Maybe it doesn’t play well with Stream.interval?
Marked As Solved
sasajuric
I think you can find an example in gen_stage code.
Also Liked
Qqwy
Consuming a collection (both finite and infinite ones) one-by-one is definitely useful. I have had conversations with @josevalim about this before. The reason Enum (and, by extension, Stream) do not support this by default is twofold:
- They expose their iteration process as a fold (which is the academic term for
Enumerable.reduce), which always consumes the complete collection. For certain kinds of Enumerables, it is impossible or impractical to take elements out one by one. An example are file streams: iterating them one-by-one means that you have to keep the file open for the whole time you take elements out one-by-one. - Basing Enumerable on top of a ‘take one per time and keep the collection intact’ interface would be much slower (if I recall correctly, this was tried and benchmarked in some pre-stable version of Elixir. From the top of my head, I believe it was 4-5x slower, which is very significant since so much time in Elixir is spent working with/on collections).
Nevertheless, there are cases like in your use-case where this is very useful. You mentioned the iter repository in your post. That is an old attempt at fleshing out what later became the Extractable library/protocol.
Extractable only comes with out-of-the-box implementations for Lists, Maps and MapSets.
I am fairly certain that you could create a custom struct for your puzzle generating logic that contains a function which returns a {puzzle, new_struct_with_function_for_next_puzzle}.
liskin
There is one additional reason why streams can’t be consumed one-by-one and instead need to be folded (Enumerable.reduced) at once: the stream generator may have side-effects (example: consuming File.stream! advances the file position) and therefore must be invoked at most once.
Try this:
{e1, cont1} = MyStream.next(File.stream!("test.txt", [], 1))
{e2, cont2} = MyStream.next(cont1)
{e2a, cont2a} = MyStream.next(cont1)
IO.puts("#{e1} #{e2} #{e2a}")
Assuming test.txt contains 12345, one would expect to see 122 but instead you get 123.
The StreamSplit library suffers from the same problem:
iex(1)> {head, tail} = StreamSplit.take_and_drop(File.stream!("test.txt", [], 1), 1)
{["1"], #Function<55.126435914/2 in Stream.resource/3>}
iex(2)> Enum.take(tail, 2)
["2", "3"]
iex(3)> Enum.take(tail, 2)
["4", "5"]
The documentation for Enumerable repeatedly mentions that “In case a reducer/0 function returns the :suspend accumulator, the :suspended tuple must be explicitly handled by the caller and never leak.” And that’s exactly what your code does — the continuation leaks and can then be called multiple times, messing up the side effects.
If one comes from a Haskell background (like me), it really is somewhat suprising. Stream resembles a lazily evaluated linked list, but the BEAM VM doesn’t have lazy evaluation and thunks, so the side-effectful Stream must be treated with care. ![]()
Here’s my stab at a safe-ish stream iterator:
defmodule StreamStepper do
def stream_stepper(stream) do
stream
|> Enumerable.reduce({:cont, nil}, &stream_stepper_suspender/2)
|> stream_stepper_continuer()
end
defp stream_stepper_suspender(head, nil) do
{:suspend, {head}}
end
defp stream_stepper_continuer({done_halted, nil}) when done_halted in [:done, :halted] do
[]
end
defp stream_stepper_continuer({done_halted, {head}}) when done_halted in [:done, :halted] do
tail = fn -> [] end
[head | tail]
end
defp stream_stepper_continuer({:suspended, {head}, tail_cont}) do
once = callable_once()
tail = fn -> once.(fn -> tail_cont.({:cont, nil}) |> stream_stepper_continuer() end) end
[head | tail]
end
defp callable_once do
seen = :atomics.new(1, [])
fn fun ->
case :atomics.compare_exchange(seen, 1, 0, 1) do
:ok -> fun.()
_ -> raise "protected fun evaluated twice!"
end
end
end
end
It’s just a hack, the continuation kind of leaks as well, but this leak is protected by callable_once which raises whenever the possibly side-effecting fun is invoked more than once.
sasajuric
Personally, I would write a small abstraction around puzzle sequence, with the following operations:
PuzzleSequence.new(seed)PuzzleSequence.next_puzzle(sequence)
From there, I’d write the rest of the system and see how it unfolds. If I notice that there is some significant overlap between what I wrote and Stream/Enum functions, I’d try to see if I can somehow reuse the existing enum logic, e.g. by doing some trickery proposed here. But before that, I’d just stick with this simple interface.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








