:queue vs genstage state

That’s not the case. Under the hood :queue uses a tuple with two separate lists, which has rather different performance characteristics for the usecase of a fifo queue (hence why :queue exists). With just a plain list either your queuing or dequeuing would be expensive due to how linked lists work. Queue shifts that by making both generally inexpensive at the expense of needing to rebalance head and tail ever so often.

But yes sheer size will eventually become a problem no matter how you store items.

:queue good. Use :queue in some GenServer and Broadway (i.e. GenStage) state.

I don’t feel like it required an excess of “defensive coding”. Do you mean checking if you got something out of it or not? You’re not doing that with a list?

That makes sense. Thank you for the explanation!

E.g. compare:

  def handle_demand(demand, queue) when demand > 0 do
    {events, new_state} =
      case :queue.len(queue) > demand do
        true ->
          {head_queue, tail_queue} = :queue.split(demand, queue)
          {:queue.to_list(head_queue), tail_queue}

        false ->
          {:queue.to_list(queue), :queue.new()}
      end

    {:noreply, events, new_state}
  end

to

  def handle_demand(demand, state) when demand > 0 do
    {events, new_state} = Enum.split(state, demand)
    {:noreply, events, new_state}
  end

Is it a good idea to use len? You could write yourself a recursive wrapper around out

defmodule Queue do
  @doc """
  Take `count` items from the front of the queue.
  
  Returns a list of items and the new queue: `{items, queue}`.  If there are
  less than `count` items in the queue, all of them will be returned.  If the
  queue is empty, an empty list is returned.
  """
  @spec take(count :: pos_integer, :queue.queue()) :: {list, :queue.queue()}

Well, you see what I mean: it requires additional work.

More code for a more appropriate and well working data structure though. You’re comparing apples to oranges here.

My point was only that with the addition of the relatively unfamiliar (?) :queue module, the GenStage examples may veer into discussions about data structures instead of focusing on how stages should be organized. I would argue that understanding how to properly organize stages is more important to GenStage’s rai·son d’ê·tre, no?

It’s not as black and white. Yes simpler examples are generally a good thing. But using a list in the examples in question is not just making the examples simpler, but also a surefire way for people shooting themselves in the foot, when they copy from the example. Using a list to model a fifo queue is plain not what you want to do if this is meant to hold more than a handful of items.

Thanks for this discussion. I’ve often wondered about the utility of :queue myself. It seems easy enough to use lists for this with the understanding that enqueueing would be expensive.

#enqueue
q = 1..5 |> Enum.reduce([], fn i, q -> q ++ [i] end)
#dequeue
[hd | new_q] = q

I suspect this only matters beyond a certain size, which is where :queue becomes important.