aash

aash

Understanding `max_demand` and `min_demand` in `GenStage`

I am trying to understand in detail how GenStage works by following the documentation. I tried fiddling with max_demand and min_demand by changing their values and understand how they work under the hood. Below is my producer, producer_consumer and consumer code:

producer.ex

defmodule GenstageExample.Producer do
  use GenStage

  def start_link(initial \\ 0) do
    GenStage.start_link(__MODULE__, initial, name: __MODULE__)
  end

  def init(counter) do
    {:producer, counter}
  end

  def handle_demand(demand, state) do
    events = Enum.to_list(state..(state + demand))
    IO.inspect({"events", Enum.count(events)})
    {:noreply, events, state + demand}
  end
end

producer_consumer.ex

defmodule GenstageExample.ProducerConsumer do
  use GenStage

  def start_link(factor) do
    GenStage.start_link(__MODULE__, factor, name: __MODULE__)
  end

  def init(factor) do
    {:producer_consumer, factor, subscribe_to: [{GenstageExample.Producer, max_demand: 4, min_demand: 2}]}
  end

  def handle_events(events, _from, factor) do
    converted_events = Enum.map(events, & &1 * factor)
    {:noreply, converted_events, factor}
  end
end

consumer.py

defmodule GenstageExample.Consumer do
  use GenStage

  def start_link(useless) do
    GenStage.start_link(__MODULE__, useless)
  end

  def init(state) do
    {:consumer, state, subscribe_to: [{GenstageExample.ProducerConsumer , max_demand: 2, min_demand: 1}]}
  end

  def handle_events(events, _from, state) do

    for event <- events do
      IO.inspect({self(), event})
    end

    IO.puts("\n")
    :timer.sleep(1000)

    {:noreply, [], state}
  end
end

For consumer, I set the demand as
max_demand: 2, min_demand: 1

For producer_consmer, I set the demand as:
max_demand: 4, min_demand: 2

I see strange behaviour in the values printed by the consumer, using the above demand configuration:

Output on running mix run --no-halt

{"events", 5}
{"events", 2}
{"events", 2}
{#PID<0.175.0>, 0}
{#PID<0.176.0>, 2}




{#PID<0.175.0>, 1}
{#PID<0.176.0>, 3}
{"events", 2}




{#PID<0.175.0>, 4}
{#PID<0.176.0>, 4}
{"events", 2}




{#PID<0.175.0>, 5}
{#PID<0.176.0>, 5}
{"events", 2}




{#PID<0.175.0>, 6}
{#PID<0.176.0>, 6}
{"events", 2}

My doubts:

  1. If the max_demand for producer_consumer is 4, then why is the first request to the producer has the demand 5?
  2. Then, immediately after the producer produces 5 events, why isn’t it printed by the consumer? Why do I see two demands to the producer for 2 events each?
  3. Strangely, each consumer is printing the same event. This shouldn’t happen if the default dispatcher is DemandDispatcher. Why?

If anybody can help me understand how max_demand and min_demand in a simple, intuitive way, it would be really helpful.

Thanks.

Most Liked

jola

jola

Okay, so I realise now that your scenario is more difficult than I thought at first. You have a producer_consumer with separate demands from the consumer, which makes it more difficult to reason about. In a lot of cases I expect you’d be fine just specifying demand in the final consumer, since it will propagate normally. But basically, the consumer_producer has 4 and 2, so it initially asks for 4 (producer always makes 1 too many, so it gets 5). It handles events in batches of 2, bringing the number of events down to 3, which is less than max demand of 4, so it asks for 1 more, and gets 2 (again, always 1 too many).

In turn, the final consumer has 2 and 1, so it asks for 2 initially, then batches handle events by 1 and asks for 1 more each time.

Could you add the code where you start the stages? How many consumers? 2? How many consumerproducers?

axelson

axelson

Scenic Core Team

I agree that max_demand and min_demand are confusing/non-intuitive names. Unfortunately I don’t have a better suggestion but I would recommend that you read over this GitHub issue that discusses alternative names: Suggestion: new language instead of max_demand and min_demand · Issue #201 · elixir-lang/gen_stage · GitHub

Last Post!

jola

jola

Ah, that’s interesting! There’s so much to learn when it comes to GenStage :slight_smile:

Where Next?

Popular in Discussions Top

New
ricklove
I was just introduced to Elixir and Phoenix. I was told about the 2 million websocket test that was done 2 years ago. From my research, t...
New
sergio
There’s a new TIOBE index report that came out that shows Elixir is still not in the top 50 used languages. It also goes on to call Elix...
New
chuck
Let me start by stating an assumption: Phoenix is a great approach to building REST APIs. There are many reasons for this, but I will ass...
New
lucaong
Hello Elixir and Nerves community, I have been working for a while on an open-source embedded key-value database for Elixir, that I call...
230 14350 124
New
eteeselink
Hi all, In the last days, two things happened: A blog post titled “They might never tell you it’s broken” made the rounds. It’s about ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Other popular topics Top

openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44139 214
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement