Why not GenStage.Producer (and family)?

Questions / Help
1

I just realized that my big obstacle to groking GenStage has always been the return type of the init/1 callback:

type :: :producer | :consumer | :producer_consumer
init(args :: term) ::
  {type, state} |
  {type, state, options} |
  :ignore |
  {:stop, reason :: any} when state: any

Why was it designed this way, does anybody know?

I have never understood why the type of stage is something that could/should be decided at runtime––every single implementation of init/1 I’ve ever seen hardcodes this value.

A lot of complications in the GenStage callback APIs and docs seem to originate in allowing this to be a runtime decision, with more complicated configuration of 3 different concepts requiring a single overlapping API that they can all share. It seems to me that they should have been defined as similar, but different modules instead: GenStage.Producer, GenStage.ProducerConsumer, and GenStage.Consumer.

A lot of the documentation is cluttered by having type-specific notes for every case for every function. A lot of odd corners of the callbacks would be able to be resolved, too:

Init return consistency:

  • init/1's return value of {type, state} could become just {:ok, state}.

  • init/1's return value of {type, state, options} would not be needed. options here is a kind of a confusing matrix of values based on type. They could just be passed into GenStage.start_link/3's options list as normal and wrong options to the wrong type could complain as appropriate.

  • This would let init/1 fully match GenServer type responses.

Handle conflation:

  • handle_demand/2 would only be required when you use GenStage.Producer.

  • handle_events/3 would only be required when you use GenStage.ConsumerProducer or GenStage.Consumer.

  • This would let both of these callbacks be proper callbacks instead of optional ones, causing compile-time errors instead of runtime ones when not implemented.

Other signature oddities:

  • handle_events/3 for a GenStage.Consumer could use a different return value. Currently it must return {:noreply, events_to_emit, new_state} because it has the same API as a producer-consumer. But it never makes sense for a consumer to emit events so the docs instruct you to return an empty array here for the consumer case. This could just be {:no_reply, new_state} for the consumer case instead.

  • handle_subscribe/4 wouldn’t need the initial producer_or_consumer hint. The confusion would remain for producer-consumers but this feels more like it should be split between a handle_subscribing(options, to, state) and handle_subscription(options, from, state) dichotomy instead, with only producer-consumers offering both, similar to how handle_demand/handle_events would work if producer-consumers could also handle demand.

If these concerns were separated I feel as if the docs would be clearer, the APIs much more familiar, and the whole thing more intuitive. I assume this was considered in the initial design but the rational for the current design eludes me. Any insights? When would you overlap producer and consumer implementations? What am I missing?

3 Likes
2

Some good answers to my question are now available here!

1 Like
3

Oh great, thanks!