Trying to better understand broadway

[polygonpusher]

Was reading https://blog.appsignal.com/2019/12/12/how-to-use-broadway-in-your-elixir-application.html
and wanted to get a better understanding of how the incoming messages can be triggered and managed.

Right out the box with this tut, it uses a WorldTemp.CityProducer example that I have a hard time understanding. There are two cases of handle_demand, one is for when demand > length(state) the other for when this is not true.

First: What does demand here represent? It starts with a default value of 10. Where does this value come from and what does it represent? What should I read to get a betting understanding of this value?

Second: The city list function is a static list. What if I wanted to do an ecto query and get a list of rows that should be processed. Is this where that request would look up said rows?

I guess so far the producer is the most confusing part to me.

Edit: I think I figured out my disconnect. This module is a GenStage. https://hexdocs.pm/gen_stage/GenStage.html?#c:handle_demand/2 and not directly related to Broadway.
Time to go learn some GenStage I guess.

[akoutmos]

Author of the article here and good questions!

The reason for the two different handle_demand functions (one with the guard and one without) is because our GenStage producer effectively creates an infinite list of cities. That first function with the guard ensures that there are enough cities in the list to satisfy the incoming demand by adding more items to the list. The demand in this case represents what the downstream Broadway consumers are requesting from this producer.

While you can use GenStage as a producer, in production you most often would use Broadway with something like RabbitMQ or Kafka. I chose GenStage as a producer for that article mostly to keep the length of the article within reason. I have a more in depth 2-part series tutorial on my personal blog that leverages RabbitMQ, Broadway, and Ecto:

Broadway, RabbitMQ, and the Rise of Elixir Part 1

Broadway, RabbitMQ, and the Rise of Elixir Part 2

[polygonpusher]

@akoutmos

While you can use GenStage as a producer, in production you most often would use Broadway with something like RabbitMQ or Kafka.

Say I wanted to query a list of cities using Echo via fetching a list of cities from the repo on an interval. Where would you start with something like that? In my case there could be times there are no cities and othertimes the db is full of cities. Is this a correct use case for this?

[akoutmos]

I did something roughly similar to what you are describing in the 2 part series. I have a GenServer that publishes messages to RabbitMQ when certain conditions are met. You could easily modify my GenServer to make Ecto queries on a regular interval instead though. With the data that gets fetched from the DB, you can publish individual messages to RabbitMQ for each item of work that needs to take place. On the other side of that, Broadway will read those messages and process them.

[polygonpusher]

Thanks for the pointers, that helps.