axelson
Using GenStage to batch events
I am working with an external API (external from the Elixir server at least) and I want to minimize the amount of API calls to a particular endpoint. It feels like GenStage is a good fit for this use case but I’ve had a little trouble getting everything to line up.
I want to:
- Collect all events every 100ms
- Batch the events into batches of up to 150
- Have three workers running in parallel to send these batched requests to the remote API
These seem to fit into three GenStage stages
Collector- GenStage:producerthat collects events from other parts of the elixir system in a FIFO queue (based on:queue) and sends the events only inhandle_demand/2Batcher- GenStage:producer_consumerthat asks theCollectorfor 1000 events every 100ms and batches them into batches of 150 (runs in:manualmode)RequestSupervisor- GenStageConsumerSupervisorthat requests the batched events from theBatcherand starts workers that call the external API
The code I have seems to work but I feel like I may be going against the ethos of GenStage since I’m not really propagating demand all the way up the chain. Specifically the Batcher and the Collector both mostly ignore demand. The Batcher is set to :manual mode and asks for a static 1000 events every 100ms.
Another issue is that this setup will always incur a penalty of 100ms on each event even if there are more than 150 events that are added to the Collector at once.
But also keep in mind that I only expect about 5-10 events every 100ms (and maybe even less). But I want to have a good base for future scaling if necessary.
Any thoughts on this architecture? Is there anything that I haven’t considered that I should consider?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 2 of 2 Posts
axelson
Or perhaps this setup would be better:
Collector- GenStage :producer that collects events from other parts of the elixir system in a FIFO queue (based on :queue).handle_demand/2Delayer- GenStage :producer_consumer that asks theCollectorfor 1000 events every 100ms:manualmode?RequestSupervisor- GenStage ConsumerSupervisor that requests the batched events from theDelayerand starts workers that call the external APII could probably find a better name than
Delayer. Does this setup seem better? I think it might be because previouslyBatcherandCollectorboth needed to work together forBatcherto be able to actually batch events. Now batching events is the responsibility of theCollector. Also the demand can handled a little better in theCollectorrather than just ignored, but now I need to prototype to make sure that I can get the batching to still work the way I want.stefanchrobot
How about just using a plain GenServer with a FIFO queue and a timer?