pba
GenStage Semi-Manual backpressure
I am writing a rate-limiter for a code executer using Genstage. My code looks like this [hope the post is not too long 
]
+-------------------+ +----------+
| ExecutionRequester| | Ticker |
| :producer | |:producer |
+-------------------+ +----------+
| +-------------------+ |
:exec_request| TickerExecutor | |:tick
+------> :producer_consumer<----+
+-------------------+
|
| +--------------+
| |ResultConsumer|
+---> :consumer |
+--------------+
TickerExecutor should consume from both ExecutionRequester and Ticker, and do a zip operation:
Only 1 :exec_request is allowed for 1 :tick event produced by Ticker.
I have implemented :manual subscription handling for the TickerExecutor → ExecutionRequester subscription, but felt like 90% of the code (:min_demand, :max_demand) is reimplementing GenStages :automatic subscription. I’m, weary of doing the same for the Ticker subscription, before asking.
As of now I’m consuming and wasting :tick events when TickerExecutor is starved of :exec_requests
Thus my 2 questions:
- Is there a way to announce to GenStage the actual consumption [Edit: or rejection] of events, and not just the delivery via
GenStage.handle_events/3? - Is there a cleaner way to do a
zipbetween two subscription event streams on a:consumer, than just going:manualon both subsctiptions ?
Edit: @admins I have found no Tags for GenStage or the Experimental Module. Would it be ok to add them?
Most Liked
josevalim
Well it buys you an easy way to subscribe to a multitude of tickers in a structured (a.k.a. documented by @josevalim
Sorry, to be clearer, I meant only the ExecutionRequester being GenStage. You should keep the rest!
So If you are undecided about Flow.zip
I will be glad to add that. Can you please open up an issue?
josevalim
Is there a way to announce to GenStage the actual consumption
I am not sure it is clear what you want to announce here? Which stage should tell the other stage what?
Is there a cleaner way to do a zip between two subscription event streams on a :consumer, than just going :manual on both subsctiptions ?
So after thinking about the use case you described above, I am wondering if it wouldn’t be better for you to have a ExecutionRequester as a separate process that you request every time there is a tick. With max_Demand of 1 and a zip, GenStage is not buying you anything for the first three processes: it would be quite equivalent to using regular processes.
The only scenario zip would make in GenStage is if we could still have a high demand value (and you would have back-pressure if one of the sources is slower but typically not). If we ever implement something like this, it would be as Flow.zip.
pba
Hi,
Thanks for answering!
Right now accepting a handle_events/3 call [EDIT: in a :consumer] is the same has having actually handled the events. In reality the Consmer can wait for events from another subscription. What I would like is to use :automatic subscriptions, but be able to freeze the demand. so that backpressure is built up on ExecutionRequester while i still accept events from Ticker.
Well it buys you an easy way to subscribe to a multitude of tickers in a structured (a.k.a. documented by @josevalim ) way .
I only now realize that Ticker could provide a flexible rate-limiter for Flow pipes. Just zip in a :tick event stream, and you know exactly how many and how often items are processed, by controlling the emission behaviour of the Ticker producer.
Right now i have implemented:
- rate per second
- maximum item count per interval (e.g 1500 per day for free goolgle geolocation calls)
- bucketed handling of [EDIT:
:tickevent allocation per interval].
So If you are undecided about Flow.zip I hope to find some time to separate that functionality from the Ticker and make a separate lib. Who knows who needs ratelimiting for Flow Pipes.
P.S. See you next week in Sofia
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"