swelham
I’m trying to put together a plan for a library that I am working on whereby I need to provide an interface that allows the client to asynchronously call out to external APIs (HTTP endpoints in this case). I also need to make sure that this is rate limited to adhere to the rate limiting on the externals APIs.
From what I have read GenStage sounds perfect for this and using a ConsumerSupervisor I would be able to dynamically start workers for each request as they come in and let GenStage limit the number of workers it will start concurrently.
Does anyone know of any good examples of this type of setup in practice?
I am trying to get my head around how this all fits together and feel a real world example would help me out here.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixir-ls
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
mechabyte
Discord published a blog post covering how they use genstage to buffer calls to Firebase APIs. It has some code examples as well as performance statistics.
swelham
That looks awesome thanks @mechabyte. I will have a proper read through when I get home from work.
swelham
In addition to the discord article linked above, I came across this talk which is probably the best explanation and demo of an end to end genstage setup I have come across.
Well worth a watch (especially if you’re new to genstage)
krapans
Nice. haven’t seen this talk yet.
Q: Is there use case for GenStage for example for simple CRUD API calls for users, projects, etc. Just for simple data requests?
swelham
From my basic understanding so far I would say yes there is. Genstage looks to be a great fit for when you’re having to deal with external forces such as networking for example (rest apis, databases, etc…).
It allows the workers to dictate how much work they are given which prevents the likely important part of the system being overloaded and to remain free to do its thing and cause the backup of events further up the stack.
Someone please correct me if I have got that wrong.
krapans
Ok, this sounds cool. Is there any example of how you write GenStage server for example for User CRUD module? I don’t have an idea how you manage those actions in GenStage server.
swelham
I would suggest definitely take the 40 minutes to watch that talk as that will probably answer your question better than I could.
The demo is a restruant simulation which demonstrates the full cycle of tables placing an order (that’s processed through genstage) and then receiving thier food all within strict timeouts. Akin to sending CRUD data into genstage and receiving some data when it’s finished.
krapans
Ok, super. Added to my tomorrows to do list. Thank you!
wfgilman
Here’s a simple example of a multi-stage pipeline. I found Jose’s talk really helpful (linked in example).
swelham
Thanks @wfgilman, that’s a very interesting example. I haven’t seen any others that are dynamically starting and linking multiple stages like that.