Disclaimer: Elixir/OTP newbie (barely 1month in) here.
So I’m rewriting a project in Elixir to learn the language and experiment.
One thing i wanted to internalize is the cron jobs we currently use to update our Database.
We have about four jobs, which
- grab a list of all ids from db,
- chunks them,
- calls an api with each chunk, and
- persists the (filtered + mapped) response to db
So my first plan was to have a supervised Queue, to which i pass my ids (loop + long sleep) (gist for the queue)
GenStage looks like it’s better suited for this kind of task, seeing as i can split my steps pretty cleanly. The problem i see though is, that i can’t response to every demand the consumers have.
In my tests handle_demand is only called once, so i can’t just ‘ignore’ the call. And blocking the response with a recursive check-or-sleep-style feels really hacky.
What’s the cleanest/idiomatic solution here?
Edit: removed half-sentence in the middle
