Confused with designing interactive server workers

Questions / Help
,
1

Hey, I’m quite new to Elixir and trying to make good architecture with my app.

I’ve got a few questions how to don’t mess up app too much (is there any tutorial for it?):
My setup is basically:
-API for requesting some work (for instance downloading videos)
-some server taking those requests and queuing them to some pool of workers.
-interactive communication between server and workers (when I request a playlist I want to get message about every video which finishes downloading from that playlist)

  1. How to achieve robustness?
    I thought about one server responsible for API with DynamicSupervisor having workers as childrens.
  2. How can I implement queue of work?
    In Programming Elixir 1.6>= author used hungry workers, I found GenStage, is it the same or a bit different purposed ideas?
  3. How can I get dynamic messaging from workers?
  4. How can I wrap everything into nice API, callable from Phoenix controllers (I’ll propably use Channels for dynamic communication)?
2

Can you be more explicit about what you’re trying to build? The setup you mentioned - is this something that you already have or something that you plan to build?

You want to expose an API that allows to start long-running jobs and then query for the result. Is that correct?

You can do that in the BEAM, but if you care about robustness, then most probably you’d want to persist the queue so that when the node goes down, the queue does not vanish. To store the queue you can use a relational database, Mnesia or introduce out-of-process queue like SQS or RabbitMQ.

DynamicSupervisor or GenStage with ConsumerSupervisor looks like a good idea (in fact, I used the latter for consuming an SQS queue).

3

I’m creating application which allows downloading videos and playlists from YT. I’ve already implemented this as stubs, so only supervision tree works. So no queue for now.

I want to expose an API to start a potentially long running job and receive some message after it finishes work. I want also check state between those two events - how many videos are downloaded from given playlist.

GenStage may be perfect fit for this usage, but I never used it, so I don’t really know. I thought about using Agent to act as a queue. I’ll try GenStage though.