Designing a Single worker to consume a queue

Questions / Help
1

The Elixir app I’m working on needs to access an external API. The issue is that the API can only be called by one client at a time.
So I’ve created a queue (Myqueue) to store the parameters I want to use when I call the external API and a worker that checks the queue and call the external API when a message is passed.

This is the code for the worker.

defmodule UniqueWorker do

  use Supervisor

  def start_link do
    Supervisor.start_link(__MODULE__, [])
  end

  def init([]) do
    children = [
      worker(UniqueWorker, [], restart: :permanent, id: "unique_worker", function: :do_task)
    ]
    supervise(children, strategy: :one_for_one)
  end

  def do_task() do
    pid=spawn_link(__MODULE__, :loop_do_task, [])
    {:ok, pid}
  end

  def loop_do_task() do
    case MyQueue.pop do
      nil -> {:empty}
      message -> IO.puts "Consume external API with #{message} " 
    end
    loop_do_task()
  end
end

I’m an total exlir newbie. Is this a correct way to solve my problem?

Thanks

2

It really should be a GenServer or GenFSM (of some variant), those are specifically for serializing requests like that.

3

fwiw I’d use Poolboy, and set the pool size to 1?! This way you can easily scale the app later should you need to let more clients consume the api…

1 Like
4

His post implied to me that the remote API only ever allows one job at a time. If possible to scale up later though then poolboy now would definitely be a better way to go (or one of the similar ones). :slight_smile:

1 Like
5

This is exactly what GenStage https://hexdocs.pm/gen_stage/Experimental.GenStage.html is for.

Couple of additional minor points:

Don’t do {:empty} instead just do :empty. There’s no point in wrapping it with a tuple.

You always want to supervise an OTP behaviour. The getting started guides give you some examples of this, I highly recommend you go through them. http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html

So, you’re right on target conceptually with the whole loop thing, but the idea with OTP behaviours is that they extract that common pattern and let you focus on individual functionality specific to your use case. It also lets the process handle debugging and other utility type messages without requiring any explicit handling on your part.

GenStage specifically is an OTP behaviour built to around the idea of handling the streaming of events between processes, and I think will fit your use case well.

2 Likes
6

Thank you so much for your suggestions. I’ll definitely check GenStage.