Share workload between workers

Questions / Help
1

Hi,

I have a bit of a gap in my understanding of Elixir and workers.

To give a bit of context, I am collecting messages from an AMQP queue with a number of consumer GenServers. The number of GenServers is determined from my configuration file. When each of the servers consumes a message, I would like it to do some work on it and, in some cases, tell another process to submit the message to a different AMQP queue.

My question is, how do I have each of the consumer processes call the same send message function on a module, and have that module handle the distribution of the request between a number of publisher workers?

I feel like this is something that should be very easy to do with Elixir but I’m just struggling to piece it all together.

Appreciate any help :slight_smile:

2

Hello and welcome,

While You could do it as You describe, there are better tools to do this.

In particular, GenStage, Broadway. The first being the lower level tool for ingesting such pipeline, with back pressure support.

For your question, Maybe You can pass arguments to the send function.

Your pipeline is probably better managed if You have a Producer - ConsumerProducer - Consumer pipeline.

3

have that module handle the distribution of the request between a number of publisher workers

“that module” will need to manage a pool of publisher workers, the send message function checks out a worker then tells it to send message.

Usually poolboy is used to implement the pool, your module just needs to configure it and obviously check out a worker and so on.

At a very high level, in order to handle distribution, there has to be a GenServer that is aware of some other GenServers and their states.