Kafka_ex producers with connection pooling

Questions / Help
,
1

Using Kafka_ex, I see reference in their docs on how to setup connection pooling with poolboy, but the example is super sparse and assumes a lot of knowledge on how poolboy and kafka_ex interact.

In my case, I have an application that needs to write to a kafka topic, and could use multiple processes to write across partitions. However, Kafka_ex out of the box uses one process to write.

My question is this: does someone have an example using connection pooling that shows the full setup?

Thank you,
-Steve

2

I believe you will need to implement a worker with the KafkaEx.Server behaviour and then just pass it to your poolboy child spec config, something along the lines of:

[
  {:name, {:local, @name}},
  {:worker_module, App.KafkaWorker},
  {:size, @size},
  {:max_overflow, @max_overflow}
]