I really need only consuming and I want the minimum amount of boilerplate. That’s going to be my main criteria.

I don’t have the code anymore so I’ll only from my bad memory but I have used spreedly/kaffe for a year and it was a solid basis to setup the project.

For consuming, if you need to customize the listeners architecture you will need to understand brod. And you will need to understand brod and kpro etc. to debug some unpleasant errors.

Before leaving the project I was seriously considering replacing that whole stack with erlkaf and made a quite convincing replacement PoC in a few hours. There is not much boilerplate either but I had problems with dependencies, hopefully fixed by now.

Do you remember how did you work around that?

We did not It was not a frequent error. Just that brod is slow to start and to setup the consumer group. Or we did something wrong but erlkaf booted way faster.

For the mocking part I put a private API in front of the kafka libraries to abstract the consuming/producing API and then I implemented and maintained a simple pubsub system mimicking the behaviours of Kafka. And some test helpers on top of that.

This mock was only for tests but it had it’s own unit tests just to be sure. You need to implement a lot of things. Even if you want to only consume, you will need to produce to your mock from the test.

Anyway, a lot of work, not recommended…

Yep, that’s why I would prefer a library that comes with a Kafka mock out of the box but oh well. In my current employer colleagues maintain a thin wrapper on top of :brod but it only supports producing and I have no free time to contribute to add consuming to it – not yet I don’t have the time anyway, Soon™ I should have it.

Ah well, I guess I can just ignore it then, though I am not keen on the idea of losing data which shouldn’t happen anyway because the Kafka topic will be filled by other apps, and the consumer group I’ll use will remember the offset so technically nothing should be lost.