Generic background job processor (for stream consumption)?

Questions / Help
1

We are looking to utilize stream processing into our infrastructure via something like Kafka. We love things like Sidekiq and Resque because of the features you get with them out of the box: admin web ui, failed job tracking, retrying, etc.

Are there any generic background job frameworks for Elixir that allow you to specify how to acquire jobs?

Or are there any that are specific to Kafka, AWS Kinesis, etc?

I basically want a stream processor that has dead letter queue functionality and a web ui for managing the DLQ, retries for DLQ, etc.

And I, uhh, don’t want to develop this from scratch… :slight_smile:

2

I guess I’d even settle for an Elixir stream processor that has error callbacks (on which I could just throw the message onto Redis in Sidekiq format and then point Exq or some such at the queue).

I guess I can just Google for that. :v:

3

Well, I think that GenStage+Flow goes a long way. Of course Flow does not itself have a web GUI, but you do have the best Elixir integration you can imagine.