Broadway Redis Producer

genstage
redis
broadway
#1

I published a simple (but opinionated) Redis producer for Broadway that I have been using in a production system I’m wrapping up. If anyone wants to check it out and beat on it, it can be found in hex and docs are published.



https://hexdocs.pm/broadway_redis/BroadwayRedis.Producer.html

TLDR - If you have things in Redis lists and want to try to process them with Broadway, it may be a fit for you. There are for sure caveats and things that can be improved, but seems to work real well so far.

Glance at the README for more info.

7 Likes

#2

Just to add a little more detail on the guts of this - for those that don’t want to dig through the README.

  • My implementation follows the ‘Reliable Queue’ pattern outlined in the Redis documentation (https://redis.io/commands/rpoplpush#pattern-reliable-queue). This means it relies on an RPOPLPUSH command to atomically move the item from the source list to a target ‘working’ or ‘processing’ list before returning said item to the producer. So be sure you push items to the left side of the Redis list you are working from (using LPUSH) - this assumes you doing FIFO work.
  • I handle the acknowledgement aspect of Broadway by batching Redis LREM commands to the working list.
  • Both Redis commands (RPOPLPUSH and LREM) are pipelined with sensible default batch sizes and limits.

Biggest caveat now is that you must supply the supervised Redis connection atom to the producer when starting. Maybe not a big deal, but I didn’t want to manage that in the initial release and keep it as simple as possible.

2 Likes