Off Broadway Redis - An opinionated Redis list connector for Broadway

Your Libraries & OS Mentoring Libraries
, ,
1

I published a simple (but opinionated) Redis producer for Broadway that I have been using in a production system I’m wrapping up. If anyone wants to check it out and beat on it, it can be found in hex and docs are published.

Edit: see update/name change info in this post.

https://hexdocs.pm/broadway_redis/BroadwayRedis.Producer.html

TLDR - If you have things in Redis lists and want to try to process them with Broadway, it may be a fit for you. There are for sure caveats and things that can be improved, but seems to work real well so far.

Glance at the README for more info.

8 Likes
2

Just to add a little more detail on the guts of this - for those that don’t want to dig through the README.

  • My implementation follows the ‘Reliable Queue’ pattern outlined in the Redis documentation (https://redis.io/commands/rpoplpush#pattern-reliable-queue). This means it relies on an RPOPLPUSH command to atomically move the item from the source list to a target ‘working’ or ‘processing’ list before returning said item to the producer. So be sure you push items to the left side of the Redis list you are working from (using LPUSH) - this assumes you doing FIFO work.
  • I handle the acknowledgement aspect of Broadway by batching Redis LREM commands to the working list.
  • Both Redis commands (RPOPLPUSH and LREM) are pipelined with sensible default batch sizes and limits.

Biggest caveat now is that you must supply the supervised Redis connection atom to the producer when starting. Maybe not a big deal, but I didn’t want to manage that in the initial release and keep it as simple as possible.

2 Likes
3

FWIW - I recently changed the library over to match the community guidelines for Broadway by renaming to off_broadway_redis.

Note the namespace changes. Outside of that, its a drop-in replacement for the OG version.

3 Likes
4

Do you use that library for queueing and processing scraping jobs, by the way?

5

I’m no longer crawling/scraping the web, but I do use the Broadway library for enterprise data integrations now. Although it is not all that different - except customers pay to access an endpoint they provide.

1 Like