Off Broadway Redis - An opinionated Redis list connector for Broadway

Just to add a little more detail on the guts of this - for those that don’t want to dig through the README.

• My implementation follows the ‘Reliable Queue’ pattern outlined in the Redis documentation (RPOPLPUSH | Docs). This means it relies on an RPOPLPUSH command to atomically move the item from the source list to a target ‘working’ or ‘processing’ list before returning said item to the producer. So be sure you push items to the left side of the Redis list you are working from (using LPUSH) - this assumes you doing FIFO work.
• I handle the acknowledgement aspect of Broadway by batching Redis LREM commands to the working list.
• Both Redis commands (RPOPLPUSH and LREM) are pipelined with sensible default batch sizes and limits.

Biggest caveat now is that you must supply the supervised Redis connection atom to the producer when starting. Maybe not a big deal, but I didn’t want to manage that in the initial release and keep it as simple as possible.

FWIW - I recently changed the library over to match the community guidelines for Broadway by renaming to off_broadway_redis.

Note the namespace changes. Outside of that, its a drop-in replacement for the OG version.

https://github.com/amokan/off_broadway_redis

Do you use that library for queueing and processing scraping jobs, by the way?

I’m no longer crawling/scraping the web, but I do use the Broadway library for enterprise data integrations now. Although it is not all that different - except customers pay to access an endpoint they provide.