SQNS (or OffBroadway.SQNS or Sequins) - SQS/SNS data processor using Broadway

Your Libraries & OS Mentoring Libraries
1

I’m not here to announce a release (yet). Instead, I’m here for some pre-release opinions.

I’ve created a library tentatively named SQNS, for Simple Queuing and Notification Service. It’s not a typical Broadway library, as it uses BroadwaySQS as its producer and acts instead as an abstract processor and batcher.

The processor’s handle_message/3 takes a message from SQS, pre-processes it a little, hands it off to a process/2 callback, then post-processes the result before sending it back to Broadway. The batcher forwards the resulting message to an SNS topic, which can then be subscribed to by any number of consumers (including one or more SQS queues, each with their own SQNS handlers). It also provides some convenient utility modules to set up the queues and topics and to establish the subscriptions among them. We’re using it as the basis for a really tidy data processing / content ingestion engine.

Here’s where my questions come in:

  1. Has the basic description piqued your interest enough to want more details? I’m trying to decide whether to keep this (for now) as an internal tool or make it a community library, and early interest would help me demonstrate (to my managers) the value of the extra time spent making it releasable.
  2. (Primarily for @josevalim) Does something like this belong in the OffBroadway namespace, or is it too much of an abstraction? Here in my office we pronounce SQNS as “Sequins,” which kind of fits the whole Broadway theme, so releasing it simply as Sequins is an attractive option.

I’ll keep updating this thread with info and examples as I continue to work towards a release.

Thanks!
MBK

4 Likes
2

Hi MBK! I was originally thinking the OffBroadway namespace would be mainly for producers. And I do think this is an abstraction on top, rather than something used by Broadway itself, I like the Sequins route. :slight_smile: Thanks for asking!

3 Likes