mbklein

mbklein

SQNS (or OffBroadway.SQNS or Sequins) - SQS/SNS data processor using Broadway

I’m not here to announce a release (yet). Instead, I’m here for some pre-release opinions.

I’ve created a library tentatively named SQNS, for Simple Queuing and Notification Service. It’s not a typical Broadway library, as it uses BroadwaySQS as its producer and acts instead as an abstract processor and batcher.

The processor’s handle_message/3 takes a message from SQS, pre-processes it a little, hands it off to a process/2 callback, then post-processes the result before sending it back to Broadway. The batcher forwards the resulting message to an SNS topic, which can then be subscribed to by any number of consumers (including one or more SQS queues, each with their own SQNS handlers). It also provides some convenient utility modules to set up the queues and topics and to establish the subscriptions among them. We’re using it as the basis for a really tidy data processing / content ingestion engine.

Here’s where my questions come in:

  1. Has the basic description piqued your interest enough to want more details? I’m trying to decide whether to keep this (for now) as an internal tool or make it a community library, and early interest would help me demonstrate (to my managers) the value of the extra time spent making it releasable.
  2. (Primarily for @josevalim) Does something like this belong in the OffBroadway namespace, or is it too much of an abstraction? Here in my office we pronounce SQNS as “Sequins,” which kind of fits the whole Broadway theme, so releasing it simply as Sequins is an attractive option.

I’ll keep updating this thread with info and examples as I continue to work towards a release.

Thanks!
MBK

Most Liked

josevalim

josevalim

Creator of Elixir

Hi MBK! I was originally thinking the OffBroadway namespace would be mainly for producers. And I do think this is an abstraction on top, rather than something used by Broadway itself, I like the Sequins route. :slight_smile: Thanks for asking!

Where Next?

Popular in Announcing Top

deadtrickster
I’ve just released stable versions of my Prometheus Elixir libs: Elixir client [docs]; Ecto collector [docs]; Plugs instrumenter/Export...
New
OvermindDL1
I created a new library (rather I pulled out a couple files from my big project), it manages an operating system PID file for the BEAM. ...
New
tmbb
I’ve published the first version of my Makeup library. It’s a syntax highlighter for Elixir in the spirit of Pygments, Currently it highl...
New
seancribbs
Today I released a new dialyzer Mix task as the dialyzex package! At the time we started writing this task, the existing dialyzer integra...
New
josevalim
Hi everyone, We would like to announce that Plataformatec is working on a new MySQL driver called MyXQL. Our goal is to eventually integ...
New
bluzky
You may know https://ui.shadcn.com/, a UI component library for React. I really love it’s design style and components. I’ve built some co...
384 14136 119
New
kip
Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir. I...
622 18934 194
New
ahamez
Hi everyone, I’ve been working on this protobuf library for 3 years. We use it in the company I work for, EasyMile, to communicate with ...
New
josevalim
Hello everyone, We have just released NimbleCSV which is a small and fast CSV parsing library for Elixir. It allows developers to define...
New
OvermindDL1
Been making an MLElixir thing (not released yet…) for fun in spare time in the past day. I’m just trying to see how much I can get an ML...
132 14057 106
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement