So, the goal of the adapters is to give you the OTP supervisor structure you would build anyways if you were just using SQS or AMQP directly. So, it’s very opinionated about the supervision structure, but does provide settings to tweak some parts of it. The best comparison I can give is to how Ecto’s Repo transparently does things for you like manage a connection pool. There are settings to manage the number of connections in the pool, but no way to say don’t use a pool and open a connection on every SQL request.

I mean a couple things when I say scalable. One is stable resource usage. So, for any given application, you should generally have near constant memory usage, connections, etc. This helps protect you from resource exhaustion, which could get you in situations where your entire application crashes. This also means that conduit is designed in such a way that you’re system should never be overwhelmed and if you are, it’s easy to tweak a few settings so that you’re not. Basically, it ensures there’s a back pressure mechanism. Conduit can’t make guarantees about the code the user writes, but it uses patterns to ensure that reasonable things happen around the users code. For example, if your messages are large, the BEAM can put them on the binary heap and they may not be GC’d for a long time. So, conduit does work to ensure that doesn’t happen. Also, your code could allocate a lot of memory, but because that’s run in an isolated process that dies after your code is done running, the BEAM can immediately reclaim that memory.

The second thing I mean by scalable, is that it is fast. I only have anecdata for this, but at work we have a couple applications that use conduit and process millions of messages per day and are idle most of the time. This isn’t a guarantee that there will never be queue backups, just that conduit is unlikely to be the reason why you have queue backups.

The third thing I mean by scalable is that it should recover gracefully. The BEAM certainly helps a lot here with that. But some things that are handled specifically by conduit are fault tolerance when an external message broker becomes unavailable. Isolation of user code from other parts of the supervision hierarchy and tools to deal with failures in user code, like the DeadLetter, Retry, and AckException plugs. By default, at least once delivery semantics. So, if something fails processing a message, you’re guaranteed to get that message again.

These quotes explain what the real goal here is:

Conduit doesn’t have a scalable OTP structure for the sake of it. It’s so the user doesn’t need to spend a bunch of time doing that themselves and can focus on their business logic.

Thanks, that’s really helpful. So I think I could summarize it as conduit leverages OTP semantics to minimize the impacts of faulty user code on the overall system while still maintaining high throughput and low latency.

ConduitSQS v0.3.0 was released recently!

This release upgrades gen_stage used internally.