Hey everyone –

A few months ago, we announced Sequin, an open-source project that combined Elixir and Postgres for message queuing. Since then, we’ve made a big pivot in our focus, and I’m excited to share where we’re headed.

https://github.com/sequinstream/sequin

v0.6 – CDC

While we started with a broader focus of being a full-featured message queue, we’ve learned the real challenge is Change Data Capture (CDC). That’s what we’re focusing on now: Sequin is a lightweight, Elixir-powered alternative to Debezium.

With Sequin, you can stream Postgres rows and changes to message queues like SQS or GCP Pub/Sub and streams like Kafka. You can start with a backfill of current data in your Postgres table(s), then continue to stream changes in real-time.

If you don’t have a queue/stream, we also support webhook destinations and offer our own native stream, Sequin Stream. Both guarantee exactly-once processing.

Why the change?

Most of our early users came to us specifically for our WAL processing capabilities – they already had their data in Postgres and needed a reliable way to capture changes.

While tools like Debezium exist for this use case, they’re complex to set up and maintain. Debezium requires Kafka and has limited tooling for monitoring and debugging. By focusing solely on Postgres, we’ve built a simpler solution that can match Debezium’s performance without the operational overhead.

What’s new in v0.6?

• We’ve refined our core CDC functionality and now support native destinations for message queues and streams.

• We’ve introduced sequin.yaml for declarative configuration (think lightweight Terraform).

• Added “Change Retention” which uses a Postgres table to retain WAL changes for a period of time. This lets you replay changes to a destination, which is super useful for debugging or error recovery.

• Added “backfills” for streaming data already in a Postgres table to a destination.

• Improved our own native stream, Sequin Stream. It has a HTTP-based consume interface a lot like SQS, but it’s Postgres-based.

• Improved support for very large transactions, large rows, and TOAST columns.

• Various performance and stability improvements.

How it works

We’re still building on the same foundation:

• Elixir implementation (no Postgres extensions needed)

• Works with any hosted Postgres database

• Web console and CLI

• Exactly-once processing guarantees

(The web console is actually LiveView + LiveSvelte, which we’ve written about here.)

Looking forward

Upcoming features we’re most excited about are transforms (write your own in Elixir/JS) and joins (combine multiple tables into one object).

Try It Out

If you’re looking for a lightweight, Elixir-native way to handle CDC from Postgres, try Sequin.

Check out the repo: GitHub - sequinstream/sequin: Postgres change data capture to streams, queues, and search indexes like Kafka, SQS, Elasticsearch, HTTP endpoints, and more · GitHub

Check out one of our quickstarts: Get started with webhooks - Sequin

Or, try it out in the cloud: https://console.sequinstream.com/

We’re excited about this new direction and would love your feedback. Whether you’re using CDC today, have used Debezium in the past, or are just exploring CDC, your thoughts and requirements are invaluable for us to hear. Feel free to comment here or send a DM.

Hey everyone,

Wanted to share all the updates we’ve made to Sequin.

We made a significant overhaul to Sequin in v0.4. Before, you’d push messages to Sequin streams via HTTP. Streams were persisted in Sequin-managed Postgres tables.

As of v0.4, Sequin instead streams your existing Postgres tables.

You can use Sequin to add streaming capabilities to existing data, like you might with a Debezium + Kafka pipeline. Or you can create new tables that you’ll use specifically to power streaming use cases as a pure Kafka alternative.

After connecting a Postgres table to Sequin, you can setup consumers for that table. A consumer can start at any “position” in the table: the beginning, at a specific row, or at the end. If it’s a push consumer, Sequin will push changes to your application via HTTP POST. If it’s a pull consumer, Sequin will provision a new HTTP endpoint that your application’s workers will use to pull messages.

You can have 1 or 1000 workers subscribe to a consumer. And you get guarantees like exactly-once processing of all rows and changes.

So, a table in Sequin is like a topic in Kafka. A consumer is like a Kafka consumer group. And instead of offsets, consumers traverse tables using a sorted column on your table.

Like Kafka, Sequin also delivers changes FIFO. So if a Postgres row changes multiple times in quick succession, each change will only be delivered after the prior change was acknowledged, avoiding race conditions.

So you can think of Sequin as an Elixir app that sits on top of your Postgres database and does 3 things:

1. Safely paginates and detects changes to your Postgres tables.

2. Manages consumer state (FIFO, fan-out, delivery, etc).

3. Offers a control and observability plane on top of all this.

Our bet is that for many (most?) teams and use cases, a Postgres table is a better storage layer than Kafka. You’re already operating Postgres. And data in Postgres is far easier to manage: you can migrate tables to change your data model and easily read it at rest with SQL. Your data is typically already in Postgres – why have a longterm copy in another system just to stream it?

Postgres also allows us to more easily build a ton of useful features, like observability tooling, replays, SQL-based filters on consumers, etc.

While you write directly to your Postgres tables with insert/update/delete, the consumer interface is via HTTP (push or pull). There are a lot of moving parts with message delivery, and we lean on Elixir/OTP to make this smooth.

You can follow along on GitHub, we open source everything: GitHub - sequinstream/sequin: Postgres change data capture to streams, queues, and search indexes like Kafka, SQS, Elasticsearch, HTTP endpoints, and more · GitHub

We have an example with an Elixir+Broadway consumer, which is a great experience:

We also just launched our hosted option, which is a good way to kick the tires.

Finally, I recorded a video yesterday that shows how all this comes together:

Much more to come – let me know if you have any thoughts or questions! We’re adding features all the time, so if we’re missing something, we’d love to hear it

Hey all –

Excited to share that we released an Elixir client for Sequin on Hex!

Sequin’s HTTP interface is straightforward enough, but the SDK is a nice wrapper for those that prefer. It’s a good fit if you don’t need a full-blown Broadway producer/pipeline. And you can use the SDK in test to create/delete streams and consumers.

Here’s an example:

# Define your stream and consumer
stream = "your-stream-name"
consumer = "your-consumer-name"

# Send a message
case Sequin.send_message(stream, "test.1", "Hello, Sequin!") do
{:ok, %{published: 1}} ->
  IO.puts("Message sent successfully")

  {:error, error} ->
    IO.puts("Error sending message: #{Exception.message(error)}")
end

# Receive a message
with {:ok, %{message: message, ack_id: ack_id}} <- Sequin.receive_message(stream, consumer),
     :ok <- YourApp.process_message(message),
     :ok <- Sequin.ack_message(stream, consumer, ack_id) do
  IO.puts("Received and acked message: #{inspect(message)}")
else
  {:ok, nil} ->
    IO.puts("No messages available")

  {:error, error} ->
    IO.puts("Error: #{Exception.message(error)}")
end

Let me know if you have any thoughts or questions