Tidefall - An ETS buffer for high-throughput writes and batch processing

Congrats on the release. I have a couple of questions.

• Why is this library an application? Couldn’t it be just a supervisor which the user adds to their supervision tree?
• Can the library be integrated with Broadway somehow?

Thanks! And both good questions.

Couldn’t it be just a supervisor which the user adds to their supervision tree?

First, an important point. The application does not own your buffers. Your buffers still live in your supervision tree. You add {MyApp.EventQueue, processor: &MyApp.Sink.export/1} to your children, and you control the restart strategy.

The rule I used is simple. If a thing belongs to one buffer, it lives in that buffer’s tree. If all buffers share a thing, it lives in the application. The partitions and their ETS tables belong to one buffer, so they stay in the buffer tree. Two things are shared, so they moved to the application.

The first is Tidefall.Metadata. It is one ETS table. It holds the current table pointer for each partition. Every write reads this pointer to find the active table. Every processing tick rewrites it when the partition swaps the two tables. It is a GenServer only because a process must own the table. No call goes through it. A call would make one process the bottleneck for every write in the system.

This is also why it is not :persistent_term. That was the first design. :persistent_term.put/2 copies the whole store on every write, and this pointer changes on every tick, for every partition, in every buffer. ETS gives fast reads and fast writes, with no copy.

The second is Tidefall.Registry. Each partition registers under its buffer name. The buffer uses it to find all of its partitions.

Now the real question. Why not put these two in the buffer supervision tree? Because then every buffer gets its own copy, and the cost is not small. The Registry uses one partition per scheduler by default. On a 16-core machine, that is 16 partitions, and each partition has its own table and its own process. Five buffers would create 80 registry partitions, only to support lookups. One shared Registry pays that cost one time. You also get better concurrency, because the partitions follow the schedulers, not the buffers.

The metadata table has the same shape. Each buffer adds only a few entries. An ETS lookup costs the same with 10 keys or with 10,000 keys. So one table per buffer adds tables, but it adds no speed.

The result is one small supervisor with two children for the whole VM. Ecto, Nebulex, and Phoenix.PubSub use the same approach. Shared infrastructure lives in the application, and your own components live in your tree.

If the Registry is still too large for your system, you can tune it with config :tidefall, registry_partitions: 2.

Can the library be integrated with Broadway somehow?

Yes, you can. But let me be honest about one thing first. If you only need to group messages into batches, use Broadway’s batchers. Do not add Tidefall. Broadway already does that job well. Tidefall solves a different problem. A batcher groups messages. Tidefall collapses them.

Here is the difference in practice. Your topic carries one million events, but they cover only one thousand entities. With a batcher, a batch of 1000 messages still holds 1000 messages. If 900 of them repeat the same key, you receive all 900. You must collapse them yourself inside handle_batch/4, and you hold all of them in memory until the batch closes.

With Tidefall.HashMap, the write collapses the duplicate immediately. The buffer keeps one row per key. One million events become one thousand ETS rows, not one million message structs. Your processor then receives one entry per key.

So the reason to put Tidefall downstream is the memory shape and the write semantics. It is not the batching.

You also get two things a batcher does not give you. The first is version-aware writes. put_newer/4 replaces a value only when the new version is higher, so a late event does not overwrite fresher data. The second is a hit counter. Each entry reports how many times its key was updated while it waited in the buffer.

There is one more difference. A batcher only sees the messages inside its pipeline. A Tidefall buffer accepts writes from any process. A Broadway pipeline and a Phoenix controller can write to the same buffer, and their writes coalesce together.

Now the caveat, because it matters. This moves the acknowledgement boundary. Broadway acks the message as soon as you write it into the buffer. Tidefall gives at most once delivery. If the processor crashes, that batch is lost. You turn end-to-end at least once into at most once after that point. That is fine for analytics, for state sync, and for cache updates. It is not fine when you need the source to send the data again after a failure.

If you need durability there, use processing_batch_size: :table. Your processor receives the whole table, and it can give the table away to a durable handler before it does any risky work.

The other direction, Tidefall in front of Broadway, is possible but awkward. Broadway producers are demand-driven, so you would write a custom producer around the buffer. I would only do that for a specific reason.

The blog post compares the two in more detail: https://medium.com/erlang-battleground/buffering-high-throughput-writes-in-elixir-introducing-tidefall-87e4d9bd6079

I hope that answers both questions. Please tell me if anything is still unclear, or if you want more detail on either one. Thanks!

Thanks for sharing it. I was looking for something like this to solve specific problems:

Triggering tasks to write to the database on every single authenticated API. These requests poses a significant performance and reliability risk. Under heavy load, this will rapidly exhaust the Ecto database connection pool.

I was thinking to create something like UsageTracker that accumulates this in batches and eventually dispatches the batch to the database and this library looks like a good fit for that.

Thanks

Triggering tasks to write to the database on every single authenticated API. These requests poses a significant performance and reliability risk. Under heavy load, this will rapidly exhaust the Ecto database connection pool.

I was thinking to create something like UsageTracker that accumulates this in batches and eventually dispatches the batch to the database and this library looks like a good fit for that.

Yeah, that is a perfect use case for Tidefall !!