Tidefall - An ETS buffer for high-throughput writes and batch processing

Hello,

I am happy to announce Tidefall.

Tidefall is an ETS buffer for high-throughput writes and batch processing. You write data into it as fast as you must. The buffer holds the data in partitioned ETS tables. At a fixed interval, it sends each batch to a processor function that you supply.

There are two buffer types. Tidefall.Queue keeps the insertion order. Tidefall.HashMap collapses writes that have the same key, so only the most recent value stays.

Tidefall gives at-most-once delivery. If your processor crashes, the buffer loses that batch. This is a deliberate choice, not a gap. On a normal shutdown, the partition traps exits and drains what remains. If you need a stronger guarantee, the processor can take ownership of the table and hand it to a durable handler.

Some of you know partitioned_buffer. I wrote it with the team at Appcues, and Appcues released it under the MIT license.

We ran it in production for months. It buffered writes to a message broker, and it collapsed duplicate events by key before expensive downstream work. Tidefall keeps that same engine, so the core is not new code.

Tidefall is a fork of that library. It is not a replacement. partitioned_buffer is alive, and Appcues maintains it. Tidefall has a different scope and a different roadmap.

The story behind the fork

I wrote a blog post about it. It is the best next read if this library interests you.

The post covers the three use cases that shaped the design. Each one added a capability: batch writes to a message broker, then coalescing by key, then version-aware updates.

Blog Post: Buffering High-Throughput Writes in Elixir: Introducing Tidefall

Thank you to the Appcues team for the original library, and to the OpenTelemetry batch processor for the initial inspiration.

Feedback and issues are welcome.

Links:

• Hex: tidefall | Hex
• Docs: Tidefall v1.0.0 — Documentation