EventStoreDB is an append-only stream database designed for Event Sourcing. EventStoreDB champions immutable data streams and real-time subscriptions, making it easy to build reactive eventually consistent systems with built-in audit logs of how data has changed. Version 20 deprecated the TCP+protobuf client-server interface and added a new gRPC+protobuf interface (among other improvements).
Spear (hex) (github) is a gRPC client library for EventStoreDB 20+. It provides a familiar t:Enumerable.t/0
interface for interacting with the EventStoreDB with functions from Enum
and Stream
.
For example, you can stream a large CSV into events and the enumeration won’t be run until each event goes over-the-wire:
{:ok, conn} = Spear.Connection.start_link(connection_string: "esdb://localhost:2113")
# e.g. a MyCsvParser module using c:NimbleCSV.parse_stream/1
File.stream!("large.csv")
|> MyCsvParser.parse_stream()
|> Stream.map(&MyCsvParser.turn_csv_line_into_spear_event/1)
|> Spear.append(conn, "ChargesFromCsvs", timeout: 15_000)
# => :ok
You can also read EventStoreDB streams lazily via Elixir Stream
s:
# say we have 125 events in this EventStoreDB stream
iex> Spear.stream!(conn, "SomeLongStream", chunk_size: 25)
#Stream<[
enum: #Function<62.80860365/2 in Stream.unfold/2>,
funs: [#Function<48.80860365/1 in Stream.map/2>]
]>
# the returned stream will fetch events in chunks of 25
iex> |> Enum.count
125
And work with subscriptions (regular and persistent) through asynchronous message passing.
iex> Spear.subscribe(conn, self(), "MyStream")
{:ok, #Reference<0.1469255564.3441164290.87228>}
iex> flush
%Spear.Event{..}
%Spear.Event{..}
%Spear.Event{..}
%Spear.Event{..}
..
:ok
Or if you’re looking for something a bit more fancy for real-time event processing with back-pressure, there are GenStage and Broadway producers in Volley (hex) for regular and persistent subscriptions, respectively.
Even if you’re not in the market for a new EventStoreDB client, Spear still has something to offer: it’s an example of a gRPC client written with just Mint (hex)! gRPC is a pretty slim specification which extends HTTP2 to add a message format and some headers. It’s not such a heavy lift to implement a gRPC client given a nice HTTP2 client like Mint. Plus Mint’s fine-grained control over each connection makes it possible to add features like efficient request streams which respect HTTP2 window size back-pressure (via t:Enumerable.continuation/0
s) and be certain about blocking and multiplexing.
Give Spear and Volley a try and see what you think! Issues and PRs are always welcome