Maybe this article can help you understanding it better and how it fits with Event Sourcing:
Kafka is not properly a database, but can almost be used like one.
See this articles to understand better what is a data lake:
Exadra37
vans163
Event sourcing involves modeling the state changes made by applications as an immutable sequence or “log”
of events. Instead of modifying the state of the application in-place, event sourcing involves storing the event
that triggers the state change in an immutable log and modeling the state changes as responses to the events
in the log.
Ah so we get this naturally as a property of using an immutable actor model language, example updating the state of a process (gen_server), that is basically an immutable log of events via the mailbox.
Consider a Facebook-like social networking app (albeit a completely hypothetical one) that updates the
profiles database when a user updates their Facebook profile. There are several applications that need to be
notified when a user updates their profile — the search application so the user’s profile can be reindexed to be
searchable on the changed attribute; the newsfeed application so the user’s connections can find out about
the profile update; the data warehouse ETL application to load the latest profile data into the central data
warehouse that powers various analytical queries and so on.
This is a disaster waiting to happen riddled with inconsistencies if say one of the nodes goes down now a bunch of events are missed.
Okay I get the idea now. Event Sourcing is basically a event subscription to all incoming data. But why not just do an event subscription to a distributed database instead, example like Cloudflare KV or Firestore? A users profile got updated? Instead of listening for the “UPDATE” event, the database itself eventually consists across all nodes, once each node gets the update that the profile got updated, the database generates an event and fires off notifications to each subscriber (connected user). The difference is if something goes down, when it comes back up it eventually consists, there is no “event to miss”, also stuff doesn’t go haywire if events come out of order.
0
Exadra37
When using Event Sourcing you need to persist the events, and you can do it with whatever database technology of your preference. Kafka is an excellent way to do it in an intermediate stage, and once it uses the RAFT consensus algorithm you will not have a disaster waiting to happen. I used the term intermediate stage, because normally events are consumed from Kafka and persisted in other databases, like Postgres.
You can take a look to the Elixir library Commanded by @slashdotdash that does Event Sourcing/CQRS in Elixir, and by default uses Postegres:
0
slashdotdash
Event sourcing is persisting application state changes as domain specific, intention revealing events (e.g. UserRegistered). Current state is built by reducing the list or stream of events, similar to Enum.reduce/3 in Elixir. For event sourcing you ideally want to use a proper event store, such as EventStoreDB or the Elixir EventStore library I wrote which uses Postgres for persistence.
Event streaming is where events are published to interested subscribers for loosely-coupled processing, service integration, etc. Kafka is designed exactly for this. It would be more similar to Elixir’s Registry module when used for pub-sub or using Stream.each/2 or the Broadway library. Often consumers use ack/nack to guarantee at-least-once message delivery so that they cannot miss any events. They may store their current processing position to a durable store so they can stop and resume processing safely. It’s possible to use change data capture to publish updates from a data store, such as table UPDATEs but unlike event sourcing these changes are not domain specific, but instead are general insert/update/delete operations.
There are two varieties of events used with event streaming: Event Notification and Event-Carried State Transfer (ECST). Event notification is used to indicate when something in an application has occured, such as a user registration, but the event contains minimal information (“thin” events). ECST is where more information about the current state is included in the event (“fat” events). You can also combine both event sourcing and event streaming by publishing domain events used for event sourcing to consumers for event streaming.
See What do you mean by “Event-Driven”? by Martin Fowler.
2