A bank account comes to mind for me. I think it’s much safer to replay all of history than to continually add to and subtract from a single “balance” entry. And of course simpler to find out what accounts looked like at exact points in the past.
2 Likes
I agree it’s a good idea to store the history in such a context (in fact it’s probably a requirement), but I don’t see how it’s safer to materialize the balances at query time. If your database is functioning properly then you should be able to transactionally update the history and balance without issues. If your database is corrupting your data then you’re cooked regardless, it could just as easily corrupt the history!
Materializing the state from the events on every single query would be prohibitively expensive for practically any production use case, I would think. Banks in particular run huge numbers of transactions through clearing accounts IIRC, there is just no way you could actually do this in practice.
Tigerbeetle for example (which is specifically designed to do this one exact thing) stores transfer history by default, but I’m fairly sure they also keep track of the balance for all accounts at the same time. They ensure the data is not corrupted by being really careful about properly checksumming everything all the time, which if you read their design documentation you will realize is actually incredibly difficult (ssd manufacturers and kernel developers hate us).
But again, if your database is not providing correctness guarantees there is no winning anyway!
Yeah, that makes sense to me, but my understanding is that’s just an audit log, and that event sourcing is doing “something else” (where I don’t understand what “something else” is).
1 Like
That’s where the aforementioned snapshotting comes in, so I don’t think you’d be constantly recreating from all of time, but I don’t have enough production experience with this stuff.
1 Like
Mhm, but that’s what throws me off: what is the difference between snapshotting on every event and just, like, not using “event sourcing”? I assume the answer is “the programming model” or something, or perhaps it’s just a blurry distinction in practice. I am curious what a production use-case looks like.
2 Likes
Supply side its not only replaying but flexibility of event versioning & for demand side biz units can produce non-disruptive independent projections: https://www.youtube.com/live/aYMKE3B5PFM (not affiliated)
1 Like
I can only speak in hypotheticals at this point as it’s been too long. I started working on a project around 10 years ago using event sourcing but wasn’t on it very long. I watched talks by Greg Young (who also happens to be an Erlang guy, I think?) which helped. I’m sure there is a big difference in terms of guarantees and ease-of-use when postgres vs using a “proper” event database like Event Store (which I just learned is now called KurrentDB).
1 Like
A bank should use double entry bookkeeping 
4 Likes
This silently assumes you are using the same database for keeping the events and keeping the projections (a bit different thing than snapshots IMO). This is great if true, but also often not true. In that case the guarantees are much more loose.
But even in one database with transactions, errors will happen. Not because the database is buggy, but because of developers mistakes. The question then is: what do you trust? If your answer is:
- “Ok, we are probably missing some events in the history. Let’s ignore that or create an equalizing event.” - then you probably don’t have ES, bud CDC
- “The state is wrong. Let’s dump it and recreate anew from the events.” - you might have event sourcing.
Some may dismiss it as a “programming model” or a “implementation detail”, but for me it’s a fundamental shift on how you think about modeling the data. Is it a series of changes over time? Or rather a set of resources in some state? In that regard I agree with @vasspilka (even though I don’t necessarily agree with the need to couple ES with DDD).
I don’t have a lot of experience with snapshotting, but I think you rarely create snapshots on every event. Rather once per some period, liek 24 hours, or every 100 events. Other people swear by organizing event in streams, where the number of events per stream is relatively low, so you can always just calculate the state from events (withing a given stream - and you should not need to do that outside of the stream).
1 Like
We use commanded where I work very heavily, but it’s too specific.
It has gotten me thinking about applying it to a problem of one of my clients in healthcare.
The use case would be storing potentially billable events as they happen. In this case it’s call-minutes of health coaches, and device readings of patients. The rules for what’s billable by the healthcare provider are extremely complex and depend on the individual events as well as a fair deal of “datemath”.
If the developer misunderstood the datemath, or used the wrong number of required device readings, or there were some other weird thing that’s been going wrong for months, with Event Sourcing you can make a business decision for how to solve it.
In the ideal world where no one has been charged money, you can decide to replay the events through some processor (aggregate), and truncate/drop and recreate/“project” your tables.
You don’t generally get that lucky though. If the bug was around for multiple billing cycles and people have to be reimbursed, and old versions of invoices kept for legal reasons, then you might have other approaches you have to take. In this client’s case they don’t use ES but if they had been a recent bug would have been easier to catch earlier, and orders of magnitude to answer “what should have happened”, but since it spanned a billing cycle where people paid money you would still have to issue balance corrections in the form of new events. You would just do it with a one-off processor to issue new events when the bug would have happened, or make them manually.
Idk, it’s all hard to think about but that’s the point. If the problem is hard to think about when it’s going right, the kind of tools you would want to be able to fix it when it’s going wrong and you don’t know how long usually end up looking like Event Sourcing. Recovering from an error is a side benefit.
4 Likes
Triple Entry Accounting ftw 
Some disclaimers before my post:
- I haven’t read all of the above in detail
- I don’t really know Ash expect for that there is so much noise and movement around it that I almost have FOMO for not trying it out
- I consider myself a DDD expert - it’s my job
Having that out of the way, I’d like to make 2 additions:
Firstly, I agree that event sourcing doesn’t say anything about storing state. The only requirement is that you can make new decisions based on the history of what happened. Technically this means that you can rebuild your state from the history of events.
So as long as you store your state and events in the same transaction this is fine if you take care of the following: making sure your events are complete.
If you first build your state and then build the event, you have risk that you forgot to put some attributes on the events, meaning you wouldn’t be able to reconstruct the state accurately.
That’s why in event sourcing we most often have the pattern where we apply the events in order to build the state. That guarantees that our events are complete.
Secondly, about DDD and event sourcing. Sure they are often used together, but they don’t require each other. You can do DDD without event sourcing and the other way around as well.
I would like to add that the value of event sourcing of indeed comes from the fact if your business has any value in storing what actually happened.
A trivial example, is the difference of AddressUpdated vs PersonRelocated relevant for you?
Or StockUpdated vs ItemScannedIn and StockManuallyCorrected?
If you (or your business - hopefully you are aligned on this 
) don’t care about this difference, then event sourcing will probably not add a lot of value and not be the best choice.
In my opinion though I’ve noticed a lot of complex domains where these kinds of things are very relevant and offer great value to the teams building this. But of course there are downsides as well, so do your own research and understand why building something event sourcing would be valuable for your software.
(Last small remark - you don’t need to apply event sourcing to your full architecture/product, maybe it’s only useful for some complex parts, and the other parts are fine with CRUD/…)
This turned out a bit longer than I had in mind, hopefully it’s useful for someone!
9 Likes
It’s true that Ash does not enforce you to a specific architecture, one could definitely use Ash along with CQRS by for example have the Ash resources being constructed through the projections (effectively the “read-only” state that must only be updated as a result of an event being handled).
But the main point here is that using ES is a shift in mindset on how we approach solving business problems with software. My argument is that if one uses Ash as the main library to drive their modeling decisions they will likely be limited in the modeling they will end up doing. As @yordisprieto mentioned there are some core concepts of ES that in Ash exist through resources. For example one could easily make a projection as a standalone GenServer or whatnot, but they would have to insert the events as actions that belong to a Resource, so you already have to have a Resource to emit an event. It makes Resources as the core concept instead of Events.
The question I would ask is can you emit an event without needing a specific resource? If yes you can model ES effectively, if not you are constrained by resource oriented architecture (in effect even though Ash itself does not enforce an architecture, developers can build their architecture around Ash resources).
That being said, I think you can have ES and Ash separately in the same application (ES managing Events and Ash being the Application state) , Ash is indeed a really good abstraction around modeling resource oriented state, and at the end of the day even in ES software the “read-only” (updated only as a result of an event) state is usually resource oriented.
Finally something that is often misunderstood is that ES will be the dominant architecture in an application. That’s not necessary or effective, usually you only want to use ES on a subset of the core domain problems and most of your application can just use whatever other architecture is most convenient to you.
1 Like
Hey,
So as per my earlier post I think the main advantage of ES (when coupled with DDD) is the shift in how software is designed and modeled.
I like to compare it a bit to the shift of mindset in functional immutable programming and OOP (though it’s in fact it’s a quite different comparison), the former can handle increasing complexity on a smoother curve than the latter. Let’s see why.
When you are handling state in a traditional application you really don’t know how/why that state got there, for all you know someone might have went to the production database and updated a couple of rows of data. In other words the application state is itself “The source of truth” but the problem with that you don’t know how it got to that state, ofcourse you can have audit logs, transactional tables etc, but at the end most applications will not store all data changes as some audit log. This results in a situation where you can’t be sure why a specific applications state has the value it has.
ES flips that around, it sais the “audit log” or better yet event log, is the source of truth, it’s an immutable append only database and the application state is derived from that log (not all application state must be ES). This offers the advantages that now you know why a specific state is what it is, and if not you can fix it. Also you can decide to construct a new state or update existing state based on that event log.
Let’s see in practice how can that work. You are a fintech company, your core domain is dealing with financial transactions, this may be double entry accounting, but also higher level actions like, “Account opened/closed”, “Contract Signed”, “Invoice Generated”, etc… Now all these actions might have multiple sideeffects on your end state, like for example “Invoice Generated” might create credit/debit pairs and create add an invoice row, “Account Closed” might delete some rows from your database and update the customer etc. With ES not only you can reason about why your state is in the state it is. But also create new state from historical data. For example you might decide that you can to create a Loan Tape (a dataset that contains information about a pool of loans) for everything that happened since the beggining of 2023. With ES you simply write a new projector tell it when to start and handle all the events since then to create your new state. This can be for any type of data, you could build an XSL or calculate metrics for a presentation, maybe you want to know what’s the percentage of people that have closed their account within 2 months that a contract was signed, and you want to calculate that for data going back 3 years. That is the power of ES.
It’s not recreating your existing state, but creating new state that you don’t even have.
But as previously mentioned, you need the business to support you in modeling the software this way, it does not work just as an engineering solution, you need the business to actively work on modeling the events and how they contrsuct the application state.
6 Likes
As to why one might want to replay up to a particular state.
I’ve never personally needed this. But imagine that something really critical happened at one specific point in time, your biggest customer opened your app, clicked on something and that caused a bug where all their balances now show 0.
It might be some very elusive bug, where you might want to recreate all state up to that event locally and try to debug the application to understand why it happened.
That being said in any ES application there are other factors as internet connection, physical equipment, non-ES state and human intervention that could have contributed. But you can verify wether your ES application code has a problem or not.
1 Like
To be clear: this is like saying “I think you can have ES and Phoenix Contexts separately in the same application”. You can model any external or internal paradigm with Ash resources. This is not obvious on the tin, I understand that, but I do want to make it clear. You could use Ash and commanded side by side if you wanted. Ash actions that create events etc.
Ash does not even require that you ever use it to map to a storage layer. This is a valid resource:
defmodule Something do
use Ash.Resource
actions do
action :say_hello, :string do
{:ok, "Hello"}
end
end
end
That Ash is about state management or fits in as an ORM primarily is generally the most common misconception about Ash.
6 Likes
Oww, yeah you are absolutely right!
I made the mistake to assume that an Ash resource would be coupled with a database table while that’s not the case.
Thank’s for spotting it and correcting me!
I suppose in this case GitHub - ash-project/ash_events: An event-architecture extension for Ash. could grow to support ES sourcing fully
Ash never fails to surprise even after using it for months now <3
Thanks again for seeing through me
4 Likes
the business to actively work on modeling the events and how they contrsuct the application state .
A couple things came to mind:
- I learned from some CQRS/ES people that a “stream aggregate rebuilt state” is “internal” to the domain, and only used really to make business decisions (domain logic/policy) given an issued command… not visible to consumers.
- And that it’s the “projections” that provide a perspective VIEW of the “domain state”; aka currently understanding about an aspect of what’s important the business. Same event stream can provide different projections/views that are tailored to the needs of the user (e.g. a Sales analyst role, or a marketing role, or warehouse manager role)
I’m wondering in your experience, the extent and truth of this?
Also, I am reminded of the Greg Young quote, “You can’t rollback a truck already driving down the highway 1000 miles away” (paraphrased).
1 Like
Thanks to everyone for the replies, these were exactly the sort of responses I had hoped for
I had not considered the value of being able to construct future views of past data in that way. I totally see how that could be useful for “enterprisey” businesses with many teams where you don’t know exactly what you will need to do with the data down the road. Storing a full log, replaying, etc all make sense in that context.
I will reply separately to any bits which I am confused or skeptical about. Hopefully I have not derailed this thread too much - it was looking kinda derailed when I posted so I wasn’t feeling so bad about it, but I got more discussion here than I had bargained for. Maybe it will have to be split off
1 Like
A fair point, though I am personally very skeptical of using multiple databases in this way in general.
I think the same problem applies here, though: a developer could make a mistake on the projection, but they could also make a mistake on the events too. What if the projection was actually correct and the events were bugged? How would you know?
There is probably an argument here that events have more information and thus it is easier to spot such issues, which I would be sympathetic to.
Oh I did not mean “programming model” to be dismissive, in fact what I actually meant was essentially what you describe - a different way to think about things.
The point I was trying to make was that snapshotting in the limit would just be “using the database normally” plus “keeping an audit log”. That is to say: if you are already doing those things, event sourcing is just “those things but worse”, where by “worse” I mean the materialized view is not always up-to-date or has to be reconciled (expensively) at query time.
I do have a better idea of how ES could be useful in other ways, though, which I did not yet have when I wrote that message
1 Like
I see the value here, but this reminded me of something else which was bugging me.
What if, for compliance reasons or whatever, you need to ensure something is removed from your system? If you merely store a tombstone event then of course nothing has actually been removed, which could create all sorts of problems.
I don’t think this is a niche issue, it’s something that affects essentially any app or product. I care a lot about privacy, and it’s very important to me that my systems are designed to promptly remove anything which a user chooses to permanently delete. How would I use event sourcing in cases like these?
I see how a mental model of “keeping track of historical state” can be useful. As a real example, I just finished work on a note system to complement my bookmarking system and it keeps track of revisions - like git, but without the merkle tree. But if a user were to delete a particular revision for whatever reason, I would want to ensure that it is promptly removed. I don’t see how I could implement this with ES.
Of course it could be the case that my use case is just not a great fit, but surely even in these sorts of enterprise settings there is a need to actually remove data sometimes. How is this handled?
2 Likes
