I’d be interested to hear more about this, especially as I’m not far from you in Winchester. We should meet for a coffee to chat solutions. Did you consider using Commanded & EventStore before building your own solution?
slashdotdash
dokie
The main reason I think was that we built it in 2015 before I think you started your work.
Sure it would be good to meetup if you fancy coming over to the science park.
Mike
0
jdumont
I’m still chipping away at this whenever I get a moment, and thought that I’d provide a quick update on where I’m at.
I tried using many different options for storing events, all using a shared interface so that I could quickly switch between them. I’ve found some wonderful little features of both Elixir and Erlang and some cool libraries to help me along the way — it really has been a brilliant learning experience!
So far I’ve tried:
- Simple
term_to_binaryandFile.open(x [:binary, :append]) - DETS
- Erlang’s
disk_log - Erlang’s
file:consultfor reading events back and a customio_lib:formatfunction to write them - CubDB (using
min_keyandmax_keyfor event number ranges was brilliant)
All had positives and negatives with disk_log probably coming out on top. It has a load of features that you really need for working with files already baked in and thoroughly tested. The deal breaker though was that it’s really hard to get events back out by their event number, as disk_log really doesn’t work with any keys, it literally just appends a given term to the end of the log.
Getting the events out required bringing all events into memory (ignoring disk_logs wonderful chunking feature) and filtering out events older than those we were interested in. This isn’t the worst thing in the world as I was planning on keeping logs partitioned by aggregate, meaning that they are unlikely to ever become so long that parsing the whole log becomes an issue as it’s incredibly fast - loading 100_000 average-sized events in approximately 75ms.
My concern was that I could end up with a compromised method for reading and writing the literal heart of this system.
—
The next phase in this experiment was yet more learning… Martin Kleppmann’s “Designing Data Intensive Applications”. I can honestly say that I learnt more reading this in a week than I did over the previous 9 months piecing together articles and documentation online. His closing conclusion of “turning the database inside out” is very in line with what I’m trying to accomplish here, although I definitely think he was talking at a much larger scale than I’m working at! 
Importantly, the book highlighted many issues that I was unwittingly making with other parts of my system. I was neglecting a total order by writing all of my events in partitions based on aggregates. It wasn’t a deal-breaker though as I could accept partial order as a trade-off in this instance as causality isn’t an issue in my domain. However, these partitions did make my projections quite complex due to needing to track their event offsets (using consumer offsets over ack) for many, many, many streams. Not an issue but one that I decided that for now at least to avoid by opting for a single unified log with total order.
I’m using a very simple writer and index pattern from the book and at the moment it’s working very well. Sending a command to an aggregate, having it validated, events created, persisted and applied to the aggregate is taking on average around 150 micro-seconds. I still have to handle log rotation and snapshots, but these should be quite simple as I’m modelling my store as a series of GenStage (yet another thing I’ve learnt since starting this).
—
I’ve decided in many cases to opt for purposely naive approaches - whereas before they were just naive 
- in order to keep my system simple. My tests have shown that I’ve got a good amount of performance margin with which to make compromises such as the single log which is a potential bottleneck, but hugely simplifies the entire system. Equally, a single event stream, local-only Registry makes it a lot easier to work with and understand. I figure that it’s easier to make a system that already works well faster, than it is fix a fast system thats yet to work.
I have had crisis’ of confidence along the way — “Why aren’t you just using Commanded and PostgreSQL like a sensible person?” - “You’re totally out of your depth here!” - “Why even bother with event sourcing, CRUD could work here after all” — but overall I’m very happy with the progress I’ve made and the things I’ve learnt. I think I’m probably a little way off being able to build anything complex with it yet, but I’m enjoying the process.
6
slashdotdash
Sounds like you’re having fun. Martin’s book is a fantastic reference for building these types of systems.
Once you have a working single-node system you can experiment with making it distributed for scalability and fault-tolerence. It looks like disk_log supports distribution, but likely not useful because it doesn’t guarantee writes go to all nodes:
A collection of open disk logs with the same name running on different nodes is said to be a distributed disk log if requests made to any of the logs are automatically made to the other logs as well.
It is not guaranteed that all log files of a distributed disk log contain the same log items. No attempt is made to synchronize the contents of the files.
1
jdumont
It’s been so much fun, albeit frustrating and head scratching at times. As you say, Martins book almost feels written with this type of system as the ultimate goal.
It’s so tempting to build on top of disk_log because it’s already there and does so much of what I want to achieve, but the hurdles I think are too great. The querying of events (or lack thereof) will become a real issue with a single event stream, and as you point out, the built in distribution isn’t all that helpful in this case.
Distribution and replication whilst maintaining consensus on event ordering is where I’ve spent a lot of time re-reading sections of the book. Unfortunately it looks like single leader replication of the log is the only really viable way to approach the problem, and therefore not hugely useful as I’ve still got a single node as a bottleneck.
The only way to distribute the log (either locally or across machines) would be to go the same route as Kafka and break my event streams into topics and partitions (as I was originally doing) which brings a lot of additional complexity with it, both in persisting events and making them available to the rest of the application.
As my use case is smaller, simpler applications I think that restricting myself to a single node (at least for the event store) is not unreasonable. I’m opening a can of worms and losing sight of the objective by trying to solve problems with distributed logs and total ordering, so for now I’m keeping my solution as simple as possible.
As both file and database backed logs run into the same restrictions around distribution, the only real question that remains is which is better in terms of scale and throughput on a single node. I suspect that’ll be a DB like Postgres, unless the lack of encoding/decoding of terms starts to play a larger role in performance. We’ll see…
1
slashdotdash
If you make the event store single node then you could expose an API that other parts of your application can use. Treat it like a database which as many nodes as required can connect to. Event Store exposes a “native interface of AtomPub over HTTP” as an example.
0
chouzar
Thanks for the fantastic thread. A week ago I stumbled upon the event sourcing concept and I’m learning quite a bit thanks to libraries like Fable and EventStore. But obviously I probably still not get the nuances of where, when and how to use this pattern correctly.
What I don’t necessarily like with the approach is that my logic is somehow also bound to event history; if something changes in the way data is being stored I now need to either:
- Keep the old and new data-structures as they were stored and just add the necessary application logic to handle multiple representations.
- Migrate stored data and reducers as necessary to keep the system tidy.
The Fable example from @benwilson512 gave me a cue:
¿Would storing the event along its transformation be too crazy of an idea?
Let’s say these are the events I want to persist (this is just general elixir, no framework or library attached):
%Event{
id: "123-abc",
event_type: "Inserted",
handler: :erlang.term_to_binary(fn x -> x end),
number: 1
}
%Event{
id: "123-abc",
type: "Incremented",
handler: :erlang.term_to_binary(fn x -> x + 1 end),
number: 2
}
We could then retrieve the desired events and apply the stored transformations to any data as a starting point. Taking it a little further:
%Event{
id: "1a2-b3c",
type: "Incremented",
handlers: %{
up: :erlang.term_to_binary(fn x -> x + 1 end),
down: :erlang.term_to_binary(fn x -> x - 1 end)
}
number: 108
}
We could store handlers for both transforming data further or doing a rollback (the Sage - Sagas project docs gave me this idea). This would let us “advance” or “rollback” data in any state to another point in time.
Aside from security or possible changes on how terms are stored… Would something like this be at odds with EventSourcing? Is there a library that already does this? I’m pretty sure there’s a lot I’m missing 
0
jdumont
Hey, I’m glad that the thread has been useful. Event sourcing is a bit of a bear to get your head around initially, and for a “simple” pattern, there’s a lot of complexity hidden in implementing it well. I’ll confess that I’m still not there myself and will keep using Commanded whilst slowly tapping away at this idea.
You’re very right that storing events leaves you bound to your past mistakes or oversights. That’s pretty much the biggest cost of the pattern, if you screw up you either have to live with it or spend a lot of time managing previous versions of your events, or migrating them to the new structure. There’s some good links floating around the forum (just search “eventsourcing”) that dive into this issue further.
For this reason, it’s often recommended to use ES in a well understood domain with a lot of time spent up front working out the data your events need. It’s not a tool to pull off the shelf when you’re still finding your way through the problem.
For what it’s worth I handle the event versioning issue by being quite overzealous in the data that each event captures. You can be quite objective with event data as it’s “something that happened” so there’s only so many ways that you can interpret it. Make sure you store everything that may be relevant to this action and you should be OK. I’ve only had one instance in the last year where I needed a new version of an event, and it was because the original was missing a data point I didn’t think was required at the time. Fortunately it was easy to add in and provide my handlers a fallback value where the older values were missing it.
One thing that often gets overlooked and which tripped me up was the boundaries of my aggregates. Perhaps because I don’t have a DDD background, but where my aggregates sat in relation to my projectors caused me some grief early on.
As for storing the function within the event: personally I’m not sure it would work. It’s fine for mapping the behaviour of that event within the aggregate, but doesn’t allow for the possibly dozens of ways this event will be used by a countless number of handlers. Not to mention you lose what is wonderful about ES: data separated from logic and therefore the ability to rewrite the handlers and operate on old data as your requirements change. It’s a nice idea and I do think that “logic as data” has it’s place, I just wouldn’t use it here.
4
chouzar
Yes I’m still trying to understand the real meaning of DDD lingo (boundaries, aggregates, projectors) I’m guessing that eventually a book (or experience) will have to do.
Yeah my approach bounds an event with a “single interpretation”, but I guess that I have this irrational feel that without my application logic a series of events by itself would be meaningless. But I guess the same could be said of any generic enough abstraction.
Thank you so much for the well put thoughts 
the picture is getting clearer bit by bit.
1
Exadra37
I thinking that disabling the Linux write cache could help here:
Not all system’s belong to the same “turn-on write-back caching” recommendation group as write-back caching caries a risk of data loss in the event such as power failure etc. In the event of power failure, data residing in the hard drive’s cache do not get a chance to be stored and a lost. This fact is especially important for database system. In order to disable write-back caching set write-caching to 0:
# hdparm -W0 /dev/sda /dev/sda: setting drive write-caching to 0 (off) write-caching = 0 (off)
0