Reading the Kafka design decisions about going with a filesystem approach while taking advantage of all the low level stuff that Linux as to offer may help you in making some good decisions.
Exadra37
jdumont
Thanks for the info and links! I’ve tabled this project whilst I work on other things — bills to pay and all that… — but the idea still keeps nagging at me. I had intended to look at the guts of systems like Kafka and EventStore properly when I revisit this.
Do you have any insight on how the Erlang VM might impact this approach? When looking at this before it seems that any access to the low level OS stuff outside the VM is more limited than with other systems, but that may just be my unfamiliarity with the guts of the VM.
0
Exadra37
This approach of using files to persist the data need to take in consideration that the Erlang library disk_log by default as a delay of 2 seconds or 64kb to write to the disk as I say here:
This setting can be tuned, but care needs to be taken to find the right balance in terms of disk IO performance, otherwise you may create a bottleneck when writing to the disk.
I am not that familiar to, but @dimitarvp may have something to say here.
From what I remember when reading the Kafka design decisions you don’t need to interact with the OS from the BEAM, you just need to tune the machine for disk IO as they recommend.
0
Exadra37
jdumont
I hadn’t — I’ve actually changed careers and work as a photographer now — but funnily enough started poking around my terminal again today and seeing what’s new in Elixir as I’ve a few ideas I want to try (on my own time). Event sourcing is still one of those things, but I’ll need to dig into it again and see whether this approach has merit versus a more conventional ES setup using Postgres.
1