Opinion on file & memory based event sourcing system

[Exadra37]

@jdumont

Do you have any insight on how the Erlang VM might impact this approach?

This approach of using files to persist the data need to take in consideration that the Erlang library disk_log by default as a delay of 2 seconds or 64kb to write to the disk as I say here:

Mnesia with Memento lost all records for all tables

The BEAM indeed has a default of 2 seconds to write data to the disk when the file is open with the defaults for delayed_write, as per docs : delayed_write The same as {delayed_write, Size, Delay} with reasonable default values for Size and Delay (roughly some 64 KB, 2 seconds). Now I just need to do some tests with a simple Elixir script and see if this “insane” default can be optimized with {delayed_write, Size, Delay} from 2 seconds to something like 2 or 20 ms, because 2 seconds is an …

This setting can be tuned, but care needs to be taken to find the right balance in terms of disk IO performance, otherwise you may create a bottleneck when writing to the disk.

@jdumont

When looking at this before it seems that any access to the low level OS stuff outside the VM is more limited than with other systems, but that may just be my unfamiliarity with the guts of the VM.

I am not that familiar to, but @dimitarvp may have something to say here.

From what I remember when reading the Kafka design decisions you don’t need to interact with the OS from the BEAM, you just need to tune the machine for disk IO as they recommend.