Hobbes - a scalable, fault-tolerant transactional record store

[Exadra37]

@garrison

Hobbes is a scalable, fault-tolerant transactional record store written in Elixir.

If you are building it in pure Elixir then you may need to account for the delayed_write with a default of 2 seconds, that may work against or favourable to you: file — kernel v10.4.1

delayed_write - The same as {delayed_write, Size, Delay} with reasonable default values for Size and Delay (roughly some 64 KB, 2 seconds).

I discovered this while investigating why Mnesia lost all data after restarting my laptop and also found later an issue with an EventLog Elixir library that lost all records inserted to the file during the default two seconds window, when closing a file: Example in README cannot read the last entry in the append only log · Issue #1 · pbudzik/eventlog · GitHub

I also talk about this delayed write issue in this forum post:

Unexpected behaviour with mnesia - not actually durable?

The issue It’s not related with any blocking in Erlang per se… As I mentioned in the post I linked previously, it’s because the BEAM has a default of 2 seconds to write data to the disk when the file is open with the defaults for delayed_write, as per docs : delayed_write The same as {delayed_write, Size, Delay} with reasonable default values for Size and Delay (roughly some 64 KB, 2 seconds). You can confirm this with a script that writes directly to the file system: defmodule FileIO do…

SIde Note: You may want to read the Kafka design decisions about using a filesystem approach to take advantage of all the low level stuff that Linux has to offer, which may help you with making some decisions on how you use the filesystem from the BEAM. Another interesting option you may want to consider is to disable Linux write cache as I mention in this post: Opinion on file & memory based event sourcing system - #21 by Exadra37 .