Crash/recovery oriented on-disk data structures

Chat / Discussions
#1

  1. Suppose we are on linux x86_64.

  2. Suppose, in the middle of a file system write / fsync, someone pulls the power and the machine shuts down.

  3. When the machine boots up, because we were in the middle of a partial write, there is not all that much guarantees on the state of the file system.

  4. Question: Does Erlang/Elixir have any libraries / data structures (besides ‘use a database engine’) for handling situations like this?

So again, the problem here, is not a single process crashing; the problem here is that the entire machine loses power in the middle of a write / fsync, the file being written to is in some inconsistent state, and we are trying to recover from this.

#2

There is disk_log which mentions in the docs:

internal format

Supports automatic repair of log files that are not properly closed…

1 Like
#3

Probably using sqlite instead of the naked FS will solve most of the potential problems. It has a lot of guards against corrupted state.

#4

This is my fault for not clearly stating this in the question: I’m playing with Write-Ahead-Logs, Log-Structured-Merge trees, append-only-b-trees, and a few other data structures.

So yes, sqlite would solve this problem, but it’s not what I want because sqlite commits to a particular on “can recover from crash disk representation”. (last I checked, it defaults to write-ahead-log).

I’m interested in exploring the space of other “can recover from crash disk representation”

1 Like
#5

Always been interested in the same but life and work keep happening so never explored it in more detail.

Sorry that I can’t help further.