Pearls (/posts!) of Wisdom Thread!

This thread was actually inspired by recent posts by Robert and Joe :003:

And here’s an older post by Joe that is worth sharing again:

Databases are basically rectangular tables of cells, where the cells contain very simple types like strings and integers - every time you access a row of an external database this list of cells has to be converted to beam internal data structures - this conversion is extremely expensive.

The best way to persist data is in a process - then no conversion is needed, but this is not fault tolerant - so you need to keep a trail of updates to the data and store this on disk.

Often you don’t need a database for example you might like to have a system where you store all the user data as in the file system with “one file per user” this will scale very nicely - just move the files to a new machine if you need more capacity.

Erlang has two primitives term_to_binary and the inverse binary_to_term that serialise any term and reconstruct it - so storing complex terms on disk is really easy.

I have mixed feelings about databases, they are great for aggregate operations (for example, find all users that have these attributes) but terrible for operations on individual users (where a single file per user is far better).

If I were designing a new system I’d go for ‘one file per user’ as much as possible and try to limit databases for operations over all users.

From: Acceptance of Erlang's 'term_to_binary' and vice versa in Elixir

3 Likes