Append only DB

If I understand you correctly you want to store historical data and have a revision tree of all your changes to your “information state”. This means that you need to store the data for every revision. If I have misunderstood your requirement then you may ignore this post :smiley:

Something like this can be implemented in most databases. It is just a question which is more suitable as there are pros and cons with everything. Datomic http://datomic.com/ might be what you are after but it is commercial.

Because you are already using postgres you can keep using it. Have a table and only do INSERTs and no UPDATES (perhaps even revoke UPDATE privilege on the table). Depending on the size of your data this may be a viable approach especially if indexed properly. Remember that today 100 of GB of data is not considered particularly large.

Perhaps some useful articles:

NoSQL databases may be more suitable for this. Primarily because they use underlying append-only data structures. (most commonly a log-structured merge-tree (https://en.wikipedia.org/wiki/Log-structured_merge-tree)). The benefit of this design is that it allows for fast writes and decent enough read speeds. On the other hand they are not ACID and doesn’t support database transactions. Keeping historical revisions of data though, is still left as an exercise for the user.

If you wanted to implement your revision history in NoSQL database you can store your account number and date of update along side with the data and then do a prefix range scan for the account to get all data back.

Which NoSQL database? Who knows :D. If you only run on 1 server you can use something like RocksDB or LevelDB, both of which has got erlang bindings and use it embedded in your application. But anyone that supports range scans and prefixed keys should do.

1 Like