Looks neat. Consider enabling snappy or similar NIF on each term to gain performance and compression. CowDB is derived from CouchDB and the test suite may be relevant - lots of corner cases to uncover.
dch
lucaong
thanks @dch !
Yes, compression would be an interesting thing to add, especially for large values. I will add it to the upcoming features to look into. I would try to keep it Elixir-only though, to make it very convenient to use in embedded scenarios.
You are right about testing and corner cases. I have a suite of property-based tests for the B-tree data structure part that proved very valuable in order to discover and fix bugs during the early phases of development.
The append-only, copy-on-write, paginated B-tree data structure is inspired by CouchDB, but CubDB is not using nor depending on any CouchDB code, so I cannot simply apply its test suite. It’s still interesting to look for specific conditions tested there though.
1
lucaong
Regarding compression, I actually just pushed version 0.6.0, that performs compression on terms before writing on disk. That’s actually really easy thanks to the :compressed Erlang option on term_to_binary/2.
1
elcritch
I find the datalog “recursive predicate” querying interesting, and especially with the time component, but honestly I’ve only dabbled with it.
My primary interest in Datomic is how columns are defined and can be combined anyway into arbitrary entities. Using pure KV stores makes it hard to verify fields, but SQL doesn’t fit a lot of time series and IoT use cases in my experience.
A Datomic query engine would require a subset of a datalog interpreter on top of the KV store and various indexes. Though I think given the nature of Elixir coming from Erlang the datalog part shouldn’t be impossible to do. I’ve been tempted to start it a few times… but I just can’t spare the time.
2
asianfilm
I’m using version 0.4, and it’s proving much more reliable than a previous solution I was employing.
The documentation could be improved a bit; ideally with an example application in a subdirectory. Also, when upgrading to 0.6, for example, does it upgrade the database automatically to the new compressed version?
Right now, I’m just using get, put, delete and compact and haven’t explored select, etc.
Thanks for the package!
1
lucaong
Thanks @asianfilm, I am happy that you are finding it useful too.
You are absolutely right about the documentation. So far, I mostly focused on the API reference, but I intend to add usage examples and a “Getting Started” before version 1.0.0.
I am following semantic versioning, and I do reserve the possibility to make backward incompatible changes to the file format before 1.0.0 if strictly necessary to introduce an improvement. That said, I will try to avoid that, and in case I will clearly communicate it (currently I list changes in the commit message of version bumps, but will add a changelog and GitHub releases before version 1.0.0). The improvements that I have planned so far should be possible without backward incompatible changes.
In your case, updating from 0.4 to 0.6 does not need any manual operation (non-compressed nodes are read just fine, and newly written ones will benefit from compression).
If you explore select and the auto compaction feature, I would be happy to know your feedback about them 
1
lucaong
Hi @seb5law,
nice that you are considering using CubDB!
There is no hard limit on the size of the value. That said, reads and writes will of course be slower than with small values. First, because of the higher amount of data to be written and read from disk, and second because CubDB organizes disk space in pages of 1024 bytes each, so it will have to add more page headers. This page size is more optimized for small entries, but there should be nothing preventing bigger values from working.
Writes should still be atomic, even in case a power failure happens half-way through writing a value. What could happen in that case, is that the next time the database might start more slowly: CubDB starts reading from the end of the file and looks for the latest “good” header, so it might have to go through many pages before finding it.
In summary, I do not expect any specific problem, apart from slower writes/reads. That said, I honestly did not run benchmarks with keys as large as tens of MB. I would definitely be interested in your feedback if you try that, and I intend to add some benchmarks and run some tests to detect possible corner cases with large values.
My recommendation is to give it a cautious try in a non-critical environment: if you find issues, I will try to fix them before version 1.0.0 (or otherwise explicitly document the limits).
2
seb5law
lucaong
Take it as a sanity check more than an exhaustive test, but I just tried saving ~20mb values in a loop in CubDB on my MacBook Pro. I did not incur in specific issues. Performance wise, each write took 0.7 seconds. Reads were much faster. Compaction was also taking more or less 0.7s * N.
I will look into how much of that write performance can be shaved off, but nothing seems to be breaking unexpectedly.
0