CubDB, a pure-Elixir embedded key-value database

Do you have feedback about the API?

I think using function/1 or mfa for querying data would be more flexible instead of own DSL that just redirects some functions to Stream

Did you compare with pure Erlang k/v stores?

I think the most recent is GitHub - martinsumner/leveled: A pure Erlang Key/Value store - based on a LSM-tree, optimised for HEAD requests · GitHub

But there is at least also GitHub - basho/bitcask: because you need another a key/value storage engine · GitHub and GitHub - refuge/cowdb: Pure Key/Value database library for Erlang Applications · GitHub – maybe others as well.

Nice, I didn’t know those projects!

At a glance, CubDB works very similarly to CowDB and bitcask in that they all use an append-only copy-on-write data structure. I will look into them and hopefully learn more about their strategy to deal with the tricky bits, like compaction.

The main difference would be, I guess, that CubDB is written in Elixir (not implying that it’s a huge advantage, as it’s trivial to use Erlang libraries from Elixir).

Nice. I’ve been looking for sqlite alternative.

• Do you plan to introduce a query API?
• Do you plan to make a strict data typed variant, more akin to Postgres?
• Would you consider adding a command to it that compacts it (and thus destroy the history of changes but oh well)?

Good point.

I initially wanted select/3 to just return a lazy Enumerable that users could use their own Stream functions on. The reason why I use that specific DSL in select/3 is that the lazy Enumerable references a specific point in the DB data file, and I need to know when no more reader references it, so a compaction operation can safely “garbage collect”. By “proxying” the stream operations, I know when they have completed, and can “check out” the reader.

Maybe there’s a better way though. I could possibly just accept a single arbitrary reduce function (that can do all that map, filter, take, … can do), but I felt it’s easier to operate with separate pipeline functions.

Hi @dimitarvp , thanks for your kind words!

The compact/1 function is already available. Even more conveniently, you can opt-in to auto-compaction either at startup or later with set_auto_compact/2.

Regarding the “strict” version, I didn’t plan it yet, but why not It should be reasonably easy to implement it as a layer on top of CubDB. I was kinda thinking of building an SQL layer as a separate library, mostly for fun, but have no idea when I will actually get to it.

About the query API, at the moment select/3 is the query workhorse. It supports efficiently selecting ranges of keys, filtering results, mapping, iterating, reducing, etc. That said, as @fuelen commented, it might be possible to improve its API to make it even simpler to use.

@lucaong - very nice!

With select/3 you can specify a key-range (min-key, max-key). It looks like a key can be any Erlang term. (is that right?) How would min/max ranges work with different key types (numbers, strings, lists, tuples, …) ??

Is there any way to use pattern-matching to select keys?

Let me give you some food for thought. Don’t take it as a wish list, I am just sharing.

There is currently a market for sqlite-like storage engines. Sqlite3 is an amazing little DB but it comes with quite a hefty load of legacy decisions – like a lack of proper timestamp type, lack of boolean type, lack of enums, accepting arbitrarily typed data in integer columns etc.

A lot of people out there use Sqlite3. It’s deployed on trillions of devices, literally. Yet it lacks some very common sense features like strict typing.

IMO the BEAM VM (and thus Erlang, Elixir, LFE, Alpaca etc.) is uniquely positioned. We basically don’t need Redis and Memcached due to ETS, DETS, Mnesia, Erlang’s persistent_term and several others. If we complement that with a self-sufficient single-file storage engine then the BEAM ecosystem becomes a de facto standard for a lot of development scenarios.

Again, don’t take this as a list of demands. It’s just my opinion that the BEAM ecosystem seriously needs a good Sqlite3-like experience.

Thanks @AndyL!

Yes, keys (and values) can be any Elixir or Erlang term. One neat thing about Erlang and Elixir is that order of arbitrary terms is well defined. Try for example :a > {1, “something”}

Even nicer, the ordering actually makes sense for tuples, because elements are compared in order. That can be used to good effects in CubDB. Imagine you want to store different “tables” in the same database. You could structure your keys as {:table_name, id} and, if you want to select only entries in the :users table, use select/3 with min_key: {:users, 0}, max_key: {{:usert, 0}, :excluded}. Because :usert is the lexical successor of :users, that would select all entries in the user table, and no other entry.

That, together with serialization of arbitrary terms, is a great thing that the Erlang VM offers and that CubDB leverages

OMG didn’t know that!

Any way to pattern match on keys? Or to fetch a [list] of keys?