Mnemonix: a generic key-value store adapter library [v0.8.1]

[christhekeele]

Mnemonix

After a couple years of playing with the concept and learning my way around ever-evolving idiomatic Elixir and venerable OTP patterns, I’ve finally sat down and implemented an Elixir library with an architecture I’m pleased with.

Rubyists might be familiar with Moneta; this is my Elixir approach to that. Mnemonix is a standardized Map-compatible API for interacting with key-value stores in an implementation-agnostic way.

The intent is to define an abstract API a key-value store should offer, and iron out discrepancies between implementations and Elixir wrappers such that they all comply and behave predictably behind it.

If you’re developing an application, it lets you trivially experiment with different backends to find the right implementation for your use-case, and unifies access to different backends chosen for different reasons behind a common, familiar API.

If you’re building a library, it lets you defer the implementation decision of what key-value store to support to your end-users.

---

Current State

As of v0.7.1, I’m beginning to move towards optimization and better error reporting. The public API has now stabilized and only new capabilities will change public function signatures.

Currently, it offers parity with the Map module, and builds upon that with extra functions for common key-value store needs: incrementing and decrementing integer values, and setting explicit keys to expire after so many seconds.

It supports several backends:

• Map
• ETS
• DETS
• Mnesia
• Redis
• Memcache

I intend to improve the existing backends, add a few more, and add a few more features to all stores before v1.0.0.

Check out the documentation to learn more, let me know of any issues you run into!

[Qqwy]

This a wonderful idea. I am definitely going to use this library in the near future.

[swelham]

This looks really cool

[jxxcarlson]

@christhekeele

codebase

I used Mnemonix tonight to solve a problem I had with the app I am working on. Mnemonix is a really useful and elegant tool. I put this code

{:ok, store} = Mnemonix.Store.start_link({Mnemonix.Map.Store, initial: %{active_notes: []}}, name: Cache)

in the APP_NAME.exs file and within two hours the job was done. Three cheers for Mnemonix!

[jxxcarlson]

Hi Chris, some info about my use case, of which there are two, both in Phoenix – one present and one future.

1. Lookup.

This is an app for storing short snippets of information – from things like the speed of light and Planck’s constant to links to various things I find on the web: political articles, github repositories, articles about code, you name it. Just two fields: title and description. URLs in the description field are rendered as links, e.g, http:foo.bar.io/yada/yada?baz=123 is rendered with link text “foo.ba.io”.

I use Mnemonix at the moment for just one thing: to record a list of IDs of records in the database result from the most recent relevant operation – search, create, edit, etc. In the case of deleting a record, the list is the empty one. This is needed to give the user a pleasant and efficient experience.

I plan to add users and authentication and then put this up on the web for all to enjoy. It will also help me, since I will then have access to my data anywhere there is a connection. I plan to use Mnemonix to hold the user’s JWT authentication token, the id list, and maybe more, e.g, preferences.

The code is at https://github.com/jxxcarlson/lookup_server

2. Rewrite the backend for Manuscripta.io:

http://www.manuscripta.io/

Manuscripta is an app for creating, editing, and distributing lecture notes, although it can be used for other things. For writing techical docmentation (code, math, physics), it uses Asciidoctor-LaTeX (https://github.com/asciidoctor/asciidoctor-latex), of which I am the principal developer.

Here is an example --my course notes – http://www.manuscripta.io/documents/jc.qft?toc

I am a refugee from the Rails world. I used Rails for the first version of Manuscripta, but (a) it was too slow, (b) the code got completely out of control. Partly my fault. It was my first real software development experience. Subsequently I split the app into a REST back end written with Hanami, a Ruby web framework which I really like (http://hanami.org). The current front end is written in Angular1 and I have an Angular2 version in the works. But I love elixir/phoenix and have found the functional programming style to my liking (I used to program in Scheme some time ago). Hence the plan to redo the backend in Phoenix. (1) Above was my starter project to learn Elixir and Phoenix. (I also wrote a command-line version of lookup – https://github.com/jxxcarlson/Lookup)

One challenge (for me) of the backend project is that Phoenix will have to communicate with other processes: ruby for rendering Asciidoc-LaTeX documents into HTML and/or LaTeX, and for converting (when necessary) LaTeX docs to PDF.

I plan to rewrite the front end in Elm, but if you have thoughts on front end technologies, I am interested.

This is probably way too much info, but there you go!

PS – I’ll comment on docs, etc. tomorrow. Late here in Ohio!

[christhekeele]

For those interested in this project, I’ve just published Mnemonix v0.6.2!

v0.6.1 represents a stabilization of the public interface. Functionally, it’s identical to v0.4.0, aside from adding Mnemonix.put_and_expire/4. However, the internals have been completely re-arranged to use less meta-programming, better module names, and have become much easier to maintain and test.

Consequentially, if you were calling Mnemonix.Store.start_link, you’ll need to update your code to use Mnemonix.Store.Server.start_link with slightly different parameters, and all stores now live under the Mnemonix.Stores namespace, like Mnemonix.Stores.Map. Otherwise everything should function identically.

These breaking changes have enabled a few benefits:

• Navigating the documentation is much easier through better namespaces, and internals have been hidden.
• The functions available on Mnemonix have been split up into different Mnemonix.Features docs for simpler browsing of related functions.
• There are quite a few new ways to manage stores beyond the humble store server:
  • Bring up and manage multiple stores with the new supervisor
  • Describe stores in your application config and have them start alongside your application without writing a line of code
  • Define your own module to interface with stores by using the builder macros

The only reason why this stabilized interface isn’t a v1.0.0 is because I want to co-ordinate an announcement around that release and would prefer having more goodies implemented.

Until then, I’ll just be adding more tests, more features, more stores, and improving performance and error handling around existing ones. Update to v0.6.2 and your upgrade path shouldn’t contain breaking changes until a far-off v2.0.0 release!

[codecakes]

Amazing! Integrates with Memcache and Redis? On my list to use!

[christhekeele]

v0.6.3 contains experimental support for using Mnemonix stores to back Plug sessions. Try it out in your Plug-powered apps and let me know if anything breaks!

[christhekeele]

v0.7.1 transparently* serializes and deserializes all keys and values in and out of erlang’s external term format for every operation.

This means that you can, true to Elixir’s maps, use any term as a key or value for all Mnemonix operations, including the features beyond Map (currently increment/decrement, expire/persist). Some out-of-memory stores like memcachex threw a fit if your keys or values weren’t strings or atoms; no more.

This also allows you to provide any store type (not just a Mnemonix.Stores.Map) with an :initial map of entries to populate the underlying store with. Each entry of the map is simply serialized and sent to the store before the store becomes available.

This incurs about a 4% performance penalty to common store operations for out-of-memory stores, but that should be mitigated by all the low-hanging out-of-memory-store optimization I keep putting off. In-memory stores skip serialization/deserialization.

* Transparent serialization/deserialization means that:

• Mnemonix users don’t need to know about it, even error messages come out deserialized.
• Mnemonix store developers don’t need to know about it, they don’t have to implement anything or reference the implementation in their store’s callbacks.
  • However, serialization/deserialization logic is an overridable callback so store developers can optimize it if they wish.
  • This is how the in-memory stores avoid the external term format serialization.
  • But no other store callbacks in Mnemonix’s architecture requires knowledge of the serialization/deserialization mechanism.

Really, I’m running out of cool things to implement–I might even get around to those optimizations soon. Although I do have a new programmable mechanical keyboard arriving tomorrow, so all bets are off.

[christhekeele]

Just released v0.8.1!

The minor bump from 7 to 8 is due to a small breaking change: the :transaction option for Mnemonix.Stores.ETS has been renamed to :concurrent to much more accurately reflect the ETS table option it corresponds to.

The teeny bump to from 0 to 1 is because there was a bug with the singleton builder.

Analogs for Map.drop/2, Map.split/2, and Map.take/2 have been added to the Mnemonix API, and to all stores.

The first outside contribution to Mnemonix (thanks @rzane and @coderrick!), a new Mnemonix.Stores.Null store has been added. This makes it trivial to test against a no-op Mnemonix store!

The star of the show is the experimental release of the Mnemonix.Features.Enumerable API. The v0.8 line will continue to experiment with this capability.

The Enumerable feature is interesting for three reasons:

• It’s the first optional feature.

  Only Map, ETS, and Null stores currently support the Enumerable functions. Interleaving the priorities of a unified Mnemonix API, default implementations for all but the most essential behaviours (fetch/2, put/3, delete/2), per-implementation overrides, and in-caller error raising were all at odds with optional features, but I found a decent way to do it. This opens the door for other Mnemonix.Features that are only applicable to a subset of stores.

• It fleshes out the Map API.

  The Enumerable API allows the supported stores to iterate over their key/value pairs, which ushers in analogs for Map.equal?/2, Map.keys/1, Map.to_list/1, and Map.values/1. This is handy if you’re using Mnemonix explicitly for in-memory caching.

  It also means only 4 remaining map functions are unsupported: Map.merge/{2,3}, (which I’m not sure make sense), Map.from_struct/1 (which will never make sense in Mnemonix), and Map.size/1 (which is set to be deprecated in Elixir 2.0).

• It brings us closer to protocol implementations.

  Both Mnemonix.Features.Enumerable and Mnemonix.Store.Server now support the calls necessary to power the Enumerable and Collectable protocols, there’s just no struct for the defimpl to latch on to. This would involve transitioning from a PID/GenServer based Mnemonix interface to a struct-based one, and I’m not yet sure how I want to do that for v0.9.0. But now there’s a motivation to explore that.

Thanks for watching, and stay tuned for more!

[christhekeele]

I finally found some time to pick this back up again, after a busy few months. v0.9.0 is now released! It’s another small public API change:

• The start_link functions on Mnemonix.Store have migrated to the main Menmonix module.
• The Mnemonix.Store.Supervisor module has been renamed to Mnemonix.Supervisor.

These changes are in service to finally fully hiding the Mnemonix.Store namespace as a private API, making usage of v0.9.0 and above much more resilient to API changes in the future.

Furthermore, the start_link functions in Mnemonix are now furnished by the new Mnemonix.Features.Supervision module, which is also used in the Mnemonix.Builder module. This means if you’re creating your own store modules, you no longer need to implement start_link yourself.

Finally, all functions provided by the Mnemonix.Builder are overridable, meaning if you are fluent in the Mnemonix feature APIs you can redefine them for your custom store modules yourself. This also means if you already implemented start_link yourself you do not need to make any changes.

I know it’s more bookkeeping than exciting features. However:

• It unifies almost all Mnemonix functions and grants them to custom stores. That work should be complete with a final addition of setting a default adapter in the builder, allowing the new functions to move into their own feature and make their way into custom stores. This will be a backwards-compatible change.

• It finalizes the module namespaces for the project’s initial v1.0.0 release, allowing me to make some of the really fun optimization changes with confidence behind Mnemonix.Store.

Also up for the next release is adding warning infrastructure to the private Store API, allowing store implementations to choose to support a feature, even if it is unwise, but emit warnings in the caller when it does so (rather than just having the option to continue or raise). This is in service to allowing stores that you probably shouldn’t enumerate over to be enumerable none-the-less.

[christhekeele]

Hey all!

Lest you think I’ve been sleeping, I thought I’d post an update on all the incognito progress I’ve been making.

If you don’t want to read through my boasting about what I’ve been up to, the most salient announcement is this:

This is the last announcement. There will be no more as the upcoming v0.11.0 will be the last before v1.0.0.

To commemorate that event and garnish more notice I’ll be starting a new thread, so many thanks to everyone that’s followed my progress here to date!

Now that that’s out of the way, here’s a wall of text:

v0.10.0

The observant of you might have noticed that I pushed out v0.10.0 a few months ago. It provided support for ElasticSearch (thanks for the contribution, @BDQ!) and the Elixir 1.5 Map.replace functions, and dropped support for Elixir 1.3 so it could use some newer language features internally (hence the minor version bump).

v0.11.0

I didn’t make an announcement for that release, however, because it drew me into tackling the two least pleasant parts of the codebase internally and I felt more breaking changes on the horizon. This prompted a lot of activity in the last few months, touching nearly every line of code (although I’m proud to report that outside of initialization functions and formatting none of the doctests changed). This will shortly culminate in penultimate release before the v1.0.0 milestone.

An overview of the more notable evolutions:

• The signatures for store initialization will change.

  The solution I hand-rolled for this in v0.9.0 was in fact unwittingly a sort of ad-hoc version of Elixir 1.5.0’s child_spec, but not similar enough to avoid being fundamentally incompatibly with it. I’m still updating the documentation to reflect this change so I’m afraid you can’t get great insight into the ramifications it by perusing it, but this will be in place for v0.11.0.

• All new Elixir 1.5 features that can be used, are, and support for 1.4 will be dropped.

  Aside from full child_spec support, all internal behaviours use @impl and behaviour-level defoverridable, and the handshake between clients and the server uses parameterized types to clean up some messy typespecs. This should make adding new features and stores to Mnemonix even easier as all sorts of tooling will warn if the internal contracts between client and server, and server and store implementation, are not quite met correctly.

• The client builder is substantially more sophisticated.

  One of the more ambitious goals I have had in mind for Mnemonix is either getting it into Phoenix, or having it be recommended by Phoenix guides, as the defacto out-of-memory cache and session store utility. Whether or not that will ever happen is completely undetermined but that hope has set the standard for the level of flexibility and rigor that I’ve been pursuing with the project, especially with regards to being used as a configurable intermediary to key/value stores in other libraries.

  If for some reason you do not want users of your library to have to know about the Mnemonix namespace, and want to forge a custom API to interact with stores, you can simply use Mnemonix.Builder into a module. This module then receives all functions to interact with a Mnemonix store as if it were the Mnemonix module itself, with all relevant documentation. These functions can optionally be inlined into raw GenServer calls instead of simply delegating to Mnemonix functions, and you can omit their docs if you’d rather. You can cherrypick which feature-sets you want to include or get them all wholesale.

  You can also build ‘singleton’ functions instead of normal ones—then the generated functions know in advance about a GenServer name for a store your library will be managing itself in its own supervision tree. All the functions in the module using the builder will no longer need to specify a store reference as their first parameter because they will be generated with knowledge of this GenServer name to use instead.

• The default implementations for all callbacks are trivially testable in isolation.

  Mnemonix is, at its core, two behaviours: one that client modules must implement to send commands to a store server, and one that store implementations must meet to understand and act upon instructions received by a server.

  Each module using one of these behaviours must define some 35 functions in order to fluently speak Mnemonix, and that’s a lot to ask. Fortunately, these two behaviours can derive default implementations for all of them, save for 4 on the store implementations: an init function, and basic map put/fetch/delete implementations. In order to DRY up the source code, make contributing to Mnemonix easier, and allow custom Mnemonix clients to override behaviour, when you use a Mnemonix Feature Behaviour or Mnemonix Store Behaviour into a module, these default implementations are provided.

  People who have ran into me on this forum or on the mailing list in certain conversations might know I’ve long been agitated by the current idiomatic solution for creating behaviours that furnish using modules with default implementations of their own callbacks. This is because the current default approach—throwing a bunch of defs inside a __using__ macro—does not scale well or offer great testability for these default implementations. This is fine for most use-cases but spectacularly insufficient for Mnemonix’s very particular architecture.

  I’ve created a very particular piece of metaprogramming to solve this pain point that also powers all of the features in the builder tool: Mnemonix.Behaviour. Both feature behaviours and store behaviours use this module; and every function they define that matches the name of a callback also defined in the module is transferred to any module that uses the behaviour as a default implementation. Crucially, this means that these default implementations concretely exist on their host modules instead of living in an un-compiled __using__ void, so they can visibly document their behaviour, emit compiler and dialyzer warnings, and be easily tested outside of where their behaviours are used.

  The current implementation is Mnemonix specific but I may get around to extracting it into a more general form and publishing it somewhere.

I will release as v0.11.0 once the lingering planned tasks are finished, which I anticipate to be sometime soon after the turn of the year.

v1.0.0

The two main obstacles remaining before takeoff are, as they always have been, performing more extensive testing outside of moduledocs and replacing derived default store function implementations with optimized store-specific versions where possible.

Aside from some extra documentation and tooling tweaks, those are also finally the only obstacles left, and in fact things have been feature-complete and nearly API-stable for quite some time. I’ve talked about those two big hurdles a bit here and on the issue tracker, so there’s really not much more to say. However the preceeding v0.11.0 can be treated as an RC since those remaining issues in no way affect any Mnemonix API or behaviour, and just polish them up.

However, I’ve put together a lot of supporting tooling with an eye to the pending Big One in mind that have taken a lot of effort that I thought I might mention before closing:

• A scriptable credo style guide is pretty much complete
• Dialyzer is happy with the entire project
• Both of those as well as code coverage and documentation coverage are executed in the build process
• The source code is fully formatted with the upcoming formatter
• The dev environment is instrumented with cortex to run tests as files change (and hopefully all of the above, soon)
• There is a wiki on the repo to host user-contributed content and guides should they emerge
• There is a project on the repo for triaging inbound issues and pull requests

I look forward to seeing you in another thread! Maybe I can contrive to drop the Big One on Valentine’s Day to help express the gratitude I have for you all.

[OvermindDL1]

I’m curious as I did not come across it quickly, how easy is it to use multiple back-ends at once? Do they have their own interfaces so you have to know which to access? Is it possible to multiplex multiple ones via the same interface and shard them, mirror them, etc… and etc…?

[christhekeele]

I’m glad you ask, actually, as this will be the subject of the planned v1.1.0 release. I’m really excited to work on it but I’m using v1.0.0 as motivation to tend to some of the more tedious touch-ups first.

I intend to offer a series of stores under the Mnemonix.Stores.Meta namespace that do exactly this.

Before then there will be a guide to implementing custom stores that should help you figure this out yourself. In the meantime I just threw together a proof-of-concept pass-through proxy here: https://gist.github.com/christhekeele/1c0dc69d23931c69e1d50fc2fe9ca842

There are 4 core functions you must implement in a store, one of which is a setup phase to convert arbitrary store-specific arguments into arbitrary store-specific state to hold in the server process. The remaining mandatory 3 put/fetch/delete functions can make use of this state however they need to; the other 30-some functions are automatically derived from them. This makes it pretty easy to set up custom stores with whatever initial state you need and bootstrap a full feature-set with minimal effort.

Some notes:

• use Store.Behaviour is what allows everything else to be derived
• use Store.Translator.Raw prevents any serialization/deserialization of the keys and values, which allows the dependent stores to handle that logic however they need to
• def setup(opts) returns {:ok, state}; that state can be accessed later under the Mnemonix.Store struct’s state field
• def put(store, key, value) and def delete(store, key) just run the operation against the frontend and backend stores sequentially, propagating errors
• def fetch(store, key) returns a found value from the frontend, then tries the backend. If found in the backend it also stores it in the frontend; otherwise it returns :error

There are some nuances that this implementation completely sidesteps:

• A lot of the other derived map operations in terms of the 3 core ones will exhibit a lot of unnecessary churn and could be overridden to minimize reads and writes; that’s exactly the sort of optimization I need to make for the existing stores before v1.
• The Mnemonix client API should not be used inside store implementations themselves since warnings and errors intended to be propagated back to the calling client function will instead occur inside the server process of the metastore.
• The arguments to start_link should be {store_module, setup_opts} (instead of pids of active stores) so that hot code updates and supervision recovery works better; this mandates the metastores also acting as a supervisor on some level so they’ll need smarter start_links.
• I suspect that the naive serialization/deserialization strategies for single stores may not be sufficient for map functions that accept transformation functions when applying them across composed stores; that API may either have to be improved or those functions overridden in metastores with a smarter treatment.

Outside of these known complications, though, this example passthrough seems to be working as advertised even for complicated derived commands like, say, get_and_update. The point of v1.1.0 will be to create metastores that address these subtleties fully.