ostinelli
Let’s write a database! Well not really, but I think it’s a little sad that there doesn’t seem to be a simple in-memory distributed KV database in the beam. Many times all I need is a consistent distributed ETS table.
The two main ones I normally consider are:
- Riak which is great, it handles loads of data and is based on DHTs. This means that when there are cluster changes there is a need for redistribution of data and the process needs to be properly managed, with handoffs and so on. It is really great but it’s eventually consistent and on many occasions it may be overkill when all I’m looking for is a simple in-memory ACI(not D) KV solution which can have 100% of its data replicated on every node.
- mnesia which could be it, but unfortunately requires special attention when initializing tables and making them distributed (which is tricky), handles net splits very badly, needs hacks to resolve conflicts, and does not really support dynamic clusters (additions can be kind of ok, but for instance you can’t remove nodes unless you stop the app).
- …other solutions? In general people end up using Foundation DB or REDIS (which has master-slave replication), so external from the beam. Pity, no?
So… 
Well I don’t plan to write a database (since ETS is awesome), rather distributing it in a cluster. I’d simply want a distributed ETS solution after all!
I’ve already started the work and released a version 0.1.0 or ram:
https://github.com/ostinelli/ram
Docs are here:
Please note this is a very early stage. It started as an experiment and it might remain one. So feedback is welcome to decide its future!
Best,
r.
Where Next?
Trending in Announcing
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Elixir, Erlang & messaging conferences from Code Sync. Next up: ElixirConf US (10–11 Sep) & Code BEAM Europe (21–22 Oct), Haarlem + online.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #blog-post
- #ai
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
kokolegorille
There is one in Elixir
ostinelli
It looks nice! Thanks for pointing it out, but it doesn’t look distributed, which is my main point. Or did I miss where it says so?
kokolegorille
I am not sure about the distributed part, it’s probably more for embedded system.
Maybe @lucaong can confirm.
eksperimental
CubDB is great, I have been using it for some of my projects. I love its simplicity,.
You can read this FAQ to learn more about it.
https://hexdocs.pm/cubdb/faq.html#content
I don’t think it is distributed.
lucaong
Yes @kokolegorille and @ostinelli , CubDB is an embedded database, therefore by design not distributed. In principle one could use it as a backend for a distributed system, but its main use case is embedded applications (think Nerves, or a mobile or desktop application), or for cases where one needs persistent and fail safe application-local storage.
It goes beyond key-value: it supports sorted selection of ranges, and atomic transactions. It is stored on disk, and optimized for robustness in case of sudden shutdown and for needing very little memory and CPU (all good qualities when running on small embedded devices).
You could evaluate it as an optional durable backend for your distributed K/V store, although in your case DETS is probably simpler to adapt if you only need K/V
ostinelli
Thanks for your input Luca! I’ve no plans on making ram persistent (it’s in the name!), there are many solutions for persistency and they solve other problems. I was just looking for something similar to a distributed ETS table.
RudManusachi
Thank you for your work!
I haven’t tried yet, but seems like also a reasonable solution could be put up together based on
rabbitmq/ra.Here is a tutorial how to build a simple distributed kv-store based on
map.ostinelli
Thanks for your input! Do you know whether this example is consistent (not eventually consistent) and whether it automatically supports dynamic clusters?
RudManusachi
The library behind it,
ra, implements raft consensus algorithm so it’s consistent.That’s the part that’s not covered in the tutorial
there are functions in
:rathat could help to achieve that, though,:ra.add_member/2,:ra.remove_member/2and:ra.members/1.Btw, thank you for responding me in twitter! I think it will help users if it’s mentioned in the description of ram that it uses 2PC to achieve consistency!
fabriziosestito
https://github.com/rabbitmq/ra#dynamically-changing-cluster-membership
This part of the README covers dynamic clusters.
Only thing to keep in mind is that only one cluster membership change at time is allowed, concurrent changes will be rejected by design (this is described in the Raft paper too if I am not mistaken). This means you have to wait for the clusters to propagate the cluster membership change prior changing it again.
About consistency guarantees, Raft guarantees strong consistency.
Have a lot of fun!