Hi All,
I have just released a project that I had been hacking on for quite a while and would love to get your feedback. It’s basically redis implemented in Elixir. The line level protocol stays the same, this way people currently using Redis who’d (maybe) wish to switch to Remixdb, needn’t have to install new drivers/packages/what have you.
It’s still super experimental and I have some crazy ideas for doing real time sharding, moving processes across nodes in a cluster etc., but it’s no where near being reality, yet.
No supervisors. Not only is this bad practice, it goes against all general claims to fault tolerance in the Readme, as well as against explicit claims to use supervisors.
No distributed functionality, despite claiming so in the readme.
Your latest commit says “Using HashDicts for better hash performance” and the commit shows you’ve replaced a bunch of Maps with HashDicts. Not only is this false, Maps perform better in literally every case, but HashDicts are deprecated.
you’re creating an escript, but a release would be more appropriate for a long running service.
extensive use of spawn_link instead of using OTP behaviours.
unlinked processes.
You aren’t following elixir conventions for file organization
I’m not trying to discourage you, I think there’s a lot of value in thinking through how a key value store in Elixir might work. However, right now your readme is 100% aspirational, but written as if it reflects the current capabilities of your code. You do yourself and your users a disservice by not being clear on this point.
Thanks for your input. Yes, I’ll be the first to agree the code is not
backing up what’s on the README. But the fact that this is being built in
Elixir means I have somewhat of a shot at making this real.
Ben Wilson - thanks for your suggestions. All of them are valid and I
should be addressing them shortly. To pick just 1 – the reason for the
switch to HashDicts was that “Elixir in Action” said they performed better
than Maps for larger collections. Is this wrong? If so, then yea it makes
sense I didn’t do it.
Again this is super experimental and all these issues such as – Making it
an OTP application, running it as a daemon are all in my list of things to
do. Heck, I need to be able to read a RDB file to be able to load in
existing Redis data!! So yea, I am in no way saying yank out Redis and go
and use this in production right now.
Thanks again for offering your inputs, looking at code.
Elixir in Action is an incredible resource and most of my knowledge of OTP comes from it. I see that you’ve already been pointed to the article that @sasajuric wrote when Elixir 1.2 came out.
Are you trying to create a Redis server? There are many client implementations that you can find redis.io itself, many Erlang and a few Elixir clients.
I appreciate your response here, and I do believe that you had no intention of misleading any one.
My only further comment is to suggest that while some of these features (distribution) are definitely things that you can wait on for now and develop in time, others form the foundation of a proper Elixir package and should be prioritized higher, even over “basic functionality”. Specifically:
Organizing your code properly. It’s really hard for people to offer advice or contribute if there’s no real structure to the files.
Using supervisors. This one is crucial. It will also simplify the spaghetti of start_link calls that happen.
Generate a valid application. mix new with the --sup flag gives you the bare bones, and it’s covered in the getting started guides and virtually every elixir book.
Having a Redis protocol compatible k-v, but with an Elixir implementation certainly sounds interesting!
However, I’d advise against hand building the distributed k-v part yourself, as it can become hard to get proper CP or AP guarantees in a clustered environment. Instead, I’d suggest looking for third party libraries which solve those problems. The first ones which come to mind are riak_core for eventual consistency, and riak_ensemble for strong consistency. I also wonder whether recent work on Phoenix presence (i.e. CRDT) could be used for a general purpose distributed k-v, and what are the trade-offs. I’ll ping @chrismccord to hear his thoughts on this
However, if you just want to practice and/or have fun, then it will probably be simpler to build from scratch, as long as you’re not optimistic about proper behaviour on netsplits. Getting that part right requires carefully applying some scientific papers and a good amount of testing. Otherwise various subtle errors might occur, and the storage might have unpredictable behaviour on netsplits.
I agree with other comments, especially those made by @benwilson512. An app should be OTP compliant, because strange bugs may occur, and it might not play nicely with the client’s project. If you’re still practicing, it’s fine to violate those principles and taking baby steps in whichever order suits your flow. However, the project Readme should clearly state that, because otherwise people may get false impression about the maturity of the library. Here’s an example of how we did it for our Phoenix socket client.
Thanks for your suggestions. I’ve gone ahead and updated the README, to
reflect the same and appreciate your suggestions on that as well.
I hadn’t thought about riak_core/riak_ensemble. Will check it out. As for
CRDTs, isn’t that a little loose with it’s consistency guarantees? I was
considering those, but Redis is very strong with it’s consistency
guarantees, so I’d like to have RemixDB meet those as well.
Appreciate your response as well. Misleading is definitely a bit of a stretch, since a whole slew of sections are TBD’ed. I’ve gone and thrown in a disclaimer now, so it should be more obvious now.
And yes, I’m in complete agreement, supervisors/other OTP constructs are definitely something that will be there when it’s usable and are a high priority. RIght now, as Sasa had picked up, it’s more of a playground than anything else. But should be usable real soon.
According to the (admittedly 3 years old) analysis by Kyle Kingsbury (aka Aphyr), Redis offers no real guarantees. This recent article by Martin Kleppmann is also an interesting read on the topic.
My takeaway is that Redis is not consistent, so it’s only suitable for some nice-to-have caches. Disclaimer: I didn’t follow the development of Redis for past few years, so perhaps something has improved here.
@santosh79 This is a small nit-pick but it would also be a good idea to include a license in your project. Potential contributors might be discouraged from helping out if they don’t know to what degree the code belongs to the community.
+1. I always look at Scala code and just appalled by the ! all over the place. I love that you provided a list of excellent points too. I should learn how to implement a Redis protocol too. I applaud the effort @santosh79!
I applaud the effort 
