I think I’ve tried 5 different graph database libraries in the last two days and not a single one has been able to connect to a remote/local db. All sorts of wonderful errors, db connection genservers crashing on MatchErrors, library only supports version 4.2 of db and the only ones offered now are 5+, etc.
Perhaps it’s because I’m using 1.14.5/otp26, but it seems more that many of these libraries are just no longer actively maintained.
Perhaps it’s a sign from the Ecto gods that I shouldn’t use a graph db in my project.
Anyway, is anyone actively using elixir with graph databases? If so, what’s your stack?
Side note: attempting to debug and troubleshoot said errors has taught me a significant amount of valuable insights into how elixir, mix, libraries, config all work together… so that is a big plus! Also, why it’s important for a library to include tests
I guess you refer at least to Neo4J. In the past, I did some experiments using bolt_sips, but as you say, it seems not to have been updated for Neo4J version 5.*.
I’m currently working on a new driver for neo4j that will support v 5+. I’m still ironing out a few things but I hope I’ll be able to release it by the end of June or so.
In the mean time neo4j v 3 and v 4 can still be downloaded. If you don’t use the commercial version, the feature set is rather stable.
Bolt_sips seems to currently top out its support at Bolt v3, and it sounds like @krstfk has maybe stepped up to backfill maintenance & support for more recent protocol versions.
The dream I have is that, since so much has already been done by the Elixir community in terms of Postgres, Apache AGE (https://age.apache.org/) could be the impetus for more robust Cypher/graph support in Ecto, which could have spillover to Neo4j usage. But that’s just a dream I’m not sure will ever materialize.
The thing is that there’s an impedance mismatch between Ecto and graph databases. I believe Dominique Vassard has done some work to provide an ecto like interface on top of bolt_sips but I’m not convinced it may work as is. I believe he also has done some work taking inspiration from ecto.
That being said, one of the things that needs ironing out on my side is providing a repo construct (which is harder than initially thought because clustering is very much a necessity for a driver for neo4j).
By the way Florin Patrescu has stated a while back that he was not able to maintain bolt_sips anymore. And I would like to make it very clear that I have not started a new project because of any issues I had with bolt_sips or any of its maintainers. The thing is when bolt_sips started (and even a couple of years ago), the bolt protocol was poorly documented and now the situation is much different which may lead to radically different design decisions and incompatibilities, which is why I have opted to work on a new driver (hopefully released by the end of June) that will definitely owe a lot to Florin Patrescu and all of the bolt_sips contributors.
Thanks for the update and for sharing your thought process! When i was sleuthing, my trail turned up cold after a January comment in a GH Issue for bolt_sips, and everything you’ve stated makes perfect sense about starting from scratch-ish.
I’m well aware of Dominique’s work and the challenges inherent to it (but I’m glad you mentioned them so others can benefit from learning about them too). While I appreciate and do not discount the journey that led him to evolve his efforts from ecto_neo4j to seraph, I also believe that having more eyes and minds on the problem would allow for finding the surface area for tractable integration solutions for Ecto, even if it isn’t 100% coverage of the problem space available to a graph database engine because of any mismatch. That’s why I feel there could be potential coming out of Ecto adapters for Apache AGE, which actually would require context-switching between SQL and Cypher. But i hear you: it’s not necessarily a simple thing, which is why i temper my hopes.
Sorry about that, the whole thing turned out to be way more difficult than I initially anticipated.
But I’m kind of hopeful now. While I’m still not sure the api is sound/good, I guess I’ll find out when people start using it.
While I’d love a rich ecosystem, right now my goals are speed and flexibility. One of the thing I really want is the ability for the user to provide their own encoding/decoding functions for so called bolt structs (meaning the ability to translate a node with label x into eliixir sruct y). That would avoid having to traverse and convert a list of results. Another goal is to have support for back pressure. I’ve actually split the concerns into two libs, one -inspired by mint- handles a single connection and the encoding/decoding, the other one closer to bolt_sips will handle pooling, routing and all of that.
The first lib is close to completion but needs a bit more tests, the second one needs a bit more work.
This is a quick update to let you know that I’m still very much working on the project. However I got side tracked and working on libs for general consumption turned out to be harder than expected.
As a result, I’ll release a couple of libs on github before any hex release. My hope is to get feedback from the community before the hex releases.
I’m mainly working on two libraries (names subject to change):
m_bolt : a very thin implementation of the bolt protocol supporting versions 1 to 5.3 (currently). I’m now working on a third (and last) iteration. The m in the name is an hommage to mint because it’s a process less library (and the transport code was stolen from mint). The goals here are efficient decoding and maintainability (bolt is a moving target). A side effect of the redesign is that it’ll be possible to translate bolt structs into elixir struct at the decoding stage. m_bolt will not provide any pooling nor routing support (apart from allowing to setup server side routing).
neo4_ex : built on top of m_bolt and db_connection, which aims to be close to what bolt_sips provided.
Status :
I’m fairly happy with the decoder in m_bolt and I believe further optimizations will bring diminishing returns.
The api for m_bolt has gotten cleaner, if a bit verbose (but that shouldn’t matter for most users).
The transport code is so/so because from neo4j 4.2 onward, the buffering of the records has been dropped which means that you get blasted with very small packets if you do eg match (n) return n where n is small. I haven’t been able to find a solution to this issue.
neo4_ex : at first I’ll only have a functional but maybe awkward support for routing. (Neo4j now supports multiple databases, each could get its own routing table).
Re. the question of this thread, I’ve started using Memgraph (via the bolt_sips lib, which is intended for neo4j but works perfectly well so far thanks to the compatible query language / API) and enjoying it so far, the built-in algorithms are handy, plus the catalogue of extra ones you can pull in (even though they’re in python, wonder if this is somewhere where Nx could come into play at some point?)
m_bolt is intended to work with memgraph, and does.
This is, part of the reason, I’ve decided to split the library between a bolt implementation and a neo4j driver.
I guess it’ll make more sense when I release the code.
(I’d like to stress again that my current work in the field doesn’t come from any hard feelings with Florin or anyone else who has contributed to bolt_sips, nor that I think it’s bad.
Historically, there was little if any documentation regarding the bolt protocol. Yet, Florin with some help from Dominique have laid out the groundwork I’m relying on. ).
Thanks for the update! I’m not sure how much value I can bring, but I’ll gladly bring what value i can to help with a review.
Re: your comment on the transport code. Maybe this highlights my ignorance or a misunderstanding of what you’re describing, but the JS driver has three different APIs: one for streaming, one for reactive sessions, and one for Promise-based sessions. I’m wondering first if that’s at all related to the issue you’re describing, and second whether those implementations offer any insights for how to approach improvements. However, it seems (and I could be totally wrong) that the streaming/reactive APIs would require a Process-backed approach that might be incompatible with what you were aiming for?
If I remember correctly all drivers implement streaming atop of the pull n feature introduced with bolt 4.0.
The issue with the transport code is that until neo4j 5.0, the server would buffer small records and emit packets only when the buffer was filled. Then, when they rewrote the server code (with netty I believe) they ditched the buffering and started emitting packets for each record, regardless of the record size. This is a known issue that will affect all drivers, according to them, it’s not much of an issue in real life scenarios since records would typically be larger (details Potential performance regression in neo4j 5.x with bolt, maybe linked to buffering · Issue #13045 · neo4j/neo4j · GitHub ). When, writing the driver that means, unfortunately, that it’s pretty much impossible to write code that consumes records efficiently for every scenario (basically if you were to set active once for each packet, with small records, you’d do it in a very tight loop leading to really bad performance).
Regarding streaming, m_bolt does support the pull n feature, but it doesn’t concern itself further with it (this will be handled at the neo4_ex level). m_bolt being inspired by mint, there is a way to receive records (by the calling process), which means it’s possible -in theory - to process records as they come. The issue here is that records should only be considered final when they’re followed by a success message, meaning that implementers of higher level drivers using this feature will need to be careful. In any case those won’t fix the issue with large amounts of small records.
Edit : the numbers provided in the github issue aren’t meant to compare drivers performance.
Thank you for such an informative reply! I thought perhaps the streaming could have been enabled by a novel implementation hiding underneath that addressed the packet/transport issue, but if the folks at Neo4j called it out as an intractable degradation, I trust their assessment (and yours, too ;-p).
Hello! I’ve been working on a fork of Bolt Sips for the past 3 months. You can find the project at: https://github.com/sagastume/boltx
It provides support for the latest versions of Neo4j and Bolt. Personally, I’ve been conducting tests with Neo4j Aura.
During this period, I made numerous changes to Bolt Sips, including refactoring PackStream, protocol, socket, and more. The idea behind these changes is to make the implementation as simple as possible, making it easy to adapt to future modifications.
Feel free to try it out and let me know if everything is working smoothly
Compatibility with Neo4j versions: 3.0.x/3.1.x/3.2.x/3.4.x/3.5.x/4.x/5.9 - 5.13.0
Compatibility with Bolt versions: 1.0/2.0/3.0/4.x/5.0/5.1/5.2/5.3/5.4
I’m using a graph database as an appliacation database (just a prototype for now).
My requirements were that I do not need to setup a server and that I can do simple snapshots for undo-redo. afaik there is only one graph-db that is suitable for that: redisgraph (or since that is discontinued: https://www.falkordb.com)
I wrote a Elixir-falkordb-client (should also work with redisgraph) if anyone is intereseted I can open-source it.
(I also tried bolt-sips but for some reason it does not work with falkordb, though it should)
