There is a vibecoded Elixir version of the Dexter. Could you please test it out too to compare the feature parity and performance?
Asd
dimitarvp
That depends from what angle are you asking your question. If you believe that quantity equals quality then you’d be flabbergasted. From my point of view, it’s very possible that Dexter’s creators are the only sane ones in a crazy world.
I am much more confused as to why is not every LSP using an embedded DB.
That’s valid. I’d probably just go all the way: write it all in Rust and use one of Rust ecosystem’s embedded DBs. That way the only plumbing would be to distribute a precompiled NIF (trivial, especialy in the age of the LLMs).
1
Asd
Cause you dont need an embedded DB to store and incrementally update the text index on disk. Even for a huge project like, for example, Android or Google Chrome, index would fit in 1GB of memory, so you just need a way to load the index from disk to memory, dump it back and to do this incrementally. That’s it. You don’t need a query language because there are no queries, data is already in memory and you can just filter it yourself.
0
dimitarvp
1GB index dumped back into memory can easily be 8GB of RAM usage so I would very strongly challenge such approach in a PR for a team I am a part of. I would not want my LSP to take 8GB of RAM.
That being said, you could very easily be correct in general.
My experience tells me that many devs brush off proper architecture with the ubiquitous-but-overused YAGNI concept and then one day admit “sh1t, we actually did need a proper DB”. I’ve seen it dozens of times and that made me much more moderate when applying YAGNI.
To me bringing in a bundled SQLite (so you don’t require your users having it installed on their machine) and accessing it via a fully-contained Rust NIF does not deserve the term “overhead” (Like my library that has been promised for years already does – I am now finalizing the Ecto layer and will be announcing it sometime this year). It’s trivial to do. (Or don’t use SQLite at all; the Rust ecosystem has a bunch of less functional but still pretty good embedded DBs.) You pay a very small housekeeping price, once, and you get a peace of mind that you can go all the way to CTEs / window functions / recursive queries if you need them one day. And you can skip rolling your own household query engine – which such projects inevitably end up doing.
Engineering tradeoffs and all, I think we all here agree, but to me this discussion is akin to “why should I pay $0.30 to get a crate of gold delivered to me when I can pay $0.00 and get a crate of tin instead”. Hence my line of argumentation.
2
jswanner
mudasobwa
Creator of Cure
FWIW, while building Ragex (it is not released yet, but it still can be considered kinda LSP-like beast) I’ve tried many options, including, but not limited to term_to_binary |> File.write, SQL, NoSQL etc.
Graph DB stored in ETS worked for me. Mostly because the sourec tree is actually a graph, and it can barely be well-projected onto relational and/or flat store.
1
dimitarvp
Asd
It’s not YAGNI. Solutions like SQLite introduce constraints for the features which you don’t need. For example, data must be tabular, it must live in a single file, you need a NIF adapter, you need to manage “connections”, data will must live on disk, parallel access will be limited by MVCC constraints, etc.
0
dimitarvp
Yeah I don’t disagree. Let’s not over-fixate on SQLite. As @mudasobwa mentioned above, it could be a different kind of a DB (graph in his case). My broader point is: why not use something that can serve your storage and querying needs from the get go instead of inevitably reinventing a mini DB engine two years in the future, based on an idealistic hand-wavy “let’s have less dependencies” vibe?
Additionally, I don’t see the “NIF adapter” being any sort of a negative point here, especially with high-quality LLM agents in the equation. You can link your Elixir LSP code with a library written in almost any programming language in an hour.
1