I get that, I just haven’t seen it before with other LSPs. I guess I’d prefer these things with as few dependencies as possible but that might just be me.
I’ve been having a ton of problems with Expert LSP though, so I’ll give Dexter a try.
There is a vibecoded Elixir version of the Dexter. Could you please test it out too to compare the feature parity and performance?
That depends from what angle are you asking your question. If you believe that quantity equals quality then you’d be flabbergasted. From my point of view, it’s very possible that Dexter’s creators are the only sane ones in a crazy world.
I am much more confused as to why is not every LSP using an embedded DB.
That’s valid. I’d probably just go all the way: write it all in Rust and use one of Rust ecosystem’s embedded DBs. That way the only plumbing would be to distribute a precompiled NIF (trivial, especialy in the age of the LLMs).
1 Like
Cause you dont need an embedded DB to store and incrementally update the text index on disk. Even for a huge project like, for example, Android or Google Chrome, index would fit in 1GB of memory, so you just need a way to load the index from disk to memory, dump it back and to do this incrementally. That’s it. You don’t need a query language because there are no queries, data is already in memory and you can just filter it yourself.
1GB index dumped back into memory can easily be 8GB of RAM usage so I would very strongly challenge such approach in a PR for a team I am a part of. I would not want my LSP to take 8GB of RAM.
That being said, you could very easily be correct in general.
My experience tells me that many devs brush off proper architecture with the ubiquitous-but-overused YAGNI concept and then one day admit “sh1t, we actually did need a proper DB”. I’ve seen it dozens of times and that made me much more moderate when applying YAGNI.
To me bringing in a bundled SQLite (so you don’t require your users having it installed on their machine) and accessing it via a fully-contained Rust NIF does not deserve the term “overhead” (Like my library that has been promised for years already does – I am now finalizing the Ecto layer and will be announcing it sometime this year). It’s trivial to do. (Or don’t use SQLite at all; the Rust ecosystem has a bunch of less functional but still pretty good embedded DBs.) You pay a very small housekeeping price, once, and you get a peace of mind that you can go all the way to CTEs / window functions / recursive queries if you need them one day. And you can skip rolling your own household query engine – which such projects inevitably end up doing.
Engineering tradeoffs and all, I think we all here agree, but to me this discussion is akin to “why should I pay $0.30 to get a crate of gold delivered to me when I can pay $0.00 and get a crate of tin instead”. Hence my line of argumentation.
2 Likes
next-ls made by one of the leads on Expert uses sqlite.
Next LS also used SQLite
FWIW, while building Ragex (it is not released yet, but it still can be considered kinda LSP-like beast) I’ve tried many options, including, but not limited to term_to_binary |> File.write, SQL, NoSQL etc.
Graph DB stored in ETS worked for me. Mostly because the sourec tree is actually a graph, and it can barely be well-projected onto relational and/or flat store.
1 Like
Nice. I can see how that works well.
It’s not YAGNI. Solutions like SQLite introduce constraints for the features which you don’t need. For example, data must be tabular, it must live in a single file, you need a NIF adapter, you need to manage “connections”, data will must live on disk, parallel access will be limited by MVCC constraints, etc.
Yeah I don’t disagree. Let’s not over-fixate on SQLite. As @mudasobwa mentioned above, it could be a different kind of a DB (graph in his case). My broader point is: why not use something that can serve your storage and querying needs from the get go instead of inevitably reinventing a mini DB engine two years in the future, based on an idealistic hand-wavy “let’s have less dependencies” vibe?
Additionally, I don’t see the “NIF adapter” being any sort of a negative point here, especially with high-quality LLM agents in the equation. You can link your Elixir LSP code with a library written in almost any programming language in an hour.
1 Like
Thank you! We actually already support this as a code action. Give it a try and let me know how it goes.
1 Like
On the SQLite discussion: one of the issues we’ve had with existing LSPs is that they end up eating up a ton of memory and are slow on large codebases. One could argue that in-memory databases might work just as well as SQLite, but SQLite already checks all the boxes we need. Most devs already have SQLite installed on their machines, and the indexing and querying we get out of the box are fantastic. Switching to an in-memory DB is certainly possible, but it’d come with its own trade-offs. SQLite does everything we need with very few drawbacks.
5 Likes
Not sure what you mean by code action but if it is supposed to work as I described it then it does not.
It’s an LSP feature. It won’t automatically alias for you unless you explicitly run the code action. In VS Code it’s something like Cmd-. and it’ll pop up available code actions.
Looks like it is in nixpkgs now! Thanks @adamcstephens!
3 Likes
Incredible work @JesseHerrick!
I allowed myself to publish a package of it on the Archlinux User Repository.
1 Like
Nice work!
One thing I’ve not noticed anyone mention yet (unless I missed it!) is the security benefits of this approach.
Elixir compilation means that executing arbitrary Elixir code, and that can do anything at all. Previously if we wanted to audit a version of a package before adding it to our project (as we should!) we would have to browse the code without any LSP assistance, as doing so would execute not-yet-trusted code. Now with Dexter we can have a proper and modern development environment while exploring new code. Bravo!
14 Likes
I tried it. It’s great. Remarkable that the first commit seems to have been made only 2 weeks ago, and it already has so many features.
I’ve had major issues with Expert LSP, and have been looking for alternatives. I think if this thing showed compiler warnings/errors inline, it would basically be perfect. But I know the whole point is to offer all these features without compiling. Le sigh.
