Extending IEx to have nREPL capabilities

Chat & Discussions Proposals: Ideas
, ,
1

IEx is a very powerfull shell and it would be awesome to have all this power integrated inside a code editor. Clojure enables something like that with nREPL.
nrepl has a set of operations that it support. I think for Elixir a good initial point would be the IEx helpers and autocomplete.

what I envision is:
you can run IEx on “network mode”, this way there is a port where IEx can receive operations and eval it and return the result of the operation to the caller.
this way any language(lua for nvim, js for vscode, elisp for emacs) could have a client that issues those commands and can use the response to extend the capabilities of the editor.

difference from LSP:
language servers are limited to what the protocol allows you to do. in other words, if there is no way to expose a functionality through LSP to the editor, it won’t be available. in the other hand with a “network iex” we could handle all the IEx helpers and shell tools directly to the editor.

advantages of that approach:
new features and functionalities added to IEx could easily become new features and capabilities in the editor. instead of the herculean job of interfacing elixir functionalities to the LSP.

there other nrepl approaches:

it’s not listed there but there is a ruby one too:

10 Likes
IEx - Expose API to evaluate code inside IEx session for REPL Integrations
2

some stuff that would be possible with a nrepl that is impossible with LSP:

  • iex buffer inside your editor without a terminal emulator. simple buffers are less resource consuming
  • work without a project(afaik, all current elixir LS kinda relies on a existing mix project to work)
  • no need to implement an expensive protocol to expose functionalities(both LSP and DAP are verbose protocols). this would reduce significantly the load when using the tool
  • no need for extra dependencies(now that erlang ships with a json encoder/decoder this could be the base of the communication or even just lines sent through a socket, even less overhead).

The initial objective here is to understand if this is a welcomed idea, we could discuss further details as the discussion moves forward.

3

I would love this.

2 Likes
4

Stuff that would be possible and great to have in that scenario would be once IEx stablishes an interface for network, other tools could expose apis ready to be used from the editor. like running credo checks or execute a particular tests and put you on the line that broke for that test and show a pop up with the error. maybe a logger backend that once attached to a editor shows the log as a pop up under the line that logged it.

I think it’s possible to have a nrepl in elixir outside of IEx, but being it inside IEx, we could start decoupling some outputs from being exposed in the shell, this way it’s easier to send it to the editor. I belive there might be some stuff like that already in place for Livebook(i need to take a look in the code).

IEx - Expose API to evaluate code inside IEx session for REPL Integrations
5

@mauricio.szabo that works on the nrepl for ruby and wrote the client for clojure nrepl on atom/pulsar did a video showing the powers of nrepl in ruby, i think that’s a good way to see how different and helpful a nrepl is from a usual language server.

3 Likes
6

Author here, ask me anything :).

So, the project itself is not that hard. nREPL itself is very simple to implement, and even bencode is not that hard (having implemented it in Clojure and ClojureScript, in a purely functional way). It give a huge benefit to development, too.

The most complicated part, for Ruby at least, was (and still is) the actual editor part - parsing code and identifying where we are in the source, and what exactly do we need to evaluate, especially considering all special syntax that Ruby have.

For Ruby, things work quite well because the “binding” of the current “context” (like, inside a method which variables are visible, what are their values, etc) is first-class - meaning that I can “store” this binding in the nREPL side and then evaluate code as if I was still running that method (again, huge debugging capabilities here!). I don’t know how much IEx exposes, but considering that Elixir is a functional language, maybe we don’t even need these “hacks” because we don’t need to create a class, then set attributes, mutate stuff, before we can actually run some function :slight_smile:

7 Likes
7

I remember there was an editor for clojure that allowed viewing of every variable of a call. Rather than doing an IO.inspect and re-running, the last variable was available.

It was a combination of debugger + editor of some kind…I think it was LightTable. Or perhaps emacs also supported it, but it has been roughly a decade, so I don’t remember anymore.

It seems that nREPL could support something like this, do you know if this type of functionality has been implemented for other languages?

8

I was thinking to approach it with the bare minimum, either json or just plain string(that would be elixir code), this way the editor could just work with interpolation and just use the responses back directly in the editor, with no encoding/decoding overhead.

I was thinking thinking to approach that with delegating it to the nrepl itself… since elixir has the Code.Fragment module. the idea was to send the current entire buffer and send the line+column that your command was being executed.

I’ve actually just copied iex code and i’m trying to make it work with just simple socket and binaries for now, the worse part is that iex expects an executing shell to do it’s work(i guess it relies heavily on the erlang shell if i understood it correctly). i’m thinking to explore livebook code to see if it would be easier to start from there. I wish I had more free time to work on that :pensive: i’m planning to do a POC of this nrepl based on iex to write a good proposal to send in the language mailing list

1 Like
9

LightTable did offer a “watch variable” (I believe) that allowed you to check the contents of the variable, but you had to watch before running the code.

I know FlowStorm (not an editor, but a different kind of debugger) allows you to inspect everything without re-running, and it’s still in active development.

Lazuli (my project for Ruby) does basically the same - it allows you to inspect the contents of a variable without re-running anything, but it’ll show the variable after things happened - meaning that if you have a parameter called a, and that is an empty array, and that gets mutated over and over (even outside the method) the only info Lazuli will have is the final, mutated version - not the original parameter (don’t know if there’s a way around that, to be honest).

1 Like
10

@joelwallis just pointed me to this

are you around @mjrusso ? :eyes:

1 Like
11

It’s a shame there’s generally not more interest in this kind of thing, but what can you do ¯\(ツ)/¯ I used the REPL constantly when I was working in Rails, especially for debugging but also for spiking. I do find myself just doing puts debugging most of the time in Elixir since state is much easier to follow in FP. I also find myself spiking less in IEx because it’s a pain to work with for that purpose. This is frustrating because IEx is so good otherwise.

Perhaps it’s the idea that this could compete with LSP? Just guessing here. The cross-editor/cross-language is a very important problem LSP (mostly) solves, but for those of us who are power users of a single editor we never plan on ditching and only work in a few languages, the possibilities that this would open up are exciting.

2 Likes
12

Hi :wave: Thanks for pinging me. Some really great discussion here.

(For posterity: I just updated the nexREPL README to include a link to the simple demo I shared earlier today on Twitter, and to also link to this thread.)

Having been immersed in this for the past few days, a few thoughts:

  • The biggest advantage of working in the existing nREPL ecosystem (IMO) is access to the editor-specific clients. However, after surveying the landscape of existing client implementations, and excluding the ones that aren’t Clojure-exclusive, there’s not a lot of options.
  • Eval’ing the expression at the given point/cursor position is pretty essential. The clients I tested don’t send all the necessary information over-the-wire to properly implement this on the server (there are optional parameters in the nREPL protocol for the “eval” operation for the file name, line number, column number, etc., which conceivably could let you put the logic on the server, but you still run into trouble with handling unsaved files etc).
  • All that being said, the relative simplicity of the nREPL protocol means that it isn’t a huge stretch to build a custom client. But at that point, why not build something fully custom that directly integrates with IEx and doesn’t have any Clojure baggage?
  • I am bullish on the idea of mediating this through LSP (as described at a high level in the nexREPL README), where LSP is the protocol that a custom “eval code” code action is sent through, the LSP server figures out the context and exactly what code you need to eval, and then sends the code directly to an existing IEx session. This of course is only one path of many options, but it solves the client issues and could reasonably be something built in to a language server (which would really help with adoption).
5 Likes
13

@mauricio.szabo is working on an extension for VS Code and Pulsar (Atom revamped) which allows the use of any nrepl language, it’s a matter of extending the plugin and it’s already working with Ruby. More info about how he did that:

https://www.reddit.com/r/Clojure/comments/1fjokt9/london_clojurians_talk_grinding_parenthesis_to/

1 Like
14

Great talk, and extremely impressive work by Mauricio!

(I’d imagine that Elixir-specific resolvers would be a lot easier to write than the Ruby versions, too, given the differences in language semantics/runtime.)

I wonder if it would be possible to delegate that work to Tree-sitter. If I’m understanding correctly, the various components of that extension are implemented in ClojureScript, and there are WASM bindings for Tree-sitter.

1 Like
15

As a totally alternative approach, I recorded a demo video showing how a REPL-over-LSP implementation could (potentially) work:

https://x.com/mjrusso/status/1837201282932633654

The benefit of this approach is the ability to leverage existing clients in your editor of choice. And the LSP server is already set up to understand your code and the context around the code you’re editing, so (I would imagine) it’s not a huge stretch for it to figure out exactly which code it needs to re-evaluate.

Worth noting: there were some concerns raised on Twitter about this LSP idea, particularly related to LSP server performance/overhead/etc. The nREPL protocol is certainly more efficient over-the-wire, but I don’t think it’s a big enough difference to matter in practice. Of course, running the LSP server itself can be extremely taxing, but the proposal is more about the client and the protocol: it should be possible to make an LSP server that essentially only does the code evaluation bits and is thus very lightweight.

1 Like
16

If this LSP approach is viable, and we implement a new “eval code” code action (as I’m proposing) in the new/official Elixir LSP, and the Elixir LSP is included in every Elixir release, then that means that every Elixir developer can have these capabilities with minimal setup.

(A lot of “ifs”, I know, but this does seem in reach.)

1 Like
17

@mjrusso problem of integrating in LSP is that we don’t actually solve client issues. I don’t believe any LSP client that exists now expects an “evaluate” operation, and while that can be added, clients don’t actually have any logic for this kind of interaction to happen - meaning there are no UI design to handle evaluations, for example.

The great advantage to reuse some nREPL client is that if the client/plug-in doesn’t send all the information, the only actual change they need to add is to send that info - every other task related to “evaluate something and get back a result”, like some “awaiting result” state, or a command to “break” the evaluation, are probably already present. Piggiebacking on something that exists, even if it needs some adjustment, can be a huge advantage compared to start something fully custom that will need to start up implementations from scratch.

I know this is kind of a weird chicken-and-egg situation, where we need client support to justify a server implementation, and we need server support tonikplement the ckient side too :slight_smile:

3 Likes
18

Great news then - I’m cureently using TreeSitter to parse Ruby in the Lazuli project :smile:

3 Likes
19

I’m stuck in a bunch of stuff that I already started, but I was planning to glue together your implementation of nrepl with the auto-complete/intelisense from LIvebook(that part of the code is pretty much isolated) to show how we could get a better autocomplete with nrepl.

I understand the idea of leveraging the already existing LSP infrastructure as a starting point, but as a long term thing it feels like diminishing all the power that comes from an nrepl approach(that is you being already inside in a live system).

2 Likes
20

I pushed up a working (but lots of rough edges) REPL-over-LSP implementation (leveraging GenLSP by forking Next LS):

Demo video here: x.com

It’s important to note that I didn’t modify Eglot (LSP client that ships with Emacs), besides telling it where to find my LSP server.

@mauricio.szabo wrote:

@mjrusso problem of integrating in LSP is that we don’t actually solve client issues. I don’t believe any LSP client that exists now expects an “evaluate” operation, and while that can be added, clients don’t actually have any logic for this kind of interaction to happen - meaning there are no UI design to handle evaluations, for example.

The great advantage to reuse some nREPL client is that if the client/plug-in doesn’t send all the information, the only actual change they need to add is to send that info - every other task related to “evaluate something and get back a result”, like some “awaiting result” state, or a command to “break” the evaluation, are probably already present. Piggiebacking on something that exists, even if it needs some adjustment, can be a huge advantage compared to start something fully custom that will need to start up implementations from scratch.

I know this is kind of a weird chicken-and-egg situation, where we need client support to justify a server implementation, and we need server support tonikplement the ckient side too :slight_smile:

To clarify my position: I think LSP support is a net positive because it makes it extremely easy for LSP users to experience the joys of REPL-driven development. That being said, I would personally use a better, more focused tool if one were available that worked with my editor-of-choice. In my ideal world we’d have both kinds of options :slight_smile:

A few more notes about REPL-over-LSP:

  • Elixir is pretty unique because (ironically) there’s no need to expose a REPL UI in the editor: you can already IEx into whatever node you want.
  • I don’t think that most of the UI issues you mention are blockers. LSP supports showing arbitrary messages, for example (although of course it didn’t work in my demo video linked above, but I have seen it work while I was implementing this… I must have broken something at some point, or hit some bug somewhere).

All that being said, you can absolutely do a better job with a focused tool, and nREPL is certainly better.

@cevado wrote:

I’m stuck in a bunch of stuff that I already started, but I was planning to glue together your implementation of nrepl with the auto-complete/intelisense from LIvebook(that part of the code is pretty much isolated) to show how we could get a better autocomplete with nrepl.

Amazing, I’m really looking forward to seeing this!

I understand the idea of leveraging the already existing LSP infrastructure as a starting point, but as a long term thing it feels like diminishing all the power that comes from an nrepl approach(that is you being already inside in a live system).

Just a small point of clarification, the two approaches are the same in the sense that you also get an equivalent live system in the LSP case. (At the cost of more moving parts: the LSP server acts as a sort of proxy for eval’ing code on a networked node.) I explain how this works in this video: x.com

1 Like