The (unloved?) cli story

Hey folks,

Elixir offers great capabilities for the web, data processing or other kinds of long running daemon, no doubts. But I wonder why the cli space is so uncared or unloved? Or do I miss something? :sweat_smile:

I am pretty sure the most of us use cli tools on a daily base. But I did not find a lot of traces like libraries for or already available cli tools in Elixir. escript and OptionParser are already good starting points but I think this is not enough. And Mix tasks are great only for the development of Elixir projects (from my point of view).

In addition, Elixir “apps” could be more easily distributed to “non-beam”-users since relase was added to Mix. Also Mix.install allows to build more sophisticated scripts which could use cli capabilities.

I would like to initiated and carrier the cli creation story but I wonder if I oversee something.

A :heart_decoration: for cli tools

2 Likes

The issue with cli tools in elixir is the startup time of the BEAM is not trivial in the context of a cli tool. It takes about half a second to launch the BEAM, which means that certain classes of cli tools which require quick feedback are not going to feel very responsive. As well, releases are not as portable as you might think they are. They do bundle the Erlang runtime, but they call out to system libraries which must be the same as the ones for the OS you compiled on. In practice this means compiling on the same operating system and same version of the OS as you intend to deploy to. This makes cross compilation a lot more difficult than other languages like go, Rust, zig, etc which have a more modern cross compilation story.

That said, I do believe there is a niche for elixir apps in the console space. Similarly to a web app, if you create an interactive tui / ncurses style cli app, then the startup time becomes a negligible part of the overall runtime of your system. I have played with Ratatouille, which is a library for building these apps and I quite like it, it has some warts but it’s actually quite a nice way to build console apps. Based off of the elm architecture if you’re familiar.

I would love it if we could get the startup time of the BEAM down to the point where it wouldn’t make a difference for a cli app, but just due to the nature of the systems Erlang and elixir get chosen for I doubt that anyone will be willing to work on that any time soon.

8 Likes

Depends on what you call CLI. My “CLI” program consists of a daemon listening on a known UNIX domain socket and a shell script wrapping around socat to talk to the daemon. So far my users have been content: instance response and rich functionalities.

1 Like

I think for the majority of cli tools this is fine. A half second is low enough to feel snappy. But I also think this is the biggest bump. I developed a small cli for measuring my work time and it took 3 seconds for starting. :-/

I am aware of this and think this is manageable. At work I developed a cli tool in Python for other (Java) developers and no one wants to install Python. So I made use of something that bundles the cli with the runtime like Mix.release does.

I am with you. Maybe it is possible to turn off some features for more speed.

Something that uses the command line interface :stuck_out_tongue: . In your case it is the script around socat.

It really depends on what you’re building, who you’re building it for, and how often it will get called.

Imagine if fzf or ripgrep required 0.5s to start! They would be unusable.

Building CLIs in Elixir is quite easy and viable – requires a little boilerplate and obeying few idioms blindly but it’s a very little sacrifice for good benefits. I’ve been happy with the CLI tools I built in Elixir but still, have these in mind:

  1. Elixir CLI is viable, very desirable even, if it integrates with your project’s data schema or other dependencies that you cannot easily reproduce if you reach for a mega-fast CLI-friendly languages like Go, Rust, Nim, Zig, D, C/C++ etc. If you need stuff that’s much easier and quicker to generalize then using Elixir CLI is not technically justified – it’s only justified if your team is more junior and people don’t want to reach out to other languages.

  2. Elixir CLI’s startup time is workable if it’s, say, a cron-plugged script that gets called at the top of every hour, or even every 15 minutes. It’s absolutely not workable if you have a shell script that calls your Elixir CLI 100+ times. Those halves of a second will add up to huge runtime.

  3. Elixir CLI is desirable because the language itself is concise and it is easy to read, and this hopefully provides your team with an escape hatch to modify it – should you leave the company. But that’s not specific to Elixir IMO. Lots of other languages can achieve the same, including the shell script itself.

Not to be dismissive of anyone but my observations is that Elixir CLIs are usually created out of a fanboy-level love of the language. Or because the team is junior and prefers to work with a single language. Or it’s too hard to duplicate a lot of functionality in another language.

If none of these are a factor in your situation then I’d strongly advise you to reach to another language. Most of the time people vastly underestimate the utility and usability of their tool and before you know it, your tool will become popular and your GitHub issue tracker will be flooded with “your program has a huge startup time”.

All other things being equal – aim for a future-proof tech stack. Elixir isn’t a good fit for CLIs in many situations.

2 Likes

that’s just 2 lines of shell script though. All the option parsing, computing and result printing and formatting in my case is in the daemon.

Not sure if it matters, but OS X and BeamJit does not play well when it comes to upstart times at the moment. On my OS X laptop it takes about 0.4 seconds to run an escript doing nothing, while on my Linux machine it takes 0.1 seconds.

If you compile Erlang/OTP 24 with --disable-jit the startup time on OS X drops to 0.1 seconds as well, but then you don’t get any JIT so it all depends on what you are doing next if having the JIT is worth it or not :slight_smile:

0.1 seconds is still a lot more than what a what a native program would do.

3 Likes

Could that become a general pattern thing? Basically using Elixir for daemon-level CLI tooling.

The obvious example here would be Docker, or any kind of language server.

I can see how packaging this into a library with some mix tasks to do the bootstrapping could be useful.

2 Likes

So I think this is already a thing. The weak spot is the client. And for some use cases a daemon is to much.

That is amazing. I will give it a try.

I am not sure if this is a typical use case for any cli tool.

I would say yes - in most cases you use these tools in scripts, so each invocation of external tool (which is almost everything in "original sh") would quickly add to considerable amount of time. Additionally if you are using this tools from within any other system, for example checking processes status on *BSD, then you probably will run such tool over and over again in short periods of time. Other examples involve for example xargs or CI runs.

2 Likes

I think of CLI tools as (mostly) falling into two (very different) categories: ‘Unixy tools’ and ‘interactive console apps’.

A Unixy tool absolutely would be (potentially) callable 100+ times (per second/minute/hour). The advantage of creating these tools as a ‘CLI’ is that they can then be combined with all of the other (existing) CLI ‘Unixy tools’, as well as being used by anything that can itself work with ‘Unixy tools’. The most recent example of these (that’s being used in an Elixir project I work on) is a tool to convert an HTML document to a PDF document.

(If there was an alternative Elixir library/app for these kinds of tools, I wouldn’t think there’s much reason not to use them (in another Elixir project), assuming all else is equal (e.g. the tools are relatively ‘good enough’ as is). It is of course often useful that these tools can also be used by all of the other Unix tools or any other programming language/environment/etc. that can also interface with them, which is almost all of them.)

‘Interactive console apps’ on the other hand are often (almost always?) NOT also ‘Unixy tools’. Tho sometimes CLI apps can work in both/either mode, e.g. some Git commands.

I don’t think Elixir is a bad language for interactive CLI apps but I found the ‘console graphics’ libraries to be pretty rough (tho I was testing them on a macOS computer and NOT a ‘regular’ Linux box). Issues like, e.g. BEAM startup time, are definitely less of an issue for these types of apps/programs.

4 Likes

I have played with Ratatouille too. Here is a tool I’ve built using it → reorder_migrations.exs · GitHub. I agree, it is a nice way to build console apps. It has high-level components, however, I still had to write some low-level manipulations just to display content nicely. While doing this I realized I miss web browser behaviour :smiley: Unfortunately, the library seems to be abandoned. termbox, which is a heart of the Ratatouille, has the following banner in its README

This library is no longer maintained. It’s pretty small if you have a big project that relies on it, just maintain it yourself. Or look for forks. Or look for alternatives. Or better - avoid using terminals for UI. Never the less, thanks to all who ever contributed.

2 Likes

I discovered the same thing when I was considering making an app (a game) as an (‘interactive’) Elixir CLI app – the best CLI libraries were either un-maintained or didn’t seem to work on my computer (a Mac). The other main option I found was to ‘roll my own’ using Elixir’s built-in console features (which are very nice), but that seemed likely to be a lot of work.

In the end, I opted for a simple web app instead – the devil you know and all.

The more I read, the more I agree. My question was asked with interactive CLI tools in mind. It seems to be clear that this unixy tools can not be made with Elixir. On the other hand, interactive tools are still doable in my opinion.

I found bakeware and it looks very promising for packaging. A simple “upper case” app can completed in 0,4 seconds. The file was 13 mb big.

1 Like

Better known as TUI (Terminal User Interface / Test-based User Interface), by the way.

1 Like

I think TUIs are a subcategory of interactive CLIs. For example openssl can also be very interactive.

I don’t think it’s quite the case that Unixy tools can’t be made with Elixir. If the tool was expected to take a substantial amount of time (many seconds or longer) to run, in most cases, then things like BEAM startup time probably wouldn’t be a big deal. But even then, it still might make sense to write one in Elixir if all of the ‘costs’ of doing so are outweighed by, e.g. using a ‘nicer’ language.

But yeah, interactive CLIs are definitely doable to an extent that Unixy tools are definitely more dicey. I’m pretty familiar with Phoenix, and the web is a really nice UI/UX target in a lot of ways, and lots of { free / open source } tools are packaged/distributed as a standalone web-server+web-app (to be run and access from a single computer), so, if anything, it’s harder to find an ‘obvious’ pick among all of the available options for some projects.

Bakeware does look nice – thanks!

1 Like