Your experiences with integrating CLI tools in an Elixir app

Chat / Discussions
1

My experience being a web developer showed me a lot of examples where a core part of a web app is calling shell command and then dealing with the result.

Best example is imagemagick. Its capabilities are pretty much unmatched by any native implemented image processing tool. I know imagemagick is usually available through a language library. But sometimes these libraries are just a wrapper for the CLI arguments.

Furthermore often times there are other tools that are used just by calling a shell command. There are a ton of command line tools that are just called from the web stack and then the string result is parsed in the app (pdftk for PDF management, tesseract for OCR, Ghostscript, libreoffice, sometimes even curl).

So my question is. What is your experience with Elixir/Phoenix apps where you integrated various CLI tools? Any problems? Any best practices?

2

Port is awesome. I often write CLI tools in other languages then use Port and ExUnit to do unit test on them. There are a few thing to keep in mind though:

  • There is no back pressure. For most CLI tools this is not needed; if you really do there are more advanced libraries out there.
  • If you have 2 way communication to the CLI you need to make sure there is no dead lock
  • If the CLI part is not behaving well (don’t terminate when stdin is closed) you may have to kill them; or there will be zombie processes

Edit: Actually, write to a port in elixir/erlang will never block. So this is eliminating the most common case of dead lock. Unless you run out of memory, of cause. This is a huge benefit.

3 Likes
3

I actually use erlexec | Hex instead of straight Ports for programs that aren’t designed specifically to be Ports. Ports expect the program it connects to to behave a certain way, and if it doesn’t you can get zombie processes and other things (which trying to actively kill them after doesn’t always work either, especially if the program forks). ErlExec above contains a program that is opened via a Port, and then that program manages the program you speak to, ensuring none of the broken stuff can happen (as long as you follow it’s docs).

But yes, I use erlexec quite a lot, highly recommended, and yes imagemagick is one of those things I call out to (among ghostscript and a dozen other things). ^.^

3 Likes
4

I’ve got job-runner app that uses Rambo. Runs bash and ruby scripts that do backups and collects metrics. Have run about 500K jobs over the past 9 months.

1 Like