Deployment with executable (compiled Go) dependencies

Questions / Help
,
1

Hi!

I would love to learn more about Elixir by moving over my website backend from Node.js, however, there is one obstacle that is providing some friction.

I’ve heard that Elixir is not great for raw computation (neither is JavaScript, to be fair), and doesn’t seem to have any solid image manipulation libraries/wrappers such as sharp. I made my own little library for Node.js called IPP that provides wrappers around pre-compiled binaries (Go) that are distributed via npm. It works for 99% of cases. The advantage is that you don’t need a Go toolchain, just pull in the package and run!

This is something I would really miss when moving to Elixir, an easy way to run something like Primitive. Ports seem like a good way to spawn a foreign process, but they expect the executable to be available/pre-installed. IPP tries to make this as easy as just installing a package, as the pre-compiled assets are stored in node_modules in the respective @ipp/... package.

Is there an elegant solution for this in Elixir with Mix? I’m very new to Elixir/Erlang concepts. Phoenix can deploy static assets (HTML, CSS, …), is there any parallel with other foreign code? The ideal solution would be to have a small Go module in the Phoenix project that can be compiled as a wrapper around the library I want to use, which I could deploy along with the built Elixir application and run on the server.

I was thinking about using Docker, it would allow for bundled dependencies (executables), without requiring the server to already have the library installed globally, such as with Mogrify. Although from my understanding, Elixir/Erlang wasn’t meant for Docker, and runs well on a VM on the host machine (for hot code swapping, for example). Are there any other downsides?

Or perhaps a several-step deployment process, where the executable is compiled to /usr/bin and then Elixir is deployed? I am kind of opposed to the idea of polluting the host system path with application-specific executables. WebAssembly looks promising, but the actual interface (transfer of settings and binary data) would be complex (and it would be easier from Rust than from Go, I’ve actually been considering re-writing IPP in Rust so it can be potentially used from any language ecosystem, as it has fantastic WebAssembly support, but that needs time!).

The intended purpose is to process uploaded thumbnails, another solution I had in mind is to have a small Node.js serverless function in the cloud that handles image processing that Elixir could call over the network, but it of course has a dependence on the cloud (and increased cost).

Thanks for your help!

2

Maybe take a look at the new esbuild stuff, it also downloads a compiled binary. From what I can see they have a mix command to download the binary. But I don’t see why you could not distribute the binary in the priv directory and just reference it from there by path.

3

The difference with ESBuild is that it is run before/during Elixir compilation in order to process static assets (HTML, CSS, JS) which are then deployed along with Erlang/BEAM code. From my understanding, there is no need for ESBuild to be present in the production system, as it doesn’t need to be run after deployment.

I would need an executable runtime dependency, which should be compiled along with the rest of the application (I played around with a Mix Compile task, with little success so far… I’ll take a look at the example you sent). I was not aware of the priv directory (yes, very new! perhaps because I disabled the web-part of Phoenix just only have an API), but come to think of it, if static assets can be deployed, a similar mechanism could work for executables, as long as they are not served to the browser by accident…

I’m trying to avoid my Node.js/npm way of doing things (where packages are just tarballs that can contain any binary files) in case they are a strong anti-pattern in Elixir. I assumed that static assets were embedded within the BEAM code so that the application can be deployed in a cluster without having to the assets to every machine.

Thanks!

4

I know esbuild is not run during runtime, I was merely suggesting looking at it to figure out their approach, you could write it so your application attempts to download the go binary if it is not already present in the system. But yeah I think I would look at the priv directory first, that seems the most appropriate to me.

1 Like
5

You can look at any project using elixir_make or rustler to compile C/Rust code within an elixir project. There’s also rambo using both rustler compilation as well as shipping precompiled binaries.

2 Likes
6

I did stumble across Rustler, it looks promising! I’m not sure if Go plays well with ABI-style code execution, this is something that C and Rust are traditionally really good at, but thanks for the input, I’ll check out the other projects you mentioned. Pre-compiling and shipping the binary alongside BEAM code is perfectly sufficient, I’m just looking for the most ergonomic way to do that.