Nix vs asdf for Elixir version management

Chat / Discussions
1

I have discovered Nix last month and I am currently on my way to migrating to it—both on macOS at home and the full NixOS distrubution at work. Now, I use it for configuration and package management, but I am considering to switch my development workflow to Nix too. Yet, I have some questions regarding the Elixir environment.

I am presently using asdf to manage Erlang and Elixir versions. I like:

  • its ease of use,
  • the availability of many versions, including pre-releases and all patch versions,
  • the way you can set a given version for each project thanks to .tool-versions,
  • automatic context switch between directories.

The only drawback I find now is the Erlang compilation dependencies management. I’ve set up an Elixir environment at work on several Linux distributions, and each time I missed some dependencies on the first run. I’m not even sure if I could manage to build Erlang using asdf on NixOS since—as far as I understand—libraries are not available outside of a nix-build. I have tried to do it in a nix-shell, but it kept complaining about OpenSSL not being available, despite it being installed and set in LD_LIBRARY_PATH.

On the other hand, using Nix as part of my Elixir workflow could address these issues. As all build dependencies are automatically managed, there would not be Erlang build issues anymore. A simple shell.nix in my projects would tell what is needed. From my first trials:

  • it is possible to set a given Erlang / Elixir version using a package like beam.packages.erlangR21.elixir_1_7;

  • writing a shell.nix lets you tell other developers what’s exactly needed. It is even more powerful than .tool-versions as you can set other binary dependencies and try reproducibility thanks to nix-shell --pure;

However:

  • there is no automatic context switching. You can then forget to nix-shell and use mistakenly a globally-installed Elixir version instead of the project one. Maybe it is just a habit to get used to, and frequent Nix users do this without thinking about it;

  • only the last patch version of each minor version is available, and it seems there are no pre-releases. This can be problematic, especially if you maintain libraries and want to test them on versions to come.

  • new versions seem to be available after a while compared to asdf.

In addition to these points, I think using Nix for projects is not as easy as asdf when it comes to contributions: asdf is easy to install at the user level and get into, where you need to sudo to install Nix.

Are there some Nix / NixOS users here? What do you think about Nix in the Elixir workflow? How do you address the few points I have noticed? And if you are a NixOS user, how would you use asdf if third-party project uses it?

8 Likes
2

I am actually in the (very slow) process if making the BEAM work better within Nix (I have posted about this elsewhere on the forum before, maybe look for that for some more information). I am writing a tool right now that will convert a rebar3 or mix project into a default.nix file similar to tools for other languages.

Automatic context switching

Like you said, there is none. You will just need to get used to calling nix-shell or some other tool to get where you want. But this isn’t necessarily a bad thing either. You are being explicit about what you want.

Limited versions

This is also true. But I don’t really see having only the latest patch for each minor version available. Everything should be backward compatible (assuming you are not relying on internal functionalty, but there is already a post about that elsewhere). So it is just bug fixes and such.

Time for a new version

Again, true. This is mostly true about any OS package manager. Nix needs to make sure that all of the packages in the ecosystem play nicely together. With that said, I did make it super easy to make your own version of Elixir if it is not found in nixpkgs. Just take a look at how we define v1.7 and the corresponding entry to actually build it. If you need a version that is not listed or has not yet been updated. Feel free to add your own for your project!

Contributions

Just like you do not need asdf installed to contribute to a package that has asdf, you do not NEED to install Nix to contribute to a package that uses it. Just make sure you outline your requirements in your contribution guide.

For the vast majority of uses, I would currently say to stay away from Nix and NixOS for BEAM specific work. Or if you want to use NixOS, use it like a regular linux for the time being (I am doing this for a couple things).

For anyone else that REALLY wants to use Nix and NixOS right now, send me a message and we will see if we can get the BEAM integration moving a little faster.

3 Likes
3

I’m a relatively happy Nix user in general, including on OSX and as a Linux desktop via NixOS, but I would be very unhappy teaching most people to use it, even those who are able to effectively adapt to both Elixir and ASDF. It’s a lot to ask for little return unless you already find it academically interesting. It has a lot of parallels in my mind to my use of Emacs - I personally find it very powerful and compelling, but struggle to articulate well to others who are skeptical, or who have different levels of tenacity in the face of ~user-unfriendly~ complex software.

I will also say that very specifically the abstractions used in the BEAM packaging within nixpkgs are over-abstracted from my POV. They are probably very useful/powerful to the internal/official maintainers, but are sort of a nightmare to consume as someone who is literate with the syntax but not an expert. If you’re not sure what I mean, try to figure out how to use a Nix overlay or shell.nix to install a particular patch release of Erlang and of Elixir, without it having been packaged already. You can’t use a simple override, nor have I found an easy way to otherwise “patch” the URL and checksum of the target download like I can with other languages’ Nix packaging.

I ultimately fell away from using Nix specifically for the BEAM ecosystem and back to ASDF, even when I persist with Nix elsewhere.

For those unfamiliar, here’s an example of shell.nix content:

# 2018-06-26 11:30
# https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.09pre143801.5ac6ab091a4/nixexprs.tar.xz
with import <nixpkgs>{};

let
  erlang = beam.packages.erlangR21.erlang;
  elixir = beam.packages.erlangR21.elixir;
  nodejs = pkgs.nodejs-10_x;
  yarn = pkgs.lib.overrideDerivation pkgs.yarn (attrs: rec {
    buildInputs = [pkgs.makeWrapper nodejs];
  });
in

stdenv.mkDerivation rec {
  name = "my_elixir_project";
  version = "0.1.0";
  buildInputs = [
    erlang
    elixir
    nodejs
    yarn

    # phoenix_live_reload
    pkgs.darwin.apple_sdk.frameworks.CoreFoundation
    pkgs.darwin.apple_sdk.frameworks.CoreServices

    # postgrex
    pkgs.postgresql100

    # developer tools
    pkgs.direnv
    pkgs.gitMinimal
    pkgs.hex2nix
  ];
}
4 Likes
4

There are some natively supported but non-obvious ways to pin back to a historical version in a specific shell.nix context. This is a sample based on the mailing list, and you’re ultimately required to track down a GitHub SHA for the timestamp within nixpkgs that has the exact release you want. This can be used concurrently with other packages from different timestamps within nixpkgs history, but you run the likelihood of maintaining several concurrent “trees” of the underlying dependencies that take up some multiple of the space consumed by only 1.

{ pkgs ? import <nixpkgs> { } }: let

  myPkgs = import (pkgs.fetchFromGitHub {
  owner = "nixos";
  repo = "nixpkgs";
  rev = "f607771";
  sha256 = "1icphqpdcl8akqhfij2pxkfr7wfn86z5sr3jdjh88p9vv1550dx7";
}) {};

in pkgs.stdenv.mkDerivation {
  name = "pandoc";
  buildInputs = [
    myPkgs.pandoc
  ];
}

Regarding space consumption, you can minimize this somewhat with nix-store --optimise -v to perform hard-linking, if the contents of those dependencies hasn’t changed much or at all. It’s really valuable to learn how the nix store and garbage collection work, because I often find I have several gigabytes of redundant files stored, even with directives like this in my /etc/nix/nix.conf:

gc-keep-outputs = true
gc-keep-derivations = true
env-keep-derivations = true

auto-optimise-store = true
1 Like
5

Thank you for your replies!

I have effectively seen some tools like this for languages like Rust or Haskell. What is the point compared to just set up the environment in a shell.nix and using mix?

How do you manage to use asdf on NixOS to build Erlang? Do you install build dependencies globally? Do you have a list of such dependencies somewhere?

6

I’ve used something comparable from the Go ecosystem to allow me to quickly package a Golang project I don’t manage. Were I a developer on the project, I wouldn’t tackle it that way, but I actually just want to use its included CLI binary without a bunch of non-reproducible go get ... commands, not hack on the project. That may be the motive in other scenarios too, or it could also be handy if you’re deploying your project from source directly via Nix and only Nix, rather than using Nix → Mix, or something like Distillery.

I’ll try to test this tonight and let you know what I have in /etc/nixos that appears to be making it work. It’s not my primary machine.

I’m still gathering my thoughts, but based on @Ankhers post above I did get cranky enough to work my way through nixpkgs source and nix repl to figure out how to call the same functions from the source files he linked. This should allow us to get a minimal shell environment of any arbitrary Erlang and Elixir versions with a shell.nix and two secondary .nix files. I’ll try to share that here too. And if there’s a way to do it one single shell.nix, I’m all ears! Here’s what I’m working with so far:

2 Likes
7

Oh, regarding your custom versions draft, this seems really neat. I was just playing around with nix-shell, editing my .zshrc not to import again PATH-modifying scripts so that I can keep some useful aliases and override my environment. I will definitely give a try to Nix to setup the environment, using custom derivations as needed.

Thank you very much!

Ok, I better understand like this. In a build / deployment process it is easyer just to have to nix-build indeed. I was thinking about the development process where one typically prefers to use tooling from the language ecosystem.

8

I have played around a bit and come to this shell.nix for Elixir 1.7.0-rc.1 using Erlang 21 from nixpkgs:

I’ve passed rebar from beam.packages.erlangR21 to the Elixir builder to avoid depending on Erlang 19—on which rebar depends otherwise.

This seems very promising. My next step is trying to setup a full environment for Nerves.

9

For those interested, I’ve set up a an environment using custom versions for Elixir and Erlang and some dependencies for Nerves:

Again, I’ve defined a custom rebar using the compiled Erlang instead of the one from nixpkgs.

To avoid the shell dependencies being garbage-collected, I’ve found you can do:

$ nix-instantiate shell.nix --indirect --add-root $PWD/shell.drv

With keep-outputs = true in your nix.conf, this prevents such long to build derivations to be collected. When you don’t work anymore on the project and want to get rid of your shell dependencies, you just have to delete shell.drv and run the garbage collector. Hope this will help :slight_smile:

By the way, I don’t find how to edit my previous post to avoid this kind of double-post.

2 Likes
10

An update for future readers: I have completely switched from asdf to Nix lately. It is a joy to use to setup a complete environment, including non-language dependencies you can need to build different kind of projects. I’ve set up standard shell.nix for both standard, Phoenix and Nerves projects. I have also written an article about it which I hope will be useful to Nix newcomers.

19 Likes