Ideas to streamline external package management

Which problem though?

Nix is nice; however, I’d like the BEAM world not to standardize on something outside the scope of BEAM. I don’t even use ASDF; apt + compile from source is good enough for me.

The recent example I can think of concerns the XLA binding (GitHub - elixir-nx/xla: Pre-compiled XLA extension · GitHub) which, depending on an environment variable, retrieves a pre-built tar.gz version of the library or rebuilds the library requiring many external components with a specific version of Bazel, Python and Numpy.

Nix solves 2 of the potential problems which are reproducibility by providing build level isolation and build implementation details that do not leaks in the user space. And caching by providing standardized reproductible cache based on hash: Cachix.

https://twitter.com/josevalim/status/1417195238393892868

Another which I didn’t think deeply about is the NodeJS/ESbuild issue, by shifting the system level package from the user to the library provider guaranteed by a declarative and cryptographic hashes of the dependencies (How Nix Works | Nix & NixOS)).

I believe, from my experience in production projects (not specifically in Erlang or Elixir) with many people, with many operating systems and with system level dependencies like protoc (Protobuf compiler), protoc-gen-go (Protobuf plugin for Go), Bazel, Go and GCC.

Or a project involving the computation of atmospheric satellite images that depended on Fortran and some system libraries that doesn’t exist on target deployment operating system package manager.

A good enough way is not viable in an environment such as software development, where systemic changes are constant.

This is my point of view and my experience. Please feel free to correct me or challenge my arguments

There must be some misunderstanding here. I merely say I don’t need it, I never challenged you for whatever setup you prefer for your need. You can already use Nix for what you need to do. You are even welcomed to write a tutorial about how to use Nix to make everything fit together. I only object to make certain setup mandatory.

Thanks for sharing a possible solution! I have one question: how does it work on Windows and macos? If it doesn’t also have first-class support on those platforms, then it is unfortunately a no-go.

I jive in. There are a good dozen of Nix BEAMers. I’m one of them.
And for projects where needed (projects with external dependencies) it really works like a charme for me.
Even for projects with PosgreSQL I could spin up a development environment with one command.
And with pinning Nixpkgs (the versions used) I’m 100% certain a project will build again in a couple of years without searching for legacy dependency versions.

On excellent macOS support is worked on as far as I know.
Native Windows support is not something that will be finished soon.
As Windows user I use WSL2 for the projects with Nix. Pre WSL2 a had a virtual machine running myself.

So no I don’t see nix getting mainstream soon.
But for me it’s the best thing to get for this kind of dependencies problems.

What I could see soon, we the community will start writing some guides for developing / deploying with Nix somewhere in our projects.
But I would still wait until Nix flakes are ready.

As mentioned by @webuhu, in recent years I have also been looking for a set of tools that are compatible on all operating systems. The Windows side is still the most complicated, even historically no official production grade package manager like APT, Nix or RPM was available, even today… The situation with Winget, chocolatey and scoop is still far from being a good solution.

To my knowledge, Nix does not have first class support for Windows, which I don’t know enough about to discuss in detail, but very good support for Unix-like distributions, including Darwin (MacOS) which I can attest to for years.

Taking into account the general direction of the development of Windows itself lately I’d just presume that people will want to use WSL2 and maybe put Nix inside. First-class support of a good package manager becomes less and less likely with time IMO.