Tooling fragmentation and separation of Elixir from rest of BEAM family

I would personally opt for the standard erlang tools, but I don’t see it happening, Elixir likes to duplicate how Ruby does a lot of things so the design of things like mix are different.

Distillery is actually pretty nice though, I think the author is even the author of one of the old Erlang tools.

As for ExDoc, the latest OTP versions have support for a new doc chunk that makes using whatever documentation format you want a lot easier.

/me would prefer if mix were just a thin (really thin) wrapper around rebar3 though

Mix was created before I started working with Elixir but I think it’s more based on leiningen rather than anything else. What in ruby is it similar too?

Maybe rebar3 should have wrapped mix since mix was created before rebar3?

Could be, I wouldn’t mind, as long as there is single tool used through community (with possible thin wrappers for “nicer feel”. Like in Java world where there is Gradle that is built in Groovy and became de facto standard in Android development.

The problem I see however is that this would require whole Elixir to be distributed together with such tool. I think that this is the dealbreaker there.

To clarify myself on the topic:

I do not think that we should ditch our tooling in favour of Erlang one, what I am saying is that we should work to uniform it. I do not care which “side” will “win”, what I care is that there will be common tooling in family. Question is how we could “promote” our solutions or how we could adopt solutions that are outside of our world. Example of such integration is EEP 48 which brings common documentation store to everyone, which is great. Now we need to think about rest of the places where we can establish common ground.

Before considering if we should merge the different tools or abandon one for the other you have to argue for what we would gain by doing so.

You have listed tools from the Erlang world that do the same things. For example relx and reltool, rebar3 and erlang.mk, and Common Test and EUnit. Have you looked at the historical reasons for why these different tools were created when similar tools already existed?

Rebar3 and erlang.mk have completely different premises, the authors of rebar3 wanted a build tool based on Erlang and the author of erlang.mk wanted a tool based on Makefiles, it would be impossible to merge those tools or use one over the other without giving up the original reason for creating the tool in the first place. Additionally, today most of the tools support building dependencies that use the other build tools, so not much is lost in terms of interoperability.

In ExUnit you use Elixir macros to write tests and you get features such describe, test tags, and setup blocks that you couldn’t write in Erlang in a similar way and you use a syntax that is natural to Elixir programmers. EUnit and Common Test also have different approaches to writing tests. What would be the benefit of using a single testing tool?

From this it sounds like you think most of tools that Elixir has added are not actually improvements (in the Elixir world) over the equivalent tools in the Erlang world. That is of course fine, but it’s a very different discussion than a discussion around fragmentation. I actually like that in the Erlang world there are multiple options for build tools, release tools and testing libraries. I hope that in the future we can get alternative tools to ExDoc, ExUnit, Distillery, and so on that explores new ideas so that we are not stagnant with the same tools and libraries forever.

rebar3 was primarily a backwards incompatible version of rebar2, which was of rebar, which go back way way to my days 15+ years ago in erlang, which I’m pretty sure is far before Elixir/Mix. ^.^

I really hated gradle at my last job, I fought for sbt, showed how the build scripts were more succinct and MUCH faster… Though finally gradle has an alternative to that horrible groovy language now that it has a kotlin interface and it is much faster now as well, though still significantly slower than sbt was (and still harder to make plugins for). GradleKT is now quite useable though, but still slow. ^.^;

EEP 48 was so awesome!

erlang.mk is also quite broken, every time I have to deal with it I keep running into bugs in it… >.>
Plus hey for lack of multi-platform!

I love having multiple documentation libraries, but I don’t like having multiple build systems.

With anything but build systems you can generally pick and choose and be fine, but with multiple build systems that means that some things just will not work at all if you choose one it doesn’t care about.

/me grumbles C++ world dependency managers…

A big challenge in unifying tools like mix and rebar3 is that they take a different approach to extensibility and configuration. Mix essentially lets you extend it by writing arbitrary tasks in arbitrary Elixir code that you evaluate within the tool.

Rebar3 aimed to have a declarative configuration format above everything and tries to actively restrict what you can do in its config files (there’s an escape hatch with rebar.config.script), and we aimed to make things extensible through the plugin system.

Mix supports semver and will, as far as I can tell, unify the dependencies and extension versions it needs. Rebar3 started in a world where a single lib would often have diverging git tags, app files, and documented versions; as such the dependency handling algorithm is based off a “distance from root” comparison (the version closest to the project root wins, with the assumption that more of its functionality is exercised). When it comes to plugins – and plugins include ways to load custom compilers and dependency fetchers – we instead handle sequences of loading and unloading dependency sets dynamically between libraries required for plugins and deps, so that they don’t clash.

That’s without mentioning the distinctions in profiles, configuration handling, releases, and so on. I think at this point the philosophies of both tools diverge enough that it would be extremely hard to unify them without breaking major parts of either ecosystem. It’s already a bit odd when it comes to dep resolving (we don’t have optional deps in rebar3!)

My prediction is that if we were to do it, we’d have to pick an intersection of features, and the rest would be best effort. So any lib from either ecosystem that uses a more advanced feature set from their build tool would be at risk of breaking.

I’m betting more on our ability to keep interoperating between mix deps and rebar3 deps as the way forward, since the bridge appears easier to gap here.

It would already help tremendously if rebar3 and mix could decide on a common format for “artifacts” (i.e. something that contains ebin, priv, include and a notion of runtime deps) and where to put them, so essentially the pure low-level build stuff. This should be possible without any breakage as the build directories get recreated on clean builds anyways.

Additionally, it would help a lot if this one was reconsidered: Publish a hex package for Elixir · Issue #6210 · elixir-lang/elixir · GitHub

What differences do you notice? rebar3 and mix have the same structure under _build/ for build artifacts.

One of the problems with publishing the Elixir core libraries as packages is that it breaks for Elixir developers. What should happen if you are using an Erlang package as a dependency and it depends on elixir 1.6.0 and you are currently using elixir 1.6.1 in your PATH? Or if one dependency depends on elixir 1.5.0 and the other on elixir 1.5.1? Remember also that rebar3 defaults to strict version requirements for package dependencies.

Why would it break for Elixir developers? Dealing with making sure the elixir used would be a problem for the rebar3 user like it is today for any package.