I feel like Elixir is getting big enough and old enough that I’m starting to experience problems with conflicting dependencies. An example from an application that I’m currently working with:
I went to try out the merquery library. This application uses version 1.0.2 of the jq library (the latest version, but from over five years ago) which in turn specifies poison ~> 4.0. The merquery library that I was adding specifies flint ~> 0.6 and flint 0.6.0, in turn, specifies poison ~> 6.0
This was actually just the beginning of the headaches and I had to chase down a few situations, making a couple of forks in the process to versions which didn’t conflict.
For a while I’ve been thinking that it’s wise practice to follow a general rule:
When specifying dependencies for an application, prefer the ~> operator (either x.y or x.y.z generally)
When specifying dependencies for a library, prefer the >= operator
I think pretty much everybody will agree with the first point. My thought on the second being: new versions of your dependencies will come along and, if there are bugs, you can fix your library (or if something is too difficult or you don’t have time you can add a ~> or <= constraint). But you shouldn’t stand in the way of applications wanting to use your library with newer versions of shared dependencies because probably most of the time they will work
Absolutely I think that you should consider your libraries use of each dependency. If you’re including jason and all you do is very simple usage of Jason.encode and Jason.decode, then the likelyhood that a major version will break things is low and you should probably use >=. If you’re using a library in a complex and intricate way, then you should consider if you should limit the versions you allow.
But I think that most people writing a library are used to using the ~> operator from building applications.
So my suggestions (given that this is the ideas/suggestions/proposals area of the forum):
have some advice in the documentation
if people are mostly in agreement, build a bot which goes to Elixir dependencies (especially more popular ones) and automatically suggests changes, along with a detailed explanation of the whys involved and things that the maintainer should consider on if they should or should not accept the change.
I’m happy to work on these things (and others) because I think it will be more and more of a big deal to help make Elixir development smoother in the long run (I’ve used npm enough to see how bad things can get )
Also, this post from the Ruby world has some good discussion on the topic as well.
You can override dependencies if necessary, but I think there is something major missing from overriding dependencies, specifically I think you ought to be able to say why you are overriding a dependency, and then mix can tell you when you don’t need to anymore.
For example:
{:jq, "~> x.x", override: [:merquery]}
says “I know that merquery wants a different version, but I’m overriding it”. Then if later you update merquery, and no longer need the override, mix will instruct you to remove it. Additionally, if you add some new dependency that wants an older version of jq, you have to acknowledge that you are also overriding that dependency.
This makes it a bit safer to use override as a consumer of libraries, which I think helps alleviate this issue a bit as well.
Yeah, good point… I didn’t know about the override option. Feels like !important is CSS
But for making things smooth for Elixir devs, they would need to know about override (I’ve been using Elixir for 6-7 years and I just learned about it!)
I think in the case I outline that you’d need to add poison to your application’s mix.exs and then force that into place maybe? Or could I force jq and then it ignores its dependencies?
I’m definitely not against the option of making override clear in the documentation (with warnings, probably, and maybe in addition to my suggestions?)
Usually when I see this option being used in a codebase, it’s a red flag for me, as you can get into a situation where the dependency override might break that other library.
I agree, but I think in some cases it is effectively unavoidable. I think we need to enhance it, not discourage its use. If I can say “I’m overriding to solve this specific dependency conflict that I’m aware of”, then most/all of the danger of using that option goes away.
Well, this is the problem, you can’t really guarantee that unless you inspect the codebase of dependencies. The dynamic nature of elixir makes this even more dangerous, as it might blow in your face at runtime.
I think a potential solution would be to run unit tests of that library with the dependency you have at hand? If I’m not wrong this is what PERL does when you install packages, to make sure you don’t have dependencies mismatch somewhere. This OFC heavily depends on the library, as some libraries require a complex setup to run tests.
I’m suggesting that this is built into the mix dependency resolver. Right now the following happens all the time:
I want to add foo to my app. I already have bar and baz.
foo depends on a newer version of bar.
I can’t update bar because baz depends on it.
I check how they baz and foo use bar, and confirm that its fine to just override.
So I add {:bar, "~> ...", override: true}
Someone else adds buzz to the app, which also depends on an old version of bar.
Problem 1: We never find out that we just overrode bar for the sake of buzz as well.
Next, foo releases an update that depends on more stuff from the old version of bar.
Problem 2: mix tells us we can update foo, so we update it and have bugs.
If instead of override: true, I could say:
{:bar, "~> x.x", override: [foo: "x.x.x"]}
which would say “this override only overrides the dependency that foo at exactly version x.x.x has on bar”, then we are protected from any of those accidental changes.
Adding buzz would produce an appropriate dependency conflict warning, solving Problem 1. We can then go look at the code/docs and decide if we want to override the bar dependency for that version of buzz as well.
foo won’t appear to be automatically upgradeable, solving Problem 2.
Great idea! It would certainly help a lot to get mismatch version errors, this is especially true for some of the bigger applications, where you might have 100+ dependencies.
I wonder if there is a language out there that solved this problem of “global” dependencies?
Yeah, for sure… I guess that’s part of why I was trying to ephasize that one should prefer the use of >= in a library, but I definitely wouldn’t say that you should always use it. I guess it feels like many people don’t think of it as an option, and so I worry about people not really thinking about it (and I get it, you’re trying to get your cool idea for a library out there in front of people)
I think if we wanted to provide guidance to people, we should give people both options on dealing with such things depending on their usage of the dependency:
poison >= 4.0
poison ~> 4.0 or ~> 5.0 or ~> 6.0 / poison >= 4.0 and < 7.0
But also perhaps some sort of tool that lets them know (as a library maintainer) when there is a new version of sub dependencies which aren’t being allowed (either as a part of mix somehow, or a bot that points in out in a pull request).
For override, I can see how either the documentation or the functionality could be improved as @zackdaniel outlined, though I’m not personally interested in tackling that problem.
I was looking at rubygems.org at some of the more popular gems, and I see a bit of a mix, but there are a lot of >= x.y.z (including some ... and <= x.y.z). Definitely some ~> x.y.z and ~> x.y. But I’m not sure how it compares to hex packages usage. rails, of course, just ties each version to exact = x.y.z of it’s custom sub-dependencies (actioncable, actionpack, actionmailer, actionview, activestorage, etc…). Those sub-dependencies use a mix of >= and ~>.
Maybe an interesting project to gather data from the two and compare usage
The actual version being used in elixir is also fixed and it’s located in mix.lock.
I was wondering more in terms of each library compiling with its own version of dependency, hence avoiding this game of having the right dependency for all libraries.
I know some java build tools experimented with this, but the wait time for the initial build without cache was about 30 minutes for a new project…
Along these lines, a common mistake in dependency listing is using ~> 1.2.3 when you actually mean ~> 1.2 and >= 1.2.3. Library authors will use ~> 1.2.3 because they want to get some fix added in 1.2.3, not realizing that they just prevented users of their library from upgrading to 1.3.0.
While I agree with @zachdaniel here and will not repeat his excellent suggestions, for the initial question at hand I believe that Elixir 1.18 having JSON functionality built-in will solve a lot of these headaches down the road… VERY down the road, likely 3-5 years from now though.
IMO the best we can do for now is start PRs to the affected libraries where these conflicts exist.
It also seems to me that having multiple versions is the only practical solution. You can’t tell if any two versions of a package have the same behavior automatically in general, so any kind of automated version resolution seems like a nonstarter.
There’s an open issue about this, for what it’s worth:
The problem comes when an incompatibility is detected in the future and you need to constrain the version. You publish a new version of your library with the version requirement changed from >= 2.0.0 to ~> 2.0. However that wont help because the version solver will always attempt to find a matching version and will just pick the version before you added the stricter constraint.
There is a tool for that: mix hex.outdated.
Do you have suggestions on how to improve the documentation for override?
You cannot use multiple versions of an OTP application in BEAM. That issue is only aiming to speed up compilation when switching between branches that are using different dependencies.
When using a 5 year old library we shouldn’t be too surprised that things don’t fully work when used with libraries that have been kept up to date. In this case the solver couldn’t find a solution because there were incompatible version constraints, it was discovered and could be fixed with override: true.
With conservative version requirements using ~> we can with the help of the version solver find the incompatible dependencies and in the cases where the requirements are wrong we can explicitly mark them with override: true.
On the other hand if we had lose requirements we would be in dependency hell with trying to figure out which dependencies are broken and hope that we have tests that cover all those cases for all of our direct and transitive dependencies. We would have just traded one set of problems with another that is harder to fix and have confidence in the solution.
The version solver is there to help you find compatible dependencies, with >= version requirements the solver becomes useless and you have to do its job yourself.
)
