Following the discussion on not using private API in our Elixir code, I was inspired by a comment asking if anyone builds all the packages on hex.pm before each release. Apparently nobody does.
There have been other discussions on the forums here in the past about curating hex packages, running checks before publishing, etc. So this topic comes up from time to time, it seems.
In my spare moments since last Friday, I wrote a little tool that tries to build all packages on hex.pm (or just packages that have updated since the last check; or just packages that previously failed; or a specific subset of packages…) and logs the results. Turns out that with Elixir 1.7.1 and Erlang OTP 21.0.4 ,simply including a package in mix.exs as a dependency and building results in a failure rate of over 10%.
Some could well be due to lacking -dev packages for libraries used by some nif-building packages… but many are just actual build failures. A good example are packages that use jsone. There are 22 of them on hex.pm as of this writing. jsone 1.4.7 builds just fine with the latest erlang release, but earlier versions do not due to a backwards-incompatible change in stacktrace API. A number of those 22 packages, and by domino-effect even more packages that depend on those packages, have not updated the jsone dependency and so build with pre-1.4.7 versions and fail.
Looking at the work to be done, it’s pretty trivial: go through those 22 packages and ensure they require at least v 1.4.7 of jsone and get a new release up on hex.pm. This assumes that the owners of those packages are active, of course.
At this point I’m considering what can be done with this information. Various thoughts:
- Where should results be posted so the community gets the most benefit?
- It would be great if build health information was made available on hex.pm itself, and was queryable
- If some of us took on janitorial work to keep more packages on hex.pm in good health, everyone would benefit from that; I have already gone through some of the failing packages so hopefully some improvements will come of all this, but I am only one person
- I wonder if some packages should be retired from hex.pm, such as some packages that are full-on phoenix apps which use rather old version of the phoenix stack and do not build properly as a dep; obviously flagging these is a first step, but then what?
- Adding credo runs (and …?) to the hexagon tool would be nice … I suppose I will get around to it at some point, but I wonder if there are others as interested and motivated to do something about package health on hex.pm as I am …?
I am very interested in your thoughts and input on this …
p.s. Over the coming weeks, as time permits, I will set up a regular run of hexagon on one of our testing servers that checks updated packages nightly and where I can do full scans with new versions (and release candidates) of Elixir/Erlang. I’m not sure where I will end up publishing the results just yet, as per the above thoughts.
p.p.s I really need to document the hexagon tool; it started as a fun “how hard would that be to do?” one-evening project and then got a little out of hand so it doesn’t necessarily follow my usual expectations for documentation etc. Apologies to anyone who looks at it before I address that …