Hex package health and janitorial work

[aseigo]

Following the discussion on not using private API in our Elixir code, I was inspired by a comment asking if anyone builds all the packages on hex.pm before each release. Apparently nobody does.

There have been other discussions on the forums here in the past about curating hex packages, running checks before publishing, etc. So this topic comes up from time to time, it seems.

In my spare moments since last Friday, I wrote a little tool that tries to build all packages on hex.pm (or just packages that have updated since the last check; or just packages that previously failed; or a specific subset of packages…) and logs the results. Turns out that with Elixir 1.7.1 and Erlang OTP 21.0.4 ,simply including a package in mix.exs as a dependency and building results in a failure rate of over 10%.

Some could well be due to lacking -dev packages for libraries used by some nif-building packages… but many are just actual build failures. A good example are packages that use jsone. There are 22 of them on hex.pm as of this writing. jsone 1.4.7 builds just fine with the latest erlang release, but earlier versions do not due to a backwards-incompatible change in stacktrace API. A number of those 22 packages, and by domino-effect even more packages that depend on those packages, have not updated the jsone dependency and so build with pre-1.4.7 versions and fail.

Looking at the work to be done, it’s pretty trivial: go through those 22 packages and ensure they require at least v 1.4.7 of jsone and get a new release up on hex.pm. This assumes that the owners of those packages are active, of course.

At this point I’m considering what can be done with this information. Various thoughts:

• Where should results be posted so the community gets the most benefit?
• It would be great if build health information was made available on hex.pm itself, and was queryable
• If some of us took on janitorial work to keep more packages on hex.pm in good health, everyone would benefit from that; I have already gone through some of the failing packages so hopefully some improvements will come of all this, but I am only one person
• I wonder if some packages should be retired from hex.pm, such as some packages that are full-on phoenix apps which use rather old version of the phoenix stack and do not build properly as a dep; obviously flagging these is a first step, but then what?
• Adding credo runs (and …?) to the hexagon tool would be nice … I suppose I will get around to it at some point, but I wonder if there are others as interested and motivated to do something about package health on hex.pm as I am …?

I am very interested in your thoughts and input on this …

p.s. Over the coming weeks, as time permits, I will set up a regular run of hexagon on one of our testing servers that checks updated packages nightly and where I can do full scans with new versions (and release candidates) of Elixir/Erlang. I’m not sure where I will end up publishing the results just yet, as per the above thoughts.

p.p.s I really need to document the hexagon tool; it started as a fun “how hard would that be to do?” one-evening project and then got a little out of hand so it doesn’t necessarily follow my usual expectations for documentation etc. Apologies to anyone who looks at it before I address that …

[tcoopman]

Very nice work!

@aseigo

Where should results be posted so the community gets the most benefit?

A nice idea could be to automatically create an issue on those packages. But maybe some packages don’t have an issue tracker, or the issue might already be present so I’m not sure if this would be the best idea.

@aseigo

It would be great if build health information was made available on hex.pm itself, and was queryable

That would be great, if the results are shown there.

@aseigo

Adding credo runs (and …?) to the hexagon tool would be nice … I suppose I will get around to it at some point, but I wonder if there are others as interested and motivated to do something about package health on hex.pm as I am …?

What would you do with this? And what credo config would you use? Some people don’t use credo. I don’t think this is something that your tool should do.

[aseigo]

@tcoopman

A nice idea could be to automatically create an issue on those packages.

My concern is that it would possibly annoy package authors … I would rather have some sort of ‘opt-in’ thing they can subscribe to and get notification of problems with their packages. I think that belongs on hex.pm … what do you think?

@tcoopman

What would you do with this?

Try to gauge the general “best practices” conformity of a package; I find this to be a helpful hint as to the general health of the code. Note every check supported in credo (or the community contrib packages) is super relevant, so I would want to use a conservative config for credo. But then, make it available as additional information … if a package is full of credo issues, I may want to help fix it up … (or, more cynically, avoid it in favor of a more “conformant” package)

@tcoopman

what credo config would you use? Some people don’t use credo.

Hexagon uses a super minimalistic mix project called “petridish” snicker that ships in the hexagon/prive/petridish directory. It pulls the packages (one by one) into the petridish as a dependnecy. This is necessary / useful because:

• not all packages can be built from their own root (most can, but a few can’t)
• some packages build fine within their root with rebar, but fail when compiled with mix (!)
• it allows Hexagon to only support mix builds (via the petridish), and so whatever mix can compile, so can Hexagon

As such, I would include credo (and a conservative configuration for it) in the petridish mix project as well and run it from there.

@tcoopman

I don’t think this is something that your tool should do.

Can you explain to me why?

[voughtdq]

I love how there are package scores on https://pub.dartlang.org.

They give packages a score based on popularity, health, and maintenance. I don’t know if they send the notifications to developers, but it’s nice to get a quick insight into whether the package is using up-to-date deps for example.

[tcoopman]

@aseigo

My concern is that it would possibly annoy package authors … I would rather have some sort of ‘opt-in’ thing they can subscribe to and get notification of problems with their packages. I think that belongs on hex.pm … what do you think?

I agree!

@aseigo

Can you explain to me why?

Does the project build and does credo find issues are 2 different things to me. I was thinking about the original idea of this tool (which packages can’t build with the current elixir master - to help safe upgrading of elixir?) and running credo looks out of scope.

I’m not saying checking the subjective health score is not useful, but maybe it should be done by a different tool? There are a lot of other metrics as well (are there test, do the tests run, is everything formatted, number of open issues, downloads,…)

[voltone]

You’re going to want to be very careful about sandboxing the server where you’re running this scan. A single malicious package can do some very nasty things to the server, or the network it is connected to…

[josevalim]

@aseigo

In my spare moments since last Friday, I wrote a little tool that tries to build all packages on hex.pm (or just packages that have updated since the last check; or just packages that previously failed; or a specific subset of packages…) and logs the results.

This looks really neat!

@aseigo

jsone 1.4.7 builds just fine with the latest erlang release, but earlier versions do not due to a backwards-incompatible change in stacktrace API.

I’m curious, which change? I thought the stacktrace change in OTP 21 was backwards compatible?

@aseigo

It would be great if build health information was made available on hex.pm itself, and was queryable

We have discussed this on other occasions but I really don’t believe any kind of ranking belongs in Hex for two main reasons:

1. Hex is the source of truth when it comes to packages. Any kind of ranking assigned by Hex will likely be seen as a source of truth too and coming up with an accurate ranking is far from straight-forward. Rankings are social, skewed, can be cheated, etc.

2. Hex is a fundamental infrastructure service to the community. I prefer the Hex team to keep focused on performance, infrastructure, security than other aspects.

Hex has public APIs and the recent release of hex_core makes consuming them even more straight-forward, so I propose for developers to continue to build on top.

[aseigo]

@tcoopman

Does the project build and does credo find issues are 2 different things to me

Yes, I agree with that. They are two distinct bodies of information. The intersection between them are:

• the package (the all relate to the same artifact, some hex.pm package)
• they are all useful to know which packages to adopt, to avoid, or to contribute to
• they all require a tool that can iterate over packages, or a subset of them, to gather that info, which is a core capability of hexagon

So that was my thinking … I’d still keep the actual processing separate (e.g. run build tests whenever a package updates, or a new version of elixir / otp comes out; but only do issue/contributor/etc. counts when a new version comes out), but this was one reason I wanted to have a local cache of the packages (so hexagon could eventually run multiple passes over the set of packages looking at different aspects of them)

@tcoopman

are there test, do the tests run, is everything formatted, number of open issues, downloads

Yep, all useful information. The tests one in particular is, at least imho, very close to the “does it build” question. If they are failing or even don’t build, that’s a sign that perhaps we need to do some janitor work on them…

[aseigo]

@josevalim

This looks really neat!

Thanks

@josevalim

I thought the stacktrace change in OTP 21 was backwards compatible?

Calls to erlang:get_stacktrace() fail at build in a mix project, and there is a message to the use the “new try/get api instead”. So in recent version of jsone there is this:

-ifdef('OTP_RELEASE').
%% The 'OTP_RELEASE' macro introduced at OTP-21,
%% so we can use it for detecting whether the Erlang compiler supports new try/catch syntax or not.
-define(CAPTURE_STACKTRACE, :__StackTrace).
-define(GET_STACKTRACE, __StackTrace).
-else.
-define(CAPTURE_STACKTRACE, ).
-define(GET_STACKTRACE, erlang:get_stacktrace()).
-endif.

@josevalim

Any kind of ranking assigned by Hex will likely be seen as a source of truth too and coming up with an accurate ranking is far from straight-forward. Rankings are social, skewed, can be cheated, etc.

I agree for information such a “how many stars on github”, but for information like “what versions of elixir does it build with” or “does it build with current versions” would be useful there imho and are not social in nature.

@josevalim

1. Hex is a fundamental infrastructure service to the community. I prefer the Hex team to keep focused on performance, infrastructure, security than other aspects.

Ok, that makes sense and is a fine perspective to have…

@josevalim

hex_core makes consuming them even more straight-forward

Indeed, I am making heavy use of that!

So … going forward I will look at ways to present this information as a complimentary and separate service to hex.pm. Thanks for the clarity, much appreciated!

[tcoopman]

@aseigo

Yep, all useful information. The tests one in particular is, at least imho, very close to the “does it build” question. If they are failing or even don’t build, that’s a sign that perhaps we need to do some janitor work on them…

But this might be much harder to verify automatically (you might need a db/other deps/…)

@aseigo

I agree for information such a “how many stars on github”, but for information like “what versions of elixir does it build with” or “does it build with current versions” would be useful there imho and are not social in nature.

That’s the reason why I would separate does it build from how well does it score (credo/tests/…)

[josevalim]

@aseigo

Calls to erlang:get_stacktrace() fail at build in a mix project, and there is a message to the use the “new try/get api instead”. So in recent version of jsone there is this:

Could this be the warning as errors option? Note that warnings_as_errors is always disabled for dependencies, which means that a project failing to build locally because of warnings_as_errors will still work as a dependency.

[aseigo]

@voltone

You’re going to want to be very careful about sandboxing the server where you’re running this scan.

Agreed.

VMs are a beautiful thing. We use KVM for hosting our build systems, as they provide a better separation than e.g. just containers … and if the build VM gets destroyed, oh well. I plan on firewalling this vm pretty carefully as well.

That said, the security aspect is one of the reasons I am at all interested in the topic. Packages on hex.pm are going to be installed on goodness knows how many innocent systems in the wild. All packages on hex.pm should be audited to make that public usage more safe.

Step one is to make sure things build. Then we can start to look for code health and quality metrics. Ultimately, I’d like to run something like sobelow, though I find its rate of false positives to be too high at the moment and its utility outside of phoenix apps to be limited … but it’s the right sort of idea.

[aseigo]

@josevalim

Note that warnings_as_errors is always disabled for dependencies, which means that a project failing to build locally because of warnings_as_errors will still work as a dependency.

Hexagon builds packages as dependencies in a mix project (template is in the priv/petridish/ directory in the hexagon repo… super minimalist). It generates a minimal mix.exs file that contains just the dependency. However, it uses the path definition to pull it from the local cache hexgon builds; that’s the only oddity I can think of. Does that change things? I suppose I could try on one of the affected packages replacing the path: based dependency with a more usual "~> version" definition and find out …

(The local cache makes it easy to try fixes in-situ; in future will make it easier to do multiple runs through the packages to gather different datasets; also makes it fairly trivial to track when packages update, though that could also be done with the build logs as well…)

[aseigo]

@tcoopman

you might need a db/other deps/

other deps are fine; db would be harder, but hopefully if they are using ecto only marginally so (might need to run migrations first and then drop db’s after), but it should be reasonably possible to run tests that come with publicly published dependencies … otherwise it is difficult to independently verify them … we’ll see, I suppose