`mix release` and dependency failures

Questions / Help
,
1

Today I’m trying for the first time to run mix release on my project… just as a sanity check: I’m failing.

I’ve run into a bit of a roadblock and I’m not precisely sure how to troubleshoot it. I am going to simplify a bit here so it is possible I’m leaving something important out, but with luck I’m including the essentials. Finally, I should note that everything in my application works as expected when I run it in my development environment, when I run the test suites, or let GitHub Actions take a crack. These issues are only seen using mix release. This will cover things I’ve tried up to the present time.

The Scenario

I am using a third party library in my application (GitHub - ExHammer/hammer: An Elixir rate-limiter with pluggable backends). Out of box, the library wants to start an an application with a supervisor and to put some services under it… however I don’t want that. I’ve actually included this library in my own little private library (wrapper library from here out) and I want to be responsible for starting the services, including Hammer’s, in the correct part of the supervision tree and at the right time.

First Effort

Initially to achieve this, in the wrapper library mix.exs I had the Hammer library as dependencies in deps like so:

    {:hammer, "~> 6.0", runtime: false},

This achieved what I wanted for my development environment, but didn’t work for mix release; I got errors about Hammer.Supervisor not being found and also warnings along the lines of:

…is used by the current application but the current application does not depend on…

I’ve since read:

So that approach was bound to fail for releases. Lets try something different.

Second Effort

I went back to the mix docs and found this:

:included_applications - specifies a list of applications that will be included in the application. It is the responsibility of the primary application to start the supervision tree of all included applications, as only the primary application will be started. A process in an included application considers itself belonging to the primary application.

Great! As described, this is almost exactly what I want. I’m a bit dubious about the primary application discussion since the wrapper library isn’t the final application which will ultimately be released but lets try this in my wrapper library mix.exs application/0 function:

included_applications: [
        :hammer
      ]

The “used by/doesn’t depend on” warnings all disappeared… but now mix release gives me this:

** (Mix) :hammer is listed both as a regular application and as an included application

Hmmm… OK.

I know that if no :applications key is given, that the applications will be inferred from the dependencies so maybe it’s inferring it from the fact that Hammer is a dependency… so try and silence that by making the mix.exs application/0 function include this:

applications: [],
included_applications: [
        :hammer
      ]

Mix release still gives me:

** (Mix) :hammer is listed both as a regular application and as an included application

Current Time

I know why it thinks its in included applications (I put it there), but I’m unsure why it thinks it is configured as a regular application as well. My little wrapper library is the only place bringing in Hammer. In the Hammer mix.exs it shows up as:

  def application do
    # Specify extra applications you'll use from Erlang/Elixir
    [mod: {Hammer.Application, []}, extra_applications: [:logger, :runtime_tools]]
  end

Which is exactly what I’m trying to suppress so that I can start things in my code, but I don’t know if this would cause that error. Naturally, I don’t want to futz with a third party library and create a one-off solution applicable to just me either.

I’m not sure how to troubleshoot further (aside from diving into the source code of mix processing/compiling). Perhaps what I’m trying to do isn’t possible… but it sounds like it should be… at least the documentation implies that it should be. I do see compiler warnings during the mix release process, but they’re all coming from unrelated third party libraries that are also part of the application.

Thanks in advance for any help/pointers on how I might proceed from here.

1 Like
2

I think I’ve found the answer; I’ll include here since I’ve seen different flavors of the same premise asked here previously without answer. The short answer is what I’m trying to accomplish appears not to be really feasible, at least in any clean way.

In my final attempt described in the original post I got to the error:

** (Mix) :hammer is listed both as a regular application and as an included application

After some searching this morning I have now found a GitHub issue in the Elixir repo which describes the same fundamental scenario and reaches the same end:

The GitHub issue models scenario of 3 different Elixir applications that depend on each other as: A depends on B, B depends on C, and C has its own startup in the normal way with applications: [mod: {_, _}]. Same as my basic scenario and they people opening the issue try same as I did: use :included_applications to suppress the C startup. They end up where I end up with the regular/included applications error. (There are actually a few different flavors of this described in the issue, but for our purposes here I think this suffices).

In the issue the suggestion is made by @josevalim:

I still think included_applications is not the mechanism to get the behaviour that you want. Here are some alternatives I can think of:

  1. Move the check if the application should be started or not to B and C start/2 callback. In case it should not, you can start an empty supervision tree.

  2. Instead of listing those as included applications, do not add B and C as dependencies of A. You can do that by passing runtime: false to B and C. Then, in a release, you explicitly include them with applications: [b: :load, c: :load].

In my case, option 1 is not really feasible because “C” for me is a third party library; its open source and I could submit a PR… but I don’t think they’d see it as a good or necessary change to make their startup process optional.

Option 2 I’m confident would work (my early tests this morning suggest that it can get there), but not I’m not too fond of what needs to happen. I start having to directly reference internal dependencies of my libraries at the top level application mix.exs. At that top level application I really want the dependencies of my dependencies to be opaque; this would be especially true if the “B” dependency above were third party as well since they could swap out its dependency on “C” with something else… sure that could cause other things to break if I’m overriding its intended startup… and maybe that’s reason enough to keep the status quo for this issue as it is.

In the end my need for this is more about trying to keep my abstractions clean whereas the GitHub issue was raised for far more practical reasons. So my solution for my project will just be to accept that if a library is starting as an application, my choice in using that library is to accept it as-is or to find an alternative library with the behavior I need (or privately fork/patch it for my needs).

I will close in saying that I feel the documented use of :included_applications in the docs is very confusing given what I’ve now learned because that documentation seems to suggest exactly what you’d want for the A -> B -> C startup issue. I’m not sure really what case that option is there to fulfill, but the documentation needs to express that nuance; if it’s because the contextual shift from Erlang to Elixir changes how you might use it (or not use it), then that difference might need to be expressed.

3

I’d be curious what reason you’d have to start hammer in your supervision tree over having it in the dependency. What do you mean by “keep my abstractions clean”?

4

Application.spec(:wrapper, :applications) will tell you if Elixir is listing it as an application or not. The other possible scenario is that something else is depending on hammer, therefore becoming a regular application for someone and an included application for wrapper. mix app.tree may help here.

2 Likes
5

It may also be an Elixir bug in that we are treating hammer as a regular dependency even if nothing else depends on it.

6

Early on I spent some time analyzing the planned functionality of the application in relation to possible failure scenarios. This was principally aimed at understanding the failure domains within my application: What services of the system could experience a failure while not necessarily impacting other services in the system? What failures should cascade to other related parts of the system? If I want to force a shutdown of certain services how might that be managed? etc. Answering questions like that and then figuring out if there were ways to simplify, flatten, etc. the resulting tree of runtime service dependencies that could be represented by a reasonable supervisor tree. In this process the services for which the rate limiting in question was required were identified and those services were to be supervised together so that failures, restarts, service shutdowns, etc. could be managed together without necessarily impacting parts of the system which didn’t depend on it.

I should say here this was all planning grade stuff and planning done with only a fair amount of “book learning” to inform the process; this is my first serious Elixir project and so I don’t have a depth of Elixir/Erlang production experience to inform some of these decisions. My process here is to get stuff done in the spirit of that plan as best as possible. Once I have enough of the application together, I will start to poke and prod and do the kind of induced failure testing that will tell me if my initial thinking about supervision in this kind of application is valid; the expectation is that things will change at that time. But I’m not there yet, so my current motivation is primarily to stick to my current plan as much as possible, try to overcome any obstacles that are reasonable to overcome in realizing it, and finding where harder, practical limits exist which should shift my thinking.

Finally, for this rate limiter service the practical reality is that it being outside of my supervision tree probably doesn’t matter that much; I did forget that my wrapper made sure that there was some Mnesia setup done prior to starting the rate limiter which was some more of the rationale of wanting to start things up myself. Sure, I think it would be nicer to be able to fit it to my specific plan… maybe a little more obvious and easier to comprehend things about the system looking at various monitoring tools and such… but I don’t think it’s a show stopper by itself.

The application is currently constructed as a bunch of small, library like components. Each one is very limited in scope, offers its own API for interacting with it, and can be fully tested and such as a unit with only its own dependencies to worry about. Essentially this is Dave Thomas’s component model, though I think in some cases I don’t go as granular as he’d advocate. The class of application I’m building traditionally turns into a giant, incomprehensible ball of spaghetti with mud sauce and this style of project organization solves some issues related to that. It’s early days yet, but overall I’m finding that the overall pluses are outweighing the overall minuses. That being said, each of these components should fully encapsulate its functionality: at the very top levels of the application, I shouldn’t care or know that Hammer was a dependency of my wrapper library: everything the dependent application should know should be in the API of that wrapper library.

Well, I’ve now gone and done what I said I’d do in my original follow up message. I’ve let Hammer start itself up (I stripped the Mnesia requirement, too, though that may come back later). But Hammer requires configuration to start it’s service. So I now need to be sure that any dependents on my wrapper library also include Hammer configuration sections. If I were to swap out Hammer for some other solution, I have to revisit those places to remove the cruft… whereas in my original design, all of the Hammer specific stuff, including things like making Hammer startup contingent on service startup params rather than config.exs, was handled by the library itself. Whether or not I should wrapping Hammer or not isn’t so much a point here (there was some licensing ambiguity when I started that made my dependency on Hammer questionable) as much as its internals leaking out and the implications of that beyond this particular library. The abstraction, in this case, ends up not being integral because of the requirement to directly interact with what should be a hidden dependency.

7

Thanks for the pointers on how to troubleshoot this. I wasn’t aware of some these tools or really how to start to analyze some of these kinds of questions

I only see Hammer being brought in via the single wrapper library (mix app.tree does look pretty helpful in this regard), but I see my wrapper library being brought in twice; doesn’t seem like this should do it, but I could very well be wrong. I can futz with that and see if it changes anything. While this sort of “feels” bug-like and that it should work or work differently, I am sufficiently inexperienced enough that I’d still bet that I’m just not doing the right things or have incorrect expectations.

I’ve actually already removed my overriding of Hammer’s self-starting behavior, so I’ve worked around the issue and have gotten a release that works. I’ll revisit once I get some quiet time.

Thanks again!