Dependencies and programmatically avoiding circular references

Questions / Help
1

This is going to be an oddball question and not really something that others are going to commonly encounter. I’m also pretty sure that what I’m going to be asking about is simply not possible… but as I prove time and time again that I don’t know everything: here I am, question in hand.

The Short, Less Clear Version

I have an Elixir project which is library-like and will be used in several other Elixir projects. This library project will also have some dependencies on other Elixir projects I’ve also built. Some of those dependencies could benefit from the library itself and so would be great to include in those projects; but of course, this creates circular references between projects. So the question becomes this: is it possible to programmatically exclude a project’s dependencies at compile time (or deps.get time which seems more likely to be what I need).

The Long, Possibly More Clear Version

(Note that I’ll be simplifying some things here for the purposes of discussion)

My application is built up from a fair (and growing) number of library-like Elixir projects which are dedicated to narrowly defined feature sets; I’ll call these “Components” because they aren’t really true libraries like you might find via Hex. Some of these Components define runtime services which the top-level application ultimately will be responsible for starting and supervising. At the end of the day, however loosely, these Components are intended to work together and therefore can assume that some of these services will have been started and supervised by that top-level application.

In terms of building a running application which I can build successfully into releases, all of this works fine and as intended. There is no issue of circular dependencies in the normal course or even threat thereof; this is avoided by design.

However… during REPL aided development and during testing, having some of these services up-and-running in some minimal way can be very helpful (or in the case of testing actually essential). Up to this point I have, in each Component that would be helped in this way, manually defined helper functions to get the requisite services loaded/started. This works, but results in a lot of boilerplate, duplicating the same configurations, supervisor setups, childspecs etc. across components where this needs to happen. I hate that. So, I think to myself, why not create yet another Component that’s only available when Mix.env() is either :dev or :test encapsulating these helper common services? I’ll still have a little boilerplate, but it will be much minimized.

This utility Component also works well much of the time, but there are some cases where this can end up in circular dependencies. Consider the following scenario:

I have the following hypothetical Components:

  • Db - functionality to build, load, start, etc. involving databases.

  • Flags - Feature Flag-like functionality backed by an ETS table fronted by a GenServer holding runtime-manageable configurations. Depends on Db for persistence.

  • DevUtils - the :dev/:test only support Component which provides and extends features from both Db and Flags for REPL aided development and testing. Depends on Db and Flags.

  • Features - higher level business logic. Depends on Db, Flags, and DevUtils (only :dev/:test).

  • App - the top-level application. Depends on Db, Flags, Features, and DevUtils (only :dev/:test)`.

As defined above, things work OK. But notice that Flags depends on Db. This means maintaining Flags would benefit from the Db supporting parts of DevUtils and there’s the rub: DevUtils already depends on Flags. Trying to make DevUtils a dependency of Flags results in a circular dependency.

The Question

Given the above, ideally, I’d love for DevUtils, while being added as a dependency to Flags, to see that fact and to not try to use Flags as its own dependency, thus avoiding the circular dependency. So… simply put, is there a way to do that?

Naturally, I’ve tried a few things hoping it might work (for example optional: true in relevant dependencies)… but the more typical mix.exs dependency exclusions or conditional compilation methods I expect won’t work because I’m relatively sure it’s already too late by the time you get to compilation where those things would matter; the following seems to suggest as much:

While I expect that’s still true and that I take its meaning correctly, that quote is nonetheless pretty old and on the off-chance things have changed or I’m missing something, I thought I’d ask.

This is nothing urgent and I have acceptable “plan B” approaches which avoid the circular dependencies, but all my alternates are less clean than having a singular DevUtils package which supports all the development support scenarios I’d like it to.

2

I don’t have any immediate solutions to suggest, but in general I’ve noticed that if I’m building phrases to label subparts of a thing it may mean there’s a name missing.

In your situation, it sounds like the “Db-supporting parts of DevUtils” and the “Flags-supporting parts of DevUtils” might be happier as separate packages.

1 Like
3

I agree with @al2o3cr that this sounds like a good opportunity for splitting parts of your codebase into new, separate packages which can be imported into each component which needs them.

I would argue that sticking with a singular DevUtils package is not actually that “clean” if you need to do a crazy dance to make the compiler happy.

4

Many thanks @al2o3cr and @Tyson for your input.

The “plan B” for this problem that I’ve put in place is just to concede, for now, that these development oriented functions are just most naturally part of the application Components. So instead of, say, database related functions going into a DbDevUtils, they just are now part of Db. The biggest downside is that this means these dev/test helper functions will be in production and release builds: while they shouldn’t interfere with normal operations, they do increase the overall theoretical attack surface of the modules and are the sort of cruft that can get misused if available in contexts where they shouldn’t be available.

Admittedly, part of my motivation in asking my question was also to ask a question behind the question: “how far can the conditional compilation techniques that are well supported at the top level be pushed into dependency compilation.”

I agree and its been my “second best” option. Indeed having committed to this “Component” based application development style, the reason there are separate Db and Flags Components is because of just this kind of analysis. A single DevUtils absolutely breaks this principle and definitely works against the natural organization of the code. The main justification for a single DevUtils is that the functionality being wrapped is so thin it’s easy to swamp the gains from reduced boilerplate & duplication with the overhead of maintaining multiple made for purpose Elixir packages… and there’s a little extra cognitive load cost increase for understanding what is essentially dev/test tooling… which just isn’t the context I want to spent my “available cognitive load”. So my quest was very much in the vein of “wanting to have my cake and eat it too”.

I would disagree with this characterization. There are conditional compilation techniques which are well supported and the question wasn’t geared to finding some sort of “loophole”, “bug-as-feature”, or a “one weird trick” kind of answer; it was really about how far down do the supported conditional compilation techniques get pushed and that I might not be aware of. If a conditional compilation method is supported, I would argue that’s not a “crazy dance to make the compiler happy”, but “making use of a supported feature to achieve a goal”. At that point we can argue about whether the goal is worthy, but the technique itself shouldn’t be in question.