Is it possible to use a dependency's structs inside of config?

Questions / Help
,
1

I am bumping into some limits of my own understanding when it comes to application configuration. Specifically, I was wanting to reference a dependency’s structs inside my app’s configuration. I’m getting errors like this:

** (CompileError) config/components.exs:27: SomeDep.__struct__/1 is undefined, cannot expand struct

Is there a way to ensure that the dependency is compiled and that its structs are available when my app is being compiled? Or am I putting the carriage before the horse?

Thanks for any clarifications!

2

Ah… from the docs:

  • config/config.exs - this file is read at build time, before we compile our application and before we even load our dependencies. This means we can’t access the code in our application nor in our dependencies. However, it means we can control how they are compiled

I guess I either move the structs over to the runtime.exs or I make the config work without structs.

1 Like
3

runtime.exs is the way indeed (unless you really need compile time config but then you have a chicken-and-egg problem), you can use your own code and dependencies there.

4

The only reason that the compile-time config would be nice in this case is for documentation purposes: I can reference the config from inside a @moduledoc and generate accurate information about how the app is set up (e.g. via a Mermaid chart). For a complex app, that starts to feel like more than a “nice-to-have”, but technically, the app is totally functional if that info is supplied at run-time.

For a quick fix (albeit ugly), I can reference the structs via apply/3 so they don’t cause problems during build-time.

A better option might be to figure out a mix task/alias that would generate the necessary charts at runtime and include these pages linked up to the other ExDoc outputs.

1 Like
5

I found a solution to this, but it is only viable within specific parameters (and apologies in advance if I’m misrepresenting any of this due to my limited understanding).

To clarify, the application itself did not REQUIRE compile-time configuration: it is completely functional when this configuration is supplied at run-time. The motivation here came from the desire to expose the application configuration in the generated documentation. I wanted something like this:

@moduledoc """
Here is a nice chart:

#{ Application.get_env(:my_app, :stuff) |> ChartGenerator() }
"""

The trick here is that there is some wiggle room between build-time and compile-time. The config/config.exs is read during build-time, and per the docs, your dependencies and its structs are not available during build-time. However, it turns out that they are available shortly thereafter (i.e. during compile-time). Instead of storing this information inside of the Application config, I instead stored it in its own .exs file – I put it in the config/ directory so it would get included with builds and I accessed it using Code.eval_file/2

I could include this “config”-like stuff at runtime and pass it as an argument where it needed to go (instead of Application.get_env(:my_app, :stuff)):

def start(_type, _args) do 
  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(
      [
        {MyDep, [stuff: Code.eval_file("config/stuff.exs") |> elem(0)]}
      ]
    opts)
end

Likewise, I could also read it inside my @moduledoc blocks where I wanted the documentation to reflect the up-to-date information from that file:

@moduledoc """
Here is a nice chart:

#{ Code.eval_file("config/stuff.exs") |> elem(0) |> ChartGenerator() }
"""

Using Application.compile_env/3 wouldn’t work because the dependencies weren’t compiled yet (the dep’s structs were not yet available). Application.get_env/3 returned nil inside a @moduledoc tag (I guess because the runtime.exs hadn’t been parsed yet?).

So the lesson is that there is a bit more subtlety in how the app is compiled… maybe config/config.exs should instead be named config/build.exs and Application.compile_env/3 should be named Application.build_env/3? Or maybe I am still missing pieces to how this all fits together.

6

I’d really be curious why you need the data to be structs in config.exs.

You should be able to do something like this:

# config/config.exs
config :my_app, :stuff, 
  a: :b,
  c: :d

# lib/config.ex
defmodule MyApp.Config do
  @conf Application.compile_env!(:my_app, :stuff)

  def stuff do
    struct!(SomeDep.Struct, @conf)
  end
end

# some other module
defmodule MyApp.Module do
  @moduledoc """
  Here is a nice chart:
  
  #{MyApp.Config.stuff |> ChartGenerator() }
  """
end

Your modules can call into other modules of yours at compile time no problem. You don’t need to store the structs directs in the config/config.exs.

All of that is compile time, but as mentioned there’s a chicken-egg type problem as compile time like runtime cannot have hard circular dependencies. Some things need to be loaded/dealt with before others.

7

I’d really be curious why you need the data to be structs in config.exs .

The simple answer is, of course, that I don’t need to do it this way. I’m just trying to figure out a “low effort” solution that works with our current code while we further weigh the pros and cons. The structs used in the configuration are used in conjunction with a custom protocol – each implementation of the protocol defines supervisor children for an app that acts something like a job queue with lots of interconnected processes.

Previously, we had configured this with a list of simple maps; the functionality pivoted based on a :type key in the maps. However, this got a bit messy – partly because it was easy to fat-finger keys and partly because the list of all these maps became quite long because the maps had to be verbose (there aren’t default values in regular maps, so each key had to be included). The configs were sometimes thousands of lines.

After several trip-ups from the use of simple maps, we thought we’d try using structs and a custom protocol. Perhaps the most important factor was that the configuration became easier to read and document when defined with structs. Readability was helped because structs can require certain keys and provide default values, and by using a protocol, we could open the door for adding customizations on top of this dependency: we have multiple apps that use this dependency in different ways – otherwise we could feed the configs into calls to struct/2 as you demonstrated.

So this latest iteration has some pros and cons that I need to sit with before taking another stab at it. I’m always wary when a solution feels too much like “pushing the river”, so I don’t think this is where things will land permanently, but I’m not yet ready to retreat back to how we had this before.