Mysterious magical :prod option in mix.exs

Yesterday I encountered a very mysterious behaviour.
A dependency was started in :prod while my app was compiling in :dev.

I use various home-made apps as dependencies, hosted at GitLab.
This is from a Exto.MixProject I am using:

  defp deps do
    [
      {:my_dep_x, git: System.get_env("MIX_MY_DEP_X", "git@gitlab.com:my-user-name/my_dep_x.git"), system_env: deps_env()},
      {:my_dep_y, git: System.get_env("MIX_MY_DEP_Y", "git@gitlab.com:my-user-name/my_dep_y.git"), system_env: deps_env()},
      {:ecto,        ">= 3.5.6"},
      {:ecto_sql,    ">= 3.5.4"},
      {:postgrex,    ">= 0.15.8"},
      {:jason,       "~> 1.2.2"},
      {:elixir_uuid, "~> 1.2.1"}
    ]
  end

Environment variables are used to fetch private repositories.
This is from a .gitlab-ci.yml I am using:

test:
  stage: test

  variables:
    MIX_ENV: "test"
    MIX_MY_DEP_X: "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/my-user-name/my_dep_x.git"
    MIX_MY_DEP_Y: "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/my-user-name/my_dep_y.git"

I use the system_env option to “pass down” the “mix env”.
I never use Mix.env(), instead I use the following:

  def mix_env do
    case System.get_env("MIX_ENV") do
      "prod"   -> :prod
      "test"   -> :test
      _default -> :dev
    end
  end

  def deps_env do
    [{"MIX_ENV", Atom.to_string(mix_env())}]
  end

After refactoring my_dep_y it is now (also) depending on my_dep_x.
Just now I ran mix deps.clean --all for my current project that depends on both x and y.
Then I noticed something I can’t explain.

$ mix deps.get
Dependencies have diverged:
* my_dep_x (git@gitlab.com:my-user-name/my_dep_x.git)
  the dependency my_dep_x in mix.exs is overriding a child dependency:

  > In mix.exs:
    {:my_dep_x, [system_env: [{"MIX_ENV", "dev"}], env: :prod, git: "git@gitlab.com:my-user-name/my_dep_x.git"]}

  > In deps/my_dep_y/mix.exs:
    {:my_dep_x, [env: :prod, git: "git@gitlab.com:my-user-name/my_dep_x.git"]}

  Ensure they match or specify one of the above in your deps and set "override: true"
** (Mix) Can't continue due to errors on dependencies

Notice env: :prod.
This probably explains why a dependency was started in :prod while my app was compiling in :dev.

I will try to override this magical setting manually.
Maybe someone wants to look into this.
I wouldn’t want my apps to suddenly start compiling/running in prod mode.

Dependencies are always compiled and run under :prod and that’s expected behavior. The mix env only affects the applications of the current mix project (can be many in umbrellas).

7 Likes

Thank you Benjamin, I didn’t know such a setting existed.
I can see why you would want dependencies to run in prod by default.
But why would it not be possible to override env: :prod.

I updated my_dep_y like this:

  defp deps do
    [
      IO.inspect(
        {:my_dep_x, git: System.get_env("MIX_MY_DEP_X", "git@gitlab.com:my-user-name/my_dep_x.git"), system_env: deps_env(), env: mix_env()},
        label: "MY_DEP_Y.MixProject fetch :my_dep_x")
    ]
  end

After running mix deps.clean --all I ran mix deps.get.

MY_DEP_Y.MixProject fetch :my_dep_x: {:my_dep_x,
 [
   git: "git@gitlab.com:my-user-name/my_dep_x.git",
   system_env: [{"MIX_ENV", "dev"}],
   env: :dev
 ]}
* Getting my_dep_x (git@gitlab.com:my-user-name/my_dep_x.git)
remote: Enumerating objects: 47, done.        
remote: Counting objects: 100% (47/47), done.        
remote: Compressing objects: 100% (44/44), done.        
remote: Total 47 (delta 16), reused 0 (delta 0), pack-reused 0

Looks good.
Then I returned to my current project to update deps/0.

  defp deps do
    [
      {:my_dep_x, git: System.get_env("MIX_MY_DEP_X", "git@gitlab.com:my-user-name/my_dep_x.git"), system_env: deps_env(), env: mix_env()},
      {:my_dep_y, git: System.get_env("MIX_MY_DEP_Y", "git@gitlab.com:my-user-name/my_dep_y.git"), system_env: deps_env(), env: mix_env()},
      {:ecto,        ">= 3.5.6"},
      {:ecto_sql,    ">= 3.5.4"},
      {:postgrex,    ">= 0.15.8"},
      {:jason,       "~> 1.2.2"},
      {:elixir_uuid, "~> 1.2.1"}
    ]
  end

Notice env: mix_env().
Run mix deps.clean --all and mix deps.get.

$ mix deps.get
* Getting my_dep_x (git@gitlab.com:my-user-name/my_dep_x.git)
remote: Enumerating objects: 47, done.        
remote: Counting objects: 100% (47/47), done.        
remote: Compressing objects: 100% (44/44), done.        
remote: Total 47 (delta 16), reused 0 (delta 0), pack-reused 0        
* Getting my_dep_y (git@gitlab.com:my-user-name/my_dep_y.git)
remote: Enumerating objects: 35, done.        
remote: Counting objects: 100% (35/35), done.        
remote: Compressing objects: 100% (32/32), done.        
remote: Total 35 (delta 11), reused 0 (delta 0), pack-reused 0        
Dependencies have diverged:
* my_dep_x (git@gitlab.com:my-user-name/my_dep_x.git)
  the dependency my_dep_x in mix.exs is overriding a child dependency:

  > In mix.exs:
    {:my_dep_x, [system_env: [{"MIX_ENV", "dev"}], git: "git@gitlab.com:my-user-name/my_dep_x.git", env: :dev]}

  > In deps/my_dep_y/mix.exs:
    {:my_dep_x, [env: :prod, git: "git@gitlab.com:my-user-name/my_dep_x.git"]}

  Ensure they match or specify one of the above in your deps and set "override: true"
** (Mix) Can't continue due to errors on dependencies

Doesn’t look good. env: :prod.

Maybe you’ll ask: why would you want to override this setting.
That’s because some of my dependencies use a database.
In my overall project I use multiple (independent) Repos.
Those Repos need a connection based on “mix env” (or something like that).
Some of the dependencies actually compile data resulting from executing queries.
Everything is compiled into functions.

Maybe you’ll ask: why don’t you use umbrella projects.
That’s because I don’t like the concept, I like my concept better.
Also I am planning to use some of the dependencies in other projects.
To have very specialized well tested repositories makes me feel good.
Also I enjoy the CI/CD implications.

While the dependencies do compile/run in :prod mix env you can still configure them from the mix project using those dependencies. So you should be able to add configuration for those deps accordingly.

Like this:

{:dep_name, git: "git@gitlab.com:user/repo.git", env: Mix.env()}

That’s not what I meant. I was talking about this:

# config/dev.exs
import Config

# Configure dependency :dep_name
config :dep_name, …

# mix.exs
{:dep_name, git: "git@gitlab.com:user/repo.git"}
1 Like

Thank you both.
Actually that’s what I did.
I guess I will also have to set "override: true".

Only when you are overriding other dependency already loaded, like when you want to load your own version of some library you have customized, or you are working on to fix a bug or add a fetutre.

1 Like

I guess I don’t understand the purpose of the setting.
Maybe Mix.env/0 should be deprecated to avoid a lot of confusion.
(or maybe I’m just not smart enough)

My solution will probably look like this.

mix_env =
  case System.get_env("MIX_ENV") do
    "prod"   -> :prod
    "test"   -> :test
    _default -> :dev
  end

Like you I also load a lot of deps from my Gitlab repos and only need to do as:

So, no deed for this:

You don’t need to write your own function to get the MIX_ENV var, because Elixir already provides Mix.env().

Can you explain why your dependency needs to know the mix env in the first place? Switching out the repo connection details is imo configuration and config shall be controlled by your project and not by dependencies. If your dependencies have a fixed set of repo options, than you can still expose the selection via configuration e.g. like so: config :dep_name, repo: :dev. This way it is explicitly defined and not implicitly bound to the mix env, which by default isn’t even inherited to dependencies.

2 Likes

But what if the dependencies are configured like this:

MyProject  <-- Y  <-- X

I think there’s a problem between Y and X right?

That’s another thing solved by having configuration dealt with only at the top level (MyProject here). Given config of dependencies are ignored in mix Y could even have different config for X, for working on Y as opposed to MyProject.

1 Like

Lots of stuff to think about.

Thank you both.