Phoenix and Heroku pipelines issue

We are using heroku to deploy Phoenix and Elixir based applications. Also we use heroku pipelines to promote from staging to production. I have set for each env the proper MIX_ENV (staging and prod)
I am having an issue regarding accessing config variables for each environment.

In staging.exs I have:

 config :facebook_menu,
   domain_url: "staging.example.com"

In prod.exs I have:

 config :facebook_menu,
   domain_url: "example.com"

I am loading this variable in a Phoenix controller like this:

    def show(conn, %{"id" => id}) do
       domain_url = Application.get_env(:facebook_menu, :domain_url)
       render conn, "show.html", id: id,
                          domain_url: domain_url,
                          facebook_app_id: @facebook_app_id
    end

Inside the show.html.eex template I am using this variable:

   <div id='menu'
            data-channel-url="//<%= @domain_url %>/v3/channels/facebook.json"

When I deploy to staging I have the staging domain.
But when I promote it to production I still have the staging value.

From my current level of understanding since I am loading this variable inside a function and not as a module attribute it is loaded at runtime not at compile time.
Does heroku moves the staging to production without restarting it and loading new environment variables?
I have no idea how to fix this other than not using heroku pipelines.

The source Application.get_env/2 reads from is built at compile time. So it doesn’t matter if you call it inside a funtion or in a module attribute.

And as far as I was able to understand the heroku docs on a quick skim, it seems as if the complete container is promoted as is, by simply changing the environment, but not recompiling stuff.

Also this skim did reveal some options to work around:

  • Use release-phases between promotion
  • Read from an environment variable directly

Since I’m not even sure if your app gets restarted on promotion you will probably even need to do it every single time and are not able to push it into app init phase…

This is just plain false. The config is loaded when the application starts.

1 Like

I wrote a ticket to Heroku and they aswered me that the problem I have is due to elixir custom buildpack that I am using: https://github.com/HashNuke/heroku-buildpack-elixir

They said that the profile is created at build time by buildpack. When we use Pipeline promotions there is no build when releasing to production. That means the application is not recompiled. So the only solution to deploy to staging and production and to compile again for each one is not to use pipelines or to set MIX_ENV=production also for staging.

Thank you guys for help

Well, from where? When I compile with MIX_ENV=staging, will I ever be able to access something that is configured for MIX_ENV=prod?

Yes. Mix config is loaded when mix starts. Each time you run mix compile or mix run or mix phoenix.server the entire config is loaded. Config is not persisted in any way.

but when we say we are running our released app, that we do not use mix to start it, but a self-contained escript?

That’s the first time this is mentioned in this thread. This indeed changes things significantly. The usual way to run on Heroku is running with mix.

Didn’t know that heroku uses mix to run the app… I assumed OTP releases just because of that’s the only way I can imagine outside of development.

Heroku does not support Elixir by default. I am using a custom buildpack that Heroku does not maintain: https://github.com/HashNuke/heroku-buildpack-elixir
The support guys that I talked to said that this buildpack causes the problem. When promoting the dyno from staging to production it does set MIX_ENV=staging overriding the one from machine.
The application on Heroku is started using mix phoenix.server from Procfile.

Since Elixir is a compiled language and since there are things that are built at this step like module attributes and logger compile_time_purge_level I prefer to deploy them separately and compile the app again.

Same issue here: I have a Heroku pipeline of 2 apps, staging & production.
Each app is properly configured with correct MIX_ENV value, but when I run a iex session on my production app :

iex(1)> Mix.env
:staging

Restarting the app does not change anything.

I guess the app need to be recompiled on each environment, which defeats the concept of Heroku app promotion :confused:

Did Heroku support say how the buildpack should be fixed? MIX_ENV is written into the profile so that we can set a default environment if it’s not set by the use. It was my understanding that the heroku config vars should override the profile. It also looks like the official buildpacks does the same thing heroku-buildpack-ruby/lib/language_pack/rack.rb at ce11aca649273103dfeed0ccc5880858be116aa6 · heroku/heroku-buildpack-ruby · GitHub.

The issue also seems to instead be that that you are reading from the environment at compile time, these values obviously wont change unless you recompile the application regardless if the buildpack is fixed or not. If you move the Application.get_env from the module attribute into the function that is evaluated at runtime it may work.

You can try my branch of the buildpack that might have a possible fix for your issue. Now we only set MIX_ENV in profile.d if the user haven’t specified their own MIX_ENV for the application.

My branch: GitHub - HashNuke/heroku-buildpack-elixir at emj/default-mix-env. Use it by running: heroku buildpacks:set -a MYAPP -i 1 https://github.com/HashNuke/heroku-buildpack-elixir\#emj/default-mix-env.

2 Likes

Thank you @ericmj for your help. I will try your branch and let you know how it works after I test.

Thanks for your help @ericmj !

I just provisioned a new Heroku app using your buildpack and then promoted my staging app to it. Unfortunately it did not solve my issue :confused:

Mix.env/1 and Application.get_env/2 are still returning staging configuration.

I’m even surprised that System.get_env("MIX_ENV") is returning "staging". I double-checked my setup and MIX_ENV value is correctly set to prodon this environment

Some progress here : I also updated my staging environment to use your buildpack branch.

Now, when I promote to production, I have a new error during the release phase, heroku complains it is not finding rebar. Quite similar to this issue : https://github.com/HashNuke/heroku-buildpack-elixir/issues/84

It seems that it cannot run the app under prod environment when dependencies were installed under staging env

imho, and just my 2 cents - your staging env on heroku should be as close to your production env, eg. MIX_ENV should be prod in both of them, and then the ENV variables will be different for DB,S3 etc etc.

this is also what heroku recommends:

You should use RAILS_ENV=production or RACK_ENV=production for your staging applications to minimize surprises while deploying to production Managing Multiple Environments for an App | Heroku Dev Center

Heroku highly recommends keeping your development and production environments as close together as possible. This is called dev/prod parity. We also recommend keeping your staging as close to production as possible. If they are too different, deploying and testing on a staging app does not verify your production instance will work. Deploying to a Custom Rails Environment | Heroku Dev Center

So I would very much keep MIX_ENV the same especially in elixir where build/compile is different in :prod and having a third behaviour for staging, makes the whole point of staging moot.

1 Like

You need to change the staging app to use the new buildpack, because we no longer persist the MIX_ENV in the first stage of the pipeline.

Sorry, I have no idea why this is happening. I would suggest starting a dyno with only bash and check if the files are where they are supposed to be. Start with heroku run -a MYAPP bash and check if rebar files are where the buildpack said they put them.

This is definitely what you should do. Mix assumes your app runs in the environment you build, for example you have to set build_per_environment: false in your mix project if you don’t.

2 Likes

Ok thanks everyone for your help, I finally updated my app to use same Mix.env == :prod in both staging & production environments and move anything specific to environment variables.

Now everything is working and is much more simple!

1 Like

Thank you guys for the idea to put MIX_ENV to “prod” booth on staging and production.
I managed to have heroku pipelines now like this:

  • I put MIX_ENV=prod on staging and also on production so that is not required to compile again the code.

  • I left the standard buildpack for elixir: https://github.com/HashNuke/heroku-buildpack-elixir.git

  • I extracted the config variables that are different for each environment to system ENV vars and instead of setting them in module attributes I added methods that loads the variable at runtime like this:

    defp api_domain do
    System.get_env(“API_DOMAIN”)
    end

Now everything works ok. I can promote servers on pipeline and the new variables are visible. So I can say I am pleased with the result.

I have a question regarding this approach: Getting the variable from system environment for each call does not have a performance penalty instead having the value inside module attribute? I know that it’s insignificant but still I am asking.

Thanks to all for your help.