Load env vars with docker-compose env_file, secrets

Hi all, I have an elixir umbrella app that I deploy via docker-compose. I’d like to use env_file directive in my docker-compose.yml for things like guardian keys etc. I have my env vars in a my_app.env file which is referred to in the docker-compose.yml. The vars do not appear to be available at runtime because when I try to generate a JWT with guardian, the phoenix endpoint generates a Bad Request. If I return the hardcoded key, it works OK.

Any recommendations to getting secrets or env vars injected into a running docker elixir app? I also tried docker secrets via the docker-compose file and I can confirm that the umbrella app cannot find the /run/secrets/secret_file path/volume, even though docker builds the image without error. I did notice that /run/secrets doesn’t appear in docker volumes ls but that might not be either here or there.

Docker secrets only work when you are using docker swarm.

And the environment variables should be available but perhaps you are reading them at the wrong time?

Can you tell us more about your build process and how you read from the environment?

1 Like

I haven’t deployed my app yet, but in development environment variables are working with Docker Compose using the same patterns that worked with other applications.

I have a .env file in the same directory as my docker-compose.yml file and then in the compose file I use:

    env_file:
      - ".env"

At this point all of the env variables are available in my Phoenix config files. For example you can use System.get_env("SOME_ENV_VAR_FROM_THE_ENV_FILE").

Are you doing something different, or maybe you forgot the env_file property on one of your services?

Is env_file: a child of services: level or a child of my_app: level?

I’m open to trying anything at this point.

It’s a child of myapp. Each individual service can have its own list of environment files (they are loaded and merged from top to bottom since technically you can have more than 1 file).

Basically, Docker isn’t going to automatically read in your environment variables. By setting env_file on a specific service, that instructs Docker to make them available.

The only file that gets automatically used is a .env file, but that’s for setting things like the COMPOSE_PROJECT_NAME or using variable substitution in the docker-compose.yml file itself. The vars in the .env file won’t get loaded into your services unless you explicitly set it in env_file.

Edit: There’s also the environment property which you can use either in combination with or as an alternative to env_file. The environment property lets you pass in key / value pairs of environment variables instead of having them loaded in from a file. Could be useful in some cases.

@maz, let me ask you again, how exactly is your build process?

The Dockerfile won’t see environment variables that are only injected at container runtime via an env file. So depending on if you are doing compiletime or runtime configuration, the environment might already be backed in as “empty”.

Sorry about the lack of response, I only had time this morning to make a quick response to @cnck1387. This is my docker compose with the env changes(they are not yet checked-in from my linux box at home which is where I can do quick turnaround development with docker)

I set - port: to localhost:4000:4000, commented out the web network, and commented-out the traefik directives in order to speed up iterations:

for building I do:

docker-compose pull 
docker-compose build --pull olivetree
docker-compose up --build -d postgres
docker-compose run --rm olivetree seed
docker-compose run --rm olivetree generate_hash_ids
docker-compose up --build olivetree

So going from what you are saying, I’d need access to those environment variables at least by docker-compose build --pull olivetree but that doesn’t seem to be the case.

[EDIT] added Dockerfile to gist.

yeah I’m still not sure why my env_file:\n - ./olivetree.env is not being honored. Or maybe it is but the timing of the SET is too late. It might be something I’m doing/not doing in my build process.

You can’t access environment variables from env_file at build time. It’s for run time.

If you want env variables to be set at build time, take a look at build arguments in the Docker Compose documentaton.

Right, so I guess I would want, say, System.get_env("GUARDIAN_SECRET_KEY") in config.exs to resolve at buildtime? Is there any situation where I’d prefer get_env() to resolve at runtime? Because if they can only be resolved at buildtime, I am barking up the wrong tree.

I never messed around with releases, but in development using mix to launch Phoenix, System.get_env("FOO") will resolve at run time and it will obtain that value thanks to Docker Compose making that environment variable available to Phoenix due to env_file (or environment) in the docker-compose.yml file.

You can double check what’s available in your container by running docker-compose run olivetree env. It should return back all of your environment variables that you set.

I see, that docker-compose run olivetree env call should help, thanks. I think the next thing could try is var substitution in the environment: section. So something like:

docker-compose.yml:

    environment:
      DATABASE_URL: ecto://olivetree:olivetree@postgres/olivetree
      HOSTNAME: localhost
      PORT: 4000
      GUARDIAN_SECRET_KEY: ${GUARDIAN_SECRET_KEY}
      ...

(source env vars at command line) OR $ export GUARDIAN_SECRET_KEY=efaw77aw6wfea6f
docker-compose up

You can get it to resolve at runtime by calling it in your code (as opposed to having it in config files, or module attributes, both of which resolve at build time). Some dependencies support taking config options at runtime, like Ecto. If, however, you want to configure eg dependencies using config/config.exs et al at runtime, Distillery supports it either through Config Providers or by setting REPLACE_OS_VARS=true and then defining your config like

config :hello, Hello.Repo,
  username: "${DATABASE_USER}",
  password: "${DATABASE_PASS}",
  hostname: "${DATABASE_HOST}",
  database: "${DATABASE_NAME}",
  port: "${DATABASE_PORT}",
  pool_size: 15

replace with your own options and env variables.

There’s more information in the DIstillery guide for working with Docker https://hexdocs.pm/distillery/guides/working_with_docker.html

Yeah, thanks for reminding me of Config Providers. I may end up needing to stop fighting Docker and use that instead.

EDIT: there are some good tips in there about using docker-compose too, guess I’ll try that first.