I’m getting ready to deploy my first Elixir app, and it manages very sensitive data. I plan to use Distillery for releases, and I’m clear on how it manages configuration. What I’m not sure of is the best way to handle secrets like API keys and the Ecto Repo DB credentials. Here are a few approaches I’ve considered:
Use config.exs or prod.exs: This isn’t compatible with Distillery because the values would be evaluated at compile time. There is also the risk that these types of files might be checked into a Git repository accidentally.
Use environment variables as discussed on the plataformatec blog and promoted by the 12-Factor App: This approach is not universally embraced. The overall consensus of this discussion seems to be that environment variables were never intended to be private, and they may be inadvertently exposed. Many argue that files are inherently more secure because access to them can be controlled and reasoned about through permissions, which is lacking for the environment.
Store the configuration in a file located away from the code, such as in /etc. This seemed appealing, until I released that I wouldn’t be able to lock down the permissions very much. The user that the release is running as will need read access, and isn’t that the riskiest user to have access?
Encrypt the secrets using something like AWS KMS. On app startup, send an HTTP request to decrypt them and store them only in memory. Yes, we still need to store the AWS credentials somehow, but at least we can restrict access by IP, and all KMS requests are logged so we can configure some type of alert every time a decryption call is made. But is this even possible? Would I then set the env manually using Application.put_env? Would I need a separate app to handle configuration decryption and loading before the target app is executing? Do I need to be worried about this warning in the docs? Do not use the functions in this module for directly accessing or modifying the environment of other applications (as it may lead to inconsistent data in the application environment).
I thought maybe I could use the transforms feature is Conform to handle decryption, but it looks like the output of this tool is an erlang sys.config file that gets written to disk. That doesn’t solve the problem.
I’m sure this must be a solved problem. How are others dealing with this? Am I just being unreasonably paranoid?
I’m also interested in this, but I also don’t have anything in production. I am guessing there are no replies since it would probably fall under the category of “it depends on your requirements”. So the “solved problem” aspect is non-trivial I would imagine.
You’ve already listed the available options that I know of as well.
This does remind me, however, of the excellent recent ElixirConf Orlando closing keynote with @boydm about needing to talk even more about security though. (I am paraphrasing…I don’t remember the exact quote). I think it was partially in context with IoT / Nerves, but possibly more generalized. Maybe we could add some of the options as you have listed to the Deployment Wiki so with further clarification, the dynamics of this can be more clear, i.e. more of a “solved problem”?
I’ve been thinking about this issue as well along with the second step of deploying to a non-container but on a different architecture. Currently experimenting with creating the build on a target server that has the secrets as a file already on the server. Still looking for the best options.
One of the things that I know of certain companies that require high security do, is to store the configuration settings in an encrypted file (stored in a logical place, away from the code somewhere, i.e. using your approach number 3.). The program will then prompt for the proper passphrase to decrypt this file when it is started.
This means that the only way for an attacker to read the secret configuration information, is to either be able to read your keystrokes while you restart the server, or read (and basically disassemble) part of the RAM that your running program is using.
I’ve also heard about servers deployed this way that ran fine for years, and then when they finally had to restart, they asked for a password that nobody remembered anymore…
We use #4 in production now and it works great. All of our apps have an upstart script that decrypts a file stored with the repo (environment.json.enc) and passes those as command arguments to our elixir application.
At the end of Paul Schoenfelder’s (@bitwalker) excellentdeployment talk on Distillery, someone asks this very question about managing secrets. His response is brief and I won’t paraphrase!
Apparently, if you use Docker Swarm then you have access to a new Docker Swarm Secrets API which is supposed to be a blob storage for things ranging from just passwords to cryptographic keys, etc. I personally haven’t used this yet, but in the near future I hope to upgrade my docker-compose approach to use Swarm and I look forward to seeing how using the Secrets API goes (I’m currently using env variables).
While this isn’t Elixir specific, we have a two prong approach for handling our secret config for our highly available and global Asp.Net systems, utilizing both configuration in Environment Variables and in a configuration environment.
Originally we were doing all configuration via environment variables. This became tedious and difficult to properly maintain as we scaled the number of configuration settings we had to maintain as well as scaling our systems to different geographic data centers, (not to mention staging systems that can be swapped between prod and staging). A lot of configuration stayed the same between each region but some things didn’t and keeping track of that was difficult.
So how we handled it was by loading configuration twice. First past is reading the deployed configuration file (which really only has defaults) then overriding them with environment variables. We then pull the connection string out and then repeat the process with the database (so config file overwritten by database overwritten by environment variables).
Each server can specify in the environment variables a configuration profile (for example it’s region), which adds an extra layer of optional overrides. This allows us to do things like keep our east coast servers queuing up data in our east coast queues and keep latencies as a whole down.
This allows us to keep the environment variable maintenance down (only requiring database connection string and prod/staging specific settings) while making it very easy to query to see what settings will be applicable to a specific server. Not only that, since our databases are backed up in a way that allow us to restore to any point in time if something gets fubar we have a way to restore them back.
As part of another project I’ve been looking at ways to securely store secrets, and one of the things that stood out when trying Vault was the ways that apps could be used with it.
So for the case of getting credentials for an app, using the Cubbyhole secret backend seems like a good way to mitigate unwanted access to tokens.
The basic idea is;
Enable Cubbyhole secret backend. (Vault enables this by default)
Create temporary token with use count of 2. (The use count specifies number of times the specific token can be used)
Create permanent token.
Store permanent token in temporary token’s cubbyhole.
Pass temporary token to container application using environment variable.
Application reads permanent token from cubbyhole using temporary token.
Even if temporary token is read by malicious user later, there is no use for it since the use count for temporary token would have expired. Out of the specified initial use count of 2 for temporary token, first count is used when writing the permanent token and the second count is used when reading the permanent token.
I’m not sure how easy it would be to integrate, I’m thinking some sort of upstart script that gets the temporary token then passes it to the elixir app on startup, then internally the app does the rest - doing something like Preparing for Production - #2 by sasajuric
Some more info on response wrapping using Cubbyholes:
This post has an example of that, as well as the DB secret backend for authenticating DB users:
We are facing a similar issue in the latest project, so I’ve decided to release new Confex version which supports adapters. I plan to write an Hashicorp Vault adapter for it, which will allow storing secrets securely.
Also, secrets must be rotated and this is a separate long talk how to do that in an existing application with Ecto.
) mentioned on 
