If it’s a static key that never changes, you can just put it in a config file which only your service can read. If you feel the file system is not secure enough (for example, some dude steals your whole server), you can use LVM to encrypt the whole hard drive with a passphrase only you know.

That’s a good idea. My actual prod implementation is not revealed. I use the env var in the example for “ease of use”.

Encrypting the hard drive is always good advice!

Nice post thanks!

It’s not clear what your “threat model” is (read https://www.usenix.org/system/files/1401_08-12_mickens.pdf for some helpful humour on this BTW).

My threat model is that due to some unexpected bug in our web stack (OS → reverse proxy →
container → phoenix+BEAM) the code & config files are exposed.

I don’t account for state-level actors (govt spies, etc) who can subvert datacenter controls and gain physical access to the servers, for example. We use separate environments and keys for dev/test vs prod, too.

In this scenario I can expect all of these to be exfiltrated (stolen):

• the code
• the config files
• any env vars I have
• other runtime stuff lying around in filesystem

The only thing I’d add to your approach is never to store the the actual key in the code or env vars, if possible.

My general approach for securing keys is as follows:

• use https://vaultproject.io/ or similar as a “cubby hole” to pass a one-time use key from the OS into the container, via env var
• the container grabs this key, and if its invalid we crash (and its a security problem as the key has been used outside this container). If not, it unwraps the actual key(s) we need, and the one-time-use key (in an env var), is of no value anymore
• the key is wrapped up in a fun() , which doesn’t show up as readable anymore in plaintext in a crashdump or similar OTP error message. This won’t stop somebody who has root level access in the container, but this requires an attacker to gain access to the container, and in addition gain root privileges. We hope our monitoring detects this!
• every time we need the fun() we invoke it as part of building the API request. the cost of the fun is lost in the noise of the API call over the network.

This doesn’t solve the problem but it does make it harder to subvert.

Personally, I prefer APIs where the authentication is added as an HTTP Header (like Authorization: Bearer abc123deadcafe123 as this can be injected outside the container, on valid outbound requests, by a reverse proxy. This way, the container can be spun up without network access at all, no root privileges, and no useful credentials. This puts a lot of trust in the reverse proxy, but it’s not in the direct path of an attacker, so we win a little bit more here. But in your case you have a (very nice) per-user API key so that doesn’t work.

It’s also possible to have a key generated on the fly for each invocation of your program, but that’s advanced vault and may be overkill for your needs.