Keeping system ENVs secure?

I’m seeking some advice to help the security posture of a package of mine, dotenvy. There has been some discussion around keeping system ENVs secure in one of the feature requests:

I’ve noticed that when one iex session uses System.put_env/1, that value is not shared with another iex session. This seems consistent with how the terminal works, where exporting a system ENV in one window does not make it available to another terminal window.

Is there a way to read system variables set via another OS process? In other words, how insecure is it to set system ENVs in Elixir code? (I realize that questions like this can require nuanced explanations).

The related question is whether or not storing variables in a process dictionary (e.g. via Application.put_env/2) offers any better security? If someone can access the running instance, then they could read those variables.

My assumption up till now has been that the protection of sensitive values is more holistic; in the case of day-to-day development, it’s better to have throw-away values that would not cause alarm if leaked, whereas the process of preparing/deploying a live “production” machine must be carefully safeguarded so any “live” credentials and the machine itself are not accessible. But I realize that such an assumption may be incomplete or wildly inaccurate so I wanted to subject this to some educated scrutiny from the community.

Thank you for your input!

TLDR don’t assume per-environment variables are secure. And consider your 'threat model" - what does an attacker gain access to, and what sort of attack sources (vectors) are you expecting, or trying to prevent here? If ultimately the secret is stored in config/secrets.exs then it’s already game over, just a File.read!/1 away.

That said, environment variables are a per-process setting. When process 1 forks a sub process, process 1’s environment is used as the base for the per-process settings of Process 2, in general. When process 2 updates its state, process 1 doesn’t change.

Whether process 1 and 2 can read each other’s state without root privileges, or other container privileges, depends on the OS.

https://web.cse.msu.edu/~cse325/Modules/Module04/module04.lecture2.pdf has a lot more detail.

On linux, the process state at launch, is available as a virtual file under /proc/:pid/environ and is string delimited by nulls. Thus tr '\0' '\n' < /proc/123/environ should show you something useful. Assuming pid 123 exists, is owned by the current user, etc.

I believe that if the env state is updated then this process state is no longer correct, but you can easily check that yourself.

IMHO this information shouldn’t be available to other processes by default. On FreeBSD and presumably other *BSDs this requires root privileges:

> ps -wwwep 67
PID TT  STAT    TIME COMMAND
 67  0  Is+  0:06.52  fish

> sudo ps -wwwep 67
PID TT  STAT    TIME COMMAND
 67  0  Is+  0:06.52 TERM=xterm-256color ... MORE... ENVS...  fish

For Windows, see https://devblogs.microsoft.com/oldnewthing/20150915-00/?p=91591 for a similar summary.

And on all platforms, if you have root, you can probably just read the raw memory directly.

Within elixir land, you can “stash” secrets inside an anonymous fun, which is then invoked on usage to retrieve the actual secret. These are effectively obfuscated, rather than actually encrypted. This anonymous fun can be stored as a persistent term if you use it repeatedly, or inside the process dictionary of a specific function / genserver if it’s only needed occasionally.

Remember that just because your process has freed a variable, does not mean that freed memory was over-written with zeros, and any hacker with sufficient privileges can read that memory and extract secrets directly.

If you really need it to be secure, then use something like a crypto box from enacl and put that inside the fun. But you’ll still have the original memory lying around until the BEAM re-allocates it, and it gets re-used. It’s almost impossible to avoid this in Elixir. enacl is sparse on docs, but easily the best maintained NIF for libsodium. There are surely other alternatives on hex, including ExCrypto — ExCrypto v0.10.0

For stronger solutions, people may be interested in GitHub - latchset/tang: Tang binding daemon (based on Tang and Clevis: shackling secrets to the network - Speaker Deck & http://www.admin-magazine.com/Archive/2018/43/Automatic-data-encryption-and-decryption-with-Clevis-and-Tang ) or any of the other key management systems such as hashicorp’s https://vaultproject.io/ and alternatives built into Azure, AWS, GCP etc.

The best way to deal with secrets is to move them “outside” your BEAM process, whether on a remote system, or via some other privilege separated process, and use a secure transport (UNIX domain socket, or OS pipes, or network TLS connection), moving the encryption/decryption outside your app, and therefore ensuring the secrets are never directly available. See slide 64+ on Privilege Separation in https://lteo.net/assets/pdf/lteo-openbsd-carolinacon15-20190427.pdf

My personal preference (for production) is to store secrets in a “vault” like hashicorp vault or 1password, and to pass a single one-time-use key in as environment variable, to allow Elixir to retrieve them securely, and directly into an anonymous fun (which can be put in the process dictionary, or inside a GenServer state), for invocation & usage. This is called “response wrapping”, see Response Wrapping | Vault by HashiCorp for a good explanation.

7 Likes

Thank you for this very thorough breakdown. Much appreciated!