Sensitive data in stacktrace

tim2CF · January 17, 2019, 11:09am

Let’s say we have a module like

defmodule Foo do
  @salt Application.get_env(:foo, :salt, [])
  def hash(password) do
    try do
      {:ok, md5(password, @salt)}
    rescue 
      e -> {:error, {e, __STACKTRACE__}}
    end
  end
  defp md5(password, salt = [_ | _]) do
    Enum.reduce(salt, password, &:erlang.md5(&2<>&1))
  end
end

And when we call &hash/1 function it returns error

iex(19)> Foo.hash "123"
{:error,
 {%FunctionClauseError{
    args: nil,
    arity: 2,
    clauses: nil,
    function: :md5,
    kind: nil,
    module: Foo
  },
  [
    {Foo, :md5, ["123", []], [file: 'iex', line: 27]},
    {Foo, :hash, 1, [file: 'iex', line: 22]},
    {:erl_eval, :do_apply, 6, [file: 'erl_eval.erl', line: 677]},
    {:elixir, :eval_forms, 4, [file: 'src/elixir.erl', line: 265]},
    {IEx.Evaluator, :handle_eval, 5, [file: 'lib/iex/evaluator.ex', line: 249]},
    {IEx.Evaluator, :do_eval, 3, [file: 'lib/iex/evaluator.ex', line: 229]},
    {IEx.Evaluator, :eval, 3, [file: 'lib/iex/evaluator.ex', line: 207]},
    {IEx.Evaluator, :loop, 1, [file: 'lib/iex/evaluator.ex', line: 94]}
  ]}}

And we want to put this stacktrace to Logger (let’s say for easier debugging)
But we see, what stacktrace can contain sensitive data (in example is password = “123”)
Of course we can remove all arguments from stacktrace manually, but in most cases arguments are useful
Maybe there is already implemented solution to remove some arguments of some functions from stacktrace by marking them in module definition somehow?

It would be nice to say somehow that n-th argument of foo function in module Bar is sensitive and should be excluded from stacktrace

sfusato · January 17, 2019, 11:26am

There is ‘Custom struct inspections’ that was just released with Elixir 1.8:

github.com

elixir-lang/elixir/blob/v1.8/CHANGELOG.md

# Changelog for Elixir v1.8

Elixir v1.8 comes with many improvements at the infrastructure level, improving compilation time, speeding up common patterns, and adding features around introspection of the system.

## Custom struct inspections

Elixir now provides a derivable implementation of the `Inspect` protocol. In a nutshell, this means it is really easy to filter data from your data structures whenever they are inspected. For example, imagine you have a user struct with security and privacy sensitive information:

```elixir
defmodule User do
  defstruct [:id, :name, :age, :email, :encrypted_password]
end
```

By default, if you inspect a user via `inspect(user)`, it will include all fields. This can cause fields such as `:email` and `:encrypted_password` to appear in logs, error reports, etc. You could always define a custom implementation of the `Inspect` protocol for such cases but Elixir v1.8 makes it simpler by allowing you to derive the `Inspect` protocol:

```elixir
defmodule User do
  @derive {Inspect, only: [:id, :name, :age]}
  defstruct [:id, :name, :age, :email, :encrypted_password]

This file has been truncated. show original

Maybe return just the tail of the stack trace and at an upper-level inspect your struct with the returned stack trace. This would obfuscate the password field or whatever you set to hide from your struct.

josevalim · January 17, 2019, 12:11pm

Here is what Plug.Crypto does regarding the stacktrace: https://github.com/elixir-plug/plug_crypto/blob/76a595f26529cd550f32166c3288358e1785fb1f/lib/plug/crypto/key_generator.ex#L47

zazaian · October 11, 2019, 6:17pm

@sfusato’s response is very good, though it seems that in some instances you’d want to expressly name the excluded struct parameters, rather than list all of the unexcluded ones.

The Inspect protocol actually allows you to name the struct attributes that are excluded from inspection, which is cleaner for circumstances in which most of a struct’s attributes are not private or secret. Here’s an example:

defmodule CustomRequest do
  @derive {Inspect, except: [:password]}
  defstruct [:name, :email, :password]
end

If you had a myriad of secret attributes within a single struct though, using only is probably cleaner and safer, as adding additional secret attributes would not require you to also add them for exclusion:

defmodule CustomRequest do
  @derive {Inspect, only: [:name, :email]}
  defstruct [
    :name,
    :email,
    :password,
    :secret_token,
    :secret_id
  ]
end

Please note that the @derive call does have to be declared above the defstruct keyword as indicated in the Inspect docs, as shown above.

chocolatedonut · September 15, 2022, 8:57am

Despite all the helpful tips, I still am not able to hide password of the following :gen_statem-based RabbitMQ consumer:

defmodule Rabbit.Consumer do
  @derive {Inspect, only: [:channel]}
  defstruct [:config, :channel]
end

I’d like that the :config field (above), which includes the password, be hidden in all these cases:

:sys.get_state Rabbit.Consumer
when Rabbit.Consumer outright crashes and produces such stacktrace (observe password in plain sight)

[error] CRASH REPORT Process 'Rabbit.Consumer' with 0 neighbours exited with reason: 
  no such process or port in call to 
  gen_server:call(<0.1026.0>, {call,{'basic.ack',1,false},none,<0.1012.0>}, 15000) 
  in gen_server:call/3 line 382

[error] Supervisor 'Rabbit.Application' had child 'Rabbit.Consumer' started with
 'Rabbit.Consumer':start_link([
   {host,<<"localhost">>},
   {username,<<"admin">>},
   {password,<<"admin">>},
   {virtual_host,<<"/">>},
   ...]) 
   at <0.3177.0> exit with reason 
   {shutdown,{gen_server,call,[<0.3196.0>,
     {call,{'queue.declare',0,nil,false,true,false,false,false,[]},none,<0.3177.0>},15000]}} 
   in context child_terminated

What I tried already

`format_status`

I added :gen_statem c:format_status/2 and also OTP 25 c:format_status/1:

def format_status(s) do  # format_status/1 in OTP 25
  Map.drop s.data, [:config]
end

This did help as :sys.get_status didn’t print the sensitive :config anymore. But, it didn’t help with neither of the cases 1. or 2. above.

Prune args from stacktrace

I wrapped Rabbit.Consumer’s call to a different process with a rescue and pruned args from System.stacktrace() (using the suggested Plug.Crypto.prune_args_from_stacktrace/1). But, this doesn’t help when :gen_statem/Rabbit.Consumer itself crashes. How can one prune_args_from_stacktrace of the process itself (be it GenServer, :gen_statem, etc.)?

`Process.flag(:sensitive, true)`

def init(opts) do
  Process.flag(:sensitive, true)
  # etc.

Neither of the cases 1. or 2. above were solved.

Docs for the :sensitive flag say

Sets or clears flag sensitive for the current process. When a process has been marked as sensitive by calling process_flag(sensitive, true), features in the runtime system that can be used for examining the data or inner working of the process are silently disabled.

Features that are disabled include (but are not limited to) the following:

Tracing. Trace flags can still be set for the process, but no trace messages of any kind are generated. (If flag sensitive is turned off, trace messages are again generated if any trace flags are set.)

Sequential tracing. The sequential trace token is propagated as usual, but no sequential trace messages are generated.

process_info/1,2 cannot be used to read out the message queue or the process dictionary (both are returned as empty lists).

Stack back-traces cannot be displayed for the process.

In crash dumps, the stack, messages, and the process dictionary are omitted.

If {save_calls,N} has been set for the process, no function calls are saved to the call saving list. (The call saving list is not cleared. Also, send, receive, and time-out events are still added to the list.)

The following bolded part from above

“Features that are disabled include (but are not limited to)”.

made me hopeful that state will also be included (among disabled features).

So, I saw the following on erlef.github.io:

“Process state […] cannot be introspected”

Alas, this doesn’t seem to be true - :sys.get_state Rabbit.Consumer still shows the password.

Here’s this section verbatim:

Finally, a process can be marked as ‘sensitive’, using erlang:process_flag/2. This has the following effect:

Message queue contents cannot be introspected, and is not written to a crash dump

Process dictionary cannot be introspected, and is not written to a crash dump

Process state of a gen_server, gen_event or gen_statem process cannot be introspected, and is not written to a crash dump

Process heap and stack are not written to a crash dump

The process cannot be traced