Filter sensitive logging data

Questions / Help
, , ,
1

I am getting closer to putting my Elixir backend API in production and have a question about logging. Currently when a 500 error occurs in an API call, the entire Plug is posted to my logging service. This includes the JWT used in the request and secret key base used to decrypt the JWT.

Is there any way to filter this sensitive data from being written to the logs? Phoenix does this automatically for things like user passwords, and I thought Ecto did something similar. Unfortunately I can’t seem to find documentation on it.

1 Like
2

config :phoenix, :filter_parameters, ["password", "secret"]

https://hexdocs.pm/phoenix/Phoenix.Logger.html

or check your logging service for equivalent…

2 Likes
3

Gotcha, but that’s just for incoming parameters correct? There’s no way to filter sensitive information logged during an error?

1 Like
4

I have the same situation right now. I am making a request to a remote system and if it fails, I log the entire HTTPoison response, which might contain login credentials. Currently:

Body:
{:form, [{"email", "user@example.com"}, {"password", "secret1234"}]}

But maybe in the future, if the remote API accepts a JSON body, the body will be a JSON string. So something more general would be useful. A quick search on hex.pm didn’t yield any results.

5

can you post full (pseudo) code and error (sensitive data deleted…)

I imagine one has to filter manually - but maybe it’s possible to create/have a “deep” nested filter that goes through things…

eg
config :phoenix, :deep_filter_logs, [“password”, “secret”]

6

Basically, what I do is this:

defimpl AppGateway.Error, for: AppGateway.Errors.RemoteServiceError do
  def status(_), do: AppGateway.Errors.RemoteServiceError.status()

  def log_message(error) do
    """
    Error while calling Remote Service!

    Method: #{error.method}
    Url: #{error.url}

    Headers:
    #{inspect(error.headers)}

    Body:
    #{inspect(error.body)}

    Response:
    #{inspect(error.response)}

    Error:
    #{inspect(error.error)}
    """
  end
end

error.body is what I passed to HTTPpoison and error.response is what I get from the {:error, value} tuple from HTTPoison, which is in most cases a %HTTPoison.Response{} struct which also contains the full %HTTPoison.Request{} with the body.

Even manually filtering it is quite some work and the maintenance efforts are high. What I hoped was a clever algorithm that uses the string representation of the structs and filters out the data.

7

look at nested_filter https://hex.pm/packages/nested_filter or deep_clean https://hex.pm/packages/deep_clean

else a regex on the string containing sensitive data?

8

I think filtering the actual data is much easier and less error prone than an unstructured string dump of the data…

9

Seems like the nested_filter lib only works with maps but I am dealing with quite a few structs. Also, if the request body contains a JSON string, it won’t help at all.