Filter sensitive logging data

wfgilman

I am getting closer to putting my Elixir backend API in production and have a question about logging. Currently when a 500 error occurs in an API call, the entire Plug is posted to my logging service. This includes the JWT used in the request and secret key base used to decrypt the JWT.

Is there any way to filter this sensitive data from being written to the logs? Phoenix does this automatically for things like user passwords, and I thought Ecto did something similar. Unfortunately I can’t seem to find documentation on it.

8 comments

/phoenix #ecto #logging #plug

1 5064 8

2019-07-05 14:08:06 UTC

Showing Posts 1 to 8

Show Best Posts
Show All (oldest first)
Show All (newest first)

outlog

config :phoenix, :filter_parameters, ["password", "secret"]

or check your logging service for equivalent..

Post #1

wfgilman

Gotcha, but that’s just for incoming parameters correct? There’s no way to filter sensitive information logged during an error?

Post #2

Phillipp

I have the same situation right now. I am making a request to a remote system and if it fails, I log the entire HTTPoison response, which might contain login credentials. Currently:

Body:
{:form, [{"email", "user@example.com"}, {"password", "secret1234"}]}

But maybe in the future, if the remote API accepts a JSON body, the body will be a JSON string. So something more general would be useful. A quick search on hex.pm didn’t yield any results.

Post #3

outlog

can you post full (pseudo) code and error (sensitive data deleted..)

I imagine one has to filter manually - but maybe it’s possible to create/have a “deep” nested filter that goes through things..

eg
config :phoenix, :deep_filter_logs, [“password”, “secret”]

Post #4

Phillipp

Basically, what I do is this:

defimpl AppGateway.Error, for: AppGateway.Errors.RemoteServiceError do
  def status(_), do: AppGateway.Errors.RemoteServiceError.status()

  def log_message(error) do
    """
    Error while calling Remote Service!

    Method: #{error.method}
    Url: #{error.url}

    Headers:
    #{inspect(error.headers)}

    Body:
    #{inspect(error.body)}

    Response:
    #{inspect(error.response)}

    Error:
    #{inspect(error.error)}
    """
  end
end

error.body is what I passed to HTTPpoison and error.response is what I get from the {:error, value} tuple from HTTPoison, which is in most cases a %HTTPoison.Response{} struct which also contains the full %HTTPoison.Request{} with the body.

Even manually filtering it is quite some work and the maintenance efforts are high. What I hoped was a clever algorithm that uses the string representation of the structs and filters out the data.

Post #5

outlog

look at nested_filter nested_filter | Hex or deep_clean deep_clean | Hex

else a regex on the string containing sensitive data?

Post #6

NobbZ

I think filtering the actual data is much easier and less error prone than an unstructured string dump of the data…

Post #7

Phillipp

Seems like the nested_filter lib only works with maps but I am dealing with quite a few structs. Also, if the request body contains a JSON string, it won’t help at all.

Post #8

— All posts loaded —