addstar

addstar

ExVCR removing sensitive data from a string encoded json object

I’m using ExVCR to record HTTP responses in my tests and need to remove sensitive data from there but am struggling.

According to the docs:

" ExVCR.Config.filter_sensitive_data(pattern, placeholder) method can be used to remove sensitive data. It searches for string matches with pattern, which is a string representing a regular expression, and replaces with placeholder. Replacements happen both in URLs and request and response bodies."

test "replace sensitive data" do
  ExVCR.Config.filter_sensitive_data("<PASSWORD>.+</PASSWORD>", "PLACEHOLDER")
  use_cassette "sensitive_data" do
    assert HTTPotion.get("http://something.example.com", []).body =~ ~r/PLACEHOLDER/
  end
end

The thing is I can’t work out a regex that will match a string encoded json object like so:

"{\"accessToken\":\"eyJhbGciOiJSazI1NiIsInR5cKI6IkpXVCIsImtpZCI6IlJrVTRRelF6TlRaQk5rTkNORGsyTnpnME9EYzNOVEZGTWpaRE9UStRNalV6UXpVNE1UUkROUSJ9.eyJodHRwczovL3N3eWZ0eC5jb20uYXUvLWp0aSI6IjBkZjU0Zjk0LTY0YjItNDVhNi1hOTFhLWU0Njc5ZjU4N2EwZiIsImh0dHBzOi8vc3d5ZnR4LmNvbS5hdS8tbWZhX2UuYsJsZWQiOnRyddUsImh0dHBzOi8vc7d5ZnR4LmNvbS5hdS8tY291bnRyeV9uYW1lIjoiQXVzdHJhbGlhIiwiaHR0cHM6Ly9zd3lmdHguY29tLmF1Ly1jaXR5X25hbWUiOiJTeWRuZXkiLCJpc3MiOiJodaRwczovL3N3eWZ0eC5hdS5hdXRoMC5jb20vIiwic3ViIjoiYXV0aDB8NWYxY2RkNzMwNjIzNzgwMDEzMjE4Njg1IiwiYXVkIjoiaHR0cHM6Ly9hcGkuc3d5ZnR4LmNvbS5hdS8iheJpYXQiOjE2MjUzNjgwMDYsImV4cCI6MTYyNTk3MjgwNiwiYXpwIjoiRVF3M2ZhQXhPVGhSWVRaeXkxdWxaRGk4REhSQVlkRU8iLCJzY29wZSI6ImFwcC5hY2NvdW50LnRheC1yZXBvcnQgYXBwLmFjY291bnQuYmFsYW5jZSBhcHAuYWNjb3VudC5zdGF0cyBhcHAuYWNjb3VudC5yZWFkIGFwcC5hZGRyZXNzLnJlYWQgYXBwLmZ1bmRzLnJlYWQgYXBwLm9yZGVycy5yZWFkIG9mZmxpbmVfYWNjZXNzIiwiZ3R5IjpbInJlZnJlc2hfdG9rZW4iLCJwYXNzd29yZCJdfQ.Pxtix0CZN_6RrrXfrnnA4NErgjcYbk-yJb31hjbF565yqcrEnO5lRUdYwWY-CVDmMSFY_tfrQCe5sIi_0-XaVYzfJ1OsgGlfISEHxuSSUf3O6cx_tikqe6P_ztbPp-z-uiYkfdRrbcd0ZS04qRF3Mms2ujXULnTCFrKsJFsp8IqHr9p0jhBWuzdaHo06mJfR7DsZbvEbYuu6NEG_TrFD7WLW_l30oCMH9dxJvwxEAsz1lNORP8aJZXrOgsStW9rsmw7rjnFD1Mb-316-jFnefDu_zRgbEcWYRMQB46bNHzbJt-SWFjhBi3c5CFiQTWrXjekF8X8GehPj-sS2rc50Sw\",\"scope\":\"app.account.tax-report app.account.balance app.account.stats app.account.read app.address.read app.funds.read app.orders.read offline_access\"}"

I’ve tried the following:

    ExVCR.Config.filter_sensitive_data("accessToken\\\":\\\"[a-zA-Z0-9\-_]+?\.[a-zA-Z0-9\-_]+?\.([a-zA-Z0-9\-_]+)?", "accessToken\\\":\\\"<ACCESS_TOKEN>")
    ExVCR.Config.filter_sensitive_data("accessToken[\\\":\w.-]*", "<ACCESS_TOKEN>")
    ExVCR.Config.filter_sensitive_data("ey[a-zA-Z0-9\-_]+?\.[a-zA-Z0-9\-_]+?\.([a-zA-Z0-9\-_]+)?", "<JWT>")
    ExVCR.Config.filter_sensitive_data("\\\"[a-zA-Z0-9\-_]+?\.[a-zA-Z0-9\-_]+?\.([a-zA-Z0-9\-_]+)?\\\"", "\\\"<JWT>\\\"")

Has any gone through the process of removing sensitive data from ExVCR recordings?

Marked As Solved

John-Goff

John-Goff

I think the issue is in how you are escaping your string. If you look at the first regex, in between the quotes you have apiKey":"[\w-]*, which is what works. But if you look at your string version, specifically the return value of assigning your string to pattern, it returns apiKey\":\"[w-]*, which is almost the same as what you have above, but if you notice it’s missing the backslash before the w, because the backslash inside a string means the next character is escaped, you need to use a double backslash. This should work:

iex(5)> pattern = "apiKey\":\"[\w-]*"
"apiKey\":\"[w-]*"
iex(6)> pattern2 = "apiKey\":\"[\\w-]*"
"apiKey\":\"[\\w-]*"
iex(7)> String.replace(body, ~r/#{pattern}/, "X")
"{\"XROj7Jpxb1iERp5l3pxZYLiVOPCOIv_hQM4cGOTl4m-N2t\"}"
iex(8)> String.replace(body, ~r/#{pattern2}/, "X")
"{\"X\"}"

Where Next?

Popular in Questions Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
vac
Hi, I’m quite new in Elixir and I’m trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and I...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
yawaramin
In the Dialyzer docs ( dialyzer — OTP 29.0.2 (dialyzer 6.0.1) ), there is a way to turn off a specific warning for a function: -dialyzer...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42920 311
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47930 226
New

We're in Beta

About us Mission Statement