awochna

awochna

Parsing JSON body where value of one of the keys is a string for another JSON object

Some background real quick!

I have an app that sends emails through AWS’s SES. I want to receive bounce notifications so I can properly mitigate bounced email issues. I’ve set everything up so that SES tells SNS about bounces and SNS has my application’s endpoint for sending notifications. All of this is working after a small hiccup: SNS subscription confirmation requests (and maybe others too) set the Content-Type header to text/plain even though the body is clearly JSON. They’ve confirmed this is a bug (Forums | AWS re:Post) and it means that some extra work is required to properly parse these requests using Plug/Phoenix. I have these requests coming through a special pipeline that includes a plug checking for this specific content-type and then overwriting the header to the more appropriate application/json. With that, the subscription confirmation is working. Thanks ex_aws_sns for making that so easy!

The problem

Now, I need to process the incoming messages for bounced emails. However, the incoming payload is JSON where one of the keys ("Message") maps to a string that is just a stringified JSON payload. I believe AWS does this so that you can confirm the rest of the message (using signature verification) without having the “trust” the value in "Message" first. Unfortunately, I think this breaks how Jason decodes JSON. Here’s an example incoming payload, parsed by a different system so that only the first “level” of JSON is decoded with anything remotely sensitive changed:

{
  "Message": "{"notificationType":"Bounce","bounce":{"bounceType":"Permanent","bounceSubType":"General","bouncedRecipients":[{"emailAddress":"bounce@simulator.amazonses.com","action":"failed","status":"5.1.1","diagnosticCode":"smtp; 550 5.1.1 user unknown"}],"timestamp":"[utc-timestamp-with-date]","feedbackId":"[some-extended-uuid]","remoteMtaIp":"[some.remote.i.p]","reportingMTA":"dsn; a1-2.smtp-out.amazonses.com"},"mail":{"timestamp":"[utc-timestamp-with-date]","source":"example@example.com","sourceArn":"arn:aws:ses:us-east-1:[some-12-digit-account-number]:identity/example@example.com","sourceIp":"[some.remote.i.p]","sendingAccountId":"[some-12-digit-account-number]","messageId":"[some-extended-uuid]","destination":["bounce@simulator.amazonses.com"]}}", 
  "MessageId": "[some-uuid]", 
  "Signature": "[base64encodedgarbage]", 
  "SignatureVersion": "1", 
  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-[some-hash].pem", 
  "Timestamp": "[utc-timestamp-with-date]", 
  "TopicArn": "arn:aws:sns:us-east-1:[some-12-digit-account-number]:ses-email-bounce", 
  "Type": "Notification", 
  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:[some-12-digit-account-number]:ses-email-bounce:[some-uuid]"
}

Perhaps because I have to manually change the Content-Type header, the body isn’t parsed by the time it hits the router’s pipelines. Right after the plug for changing the Content-Type header, I have:

plug Plug.Parsers, parsers: [:json], json_decoder: Jason

When my application receives the payload from SNS, this results in a Jason.DecodeError, the unexpected byte it’s referring to is always the first ‘n’ for the value of "Message" in the example above.

I just need to access the list of "bouncedRecipients" within the JSON payload that’s actually a string which is the value of a key inside another JSON payload that used to be a string.

Is this technically a bug in Jason? I haven’t worked with embedded payloads like this in Elixir/Phoenix/Plug/Jason before. I’m not seeing any options or other ways to decode only the “first level” of stringification in Jason. Is this something that ex_aws is adept at handling and I’m just missing something? Do I just need to write my own parser here?

Bonus points: What do you call this situation?

Most Liked

Nicd

Nicd

This is not a bug in Jason, as the snippet you posted is not valid JSON. Jason is working as it should, as it refuses to parse that.

al2o3cr

al2o3cr

If it is, it’s something more complicated than just the embedded JSON - a simple example works:

iex(4)> s = "{\"foo\": \"bar\", \"baz\": \"{\\\"blam\\\": \\\"wat\\\"}\"}"
"{\"foo\": \"bar\", \"baz\": \"{\\\"blam\\\": \\\"wat\\\"}\"}"
iex(5)> IO.puts(s)
{"foo": "bar", "baz": "{\"blam\": \"wat\"}"}
:ok
iex(6)> Jason.decode(s)
{:ok, %{"baz" => "{\"blam\": \"wat\"}", "foo" => "bar"}}

(this is on Elixir 1.9.0 with Jason 1.1.2, because that’s what was handy)

Where Next?

Popular in Questions Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54120 245
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement