Poison.Encoder exception when uploading PDF successfully to S3 with ExAws

Phoenix Forum Questions / Help
, , , , ,
1

In a bit of a continuation of my last post Bitstring with codepoints of non-ASCII characters, I’m trying to understand why the following issue with Poison happens.

I am uploading a PDF to AWS S3 with the PutObject action from the S3 API through the ExAws.S3.put_object/4 function. I have used this many times before and hadn’t stumbled upon this problem. The complications here arise from the fact that the PDF is received from an email from AWS SES that is first uploaded raw (the whole email) to AWS S3 separately. In it, the PDF attachment is base64 encoded. After decoding it and uploading it as a binary to AWS S3 I receive the following error:

** (FunctionClauseError) no function clause matching in Poison.Encoder.BitString.chunk_size/3
        (poison) lib/poison/encoder.ex:139: Poison.Encoder.BitString.chunk_size(<<255, 255, 255, 255, 255, 255, 255, 255, 255, 147, 97, 80, 94, 84, 131, 5, 40, 48, 108, 15, 6, 230, 195, 5, 108, 23, 42, 195, 87, 247, 13, 72, 192, 100, 255, 106, 8, 31, 251, 80, 64, 255, 216, 80, 131, 255, 13, 66, 15, 253, ...>>, nil, 0)
        (poison) lib/poison/encoder.ex:134: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:95: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:136: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:95: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:136: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:95: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:136: Poison.Encoder.BitString.escape/2

I should mention that I’ve tried with two different PDFs and the bitstring in the exception is, I suppose naturally, different in those cases, but everything else behaves in the same manner.

I have tried to pinpoint exactly where it happens and log messages after each function call and mostly this was happening in the logs after the ExAws.request!/2 calls, but I there were also cases when it was logged even before ExAws.S3.put_object/4, but I suppose this might be related to some Logger details. In general, I think it happens after the ExAws.S3.request!/2 call.

BUT, nevertheless, everything works as expected - the PDF is uploaded to AWS S3 as a binary and is a valid PDF, and everything after this point also works fine. What is even stranger is that our Phoenix application is configured to use Jason and ExAws is also configured to use Jason, but I still get the Poison exception. Which leads me to think that it might be from another library, but this means that I have even less of a clue as to what is going on, as ExAws is the only library that is used in this functionality, although there might be a bug with it, I guess.

Still, I always get this Poison exception. I assume it has something to do with the encoding of the email. I believe that the encoder expects utf-8 encoded data, but I’m not sure why the Poison encoder is called here in the first place and why this exception is raised even though the binary is a valid PDF and I have not had any problems using the same functionality to upload other PDFs in the past.

2

You can use :erlang.system_flag/2 to increase the maximum stack trace depth, it might affect overall performance of the system though! Eg.: :erlang.system_flag(:backtrace_depth, 50).

Increasing the depth might point you to the point from where those poison stuff is actually called.

3 Likes
3

Thank you very much! I’ll try that, it is very helpful in general.

I have a suspicion that the problem might be related to the fact that I use a custom module with the ExAws.Request.HttpClient behaviour that still invokes HTTPoison for the ExAws requests, which is the default, but our custom module is set to use different pools in different cases and has increased timeout. I guess some of the configuration for ExAws might not be related to this custom module but to the defaults, I’m not sure though. This is the only explanation I can think of for a Poison, not Jason, exception to be raised.

I’ll write if I find something new.

4

Actually, it doesn’t seem that HTTPoison uses Poison, so the above might be wrong.

5

Unfortunately, the increased stack trace depth results in the same. It begins with:

(poison) lib/poison/encoder.ex:134: Poison.Encoder.BitString.escape/2
(poison) lib/poison/encoder.ex:95: Poison.Encoder.BitString.escape/2
(poison) lib/poison/encoder.ex:136: Poison.Encoder.BitString.escape/2
(poison) lib/poison/encoder.ex:95: Poison.Encoder.BitString.escape/2

With the second two lines repeating on and on.

6

I tested without the custom HTTP client for ExAws and with the default client with the json_codec set to Jason the same exception is raised again.

7

Increase stack depth even further… There has to be an entry point somewhere…

Of course weaare in trouble if it’s a process spawned directly into the potion callchain.

8

I increased it to 150 and decided to try something else. Might try to set it to even more later, I guess.

9

Updating the Phoenix version from 1.3.4 to 1.4.0 resulted in the Poison error being replaced by a Jason one. At least that is now expected from the configuration.

10

Phoenix 1.3.4 used potion internally where today jason is used. jason was introduced in 1.4.0, and also, only since then its an optional dependency.

11

Yes. And I actually managed to find the problem and fix it thanks to Jason. The Jason exception’s stack trace was immensely more helpful and pointed the exception to the controller. It turns out that after decoding the base64 the PDF handling was happening in a separate process started with a Task and the controller was trying to return the PDF binary with a json/2 call which of course could not happen.

It was obvious that the exception was happening in a process separate from the one that’s handling the PDF, I should have thought of that earlier. But ultimately the change to Jason helped.

4 Likes
12

All bugs are obvious in hindsight.

Kudos to you for diligently tracking and fixing this. Well done. :023: :024:

1 Like