How do you POST a byte array inside a form field to a Phoenix endpoint via `curl` or such?

Phoenix Forum Questions / Help
, , ,
1

So I have this callback endpoint where I accept a delayed status update for jobs that I tasked a remote server with. I have made a table that only has ID, data field (jsonb type) and inserted_at and just store the response with zero processing. Here’s what error I get upon doing Repo.insert:

** (Postgrex.Error) ERROR 22P05 (untranslatable_character) unsupported Unicode escape sequence
\u0000 cannot be converted to text.

Reason is that the inspect-ed POST form parameters look similar to this (redacted for privacy):

%{
  "stuff" => "246",
  "time" => "1728063410",
  "inqueue" => "2",
  "someid" => "26",
  "outqueue" => "6",
  "binarystuff" => <<113, 0>>
}

I believe the last value trips it up. I wrote a test to reproduce it but that’s still just using Elixir code to POST the payload.

Two questions:

  • How do you even curl such a payload to an HTTP endpoint? Namely with one form parameter containing a byte array.
  • How would you serialize a byte array inside a jsonb field?
2

I haven’t futzed with the Elixir side of this, but on the PostgreSQL side I’d either use:

encode (PostgreSQL: Documentation: 17: 9.5. Binary String Functions and Operators) or (in a pinch) something along the lines of SET bytea_output = 'escape'; in the transaction.

Not sure that helps at all in this case, but that’s the gist.

1 Like
3

FWIW, there appears to be a hard limitation on PG’s end regarding \u0000 values in strings inside jsonb:

Usual practice for this situation would be to base64 the value somewhere in the dataflow; doing it at the origin might be easiest since then you’re always POSTing “normal” characters.

2 Likes
4

Well, there are two options:

  1. Sanitize the input and change f.ex. <<1, 2>> to its inspect-ed form i.e. ""<<1, 2>>". That introduces a specially-treated value you have to look for and I hate it because it’s easy to forget to do so.
  2. Change the table column from jsonb to json because that one allows \u0000’.

@al2o3cr Thanks, I found the same, also this: How to insert JSON containing escape codes into a JSONB column in PostgreSQL using GORM - Stack Overflow

So I opted for solution #2 and it all worked immediately.

Oh, and as for curl:

curl -v --data 'field=%00' -XPOST 'http://wherever.com'

Got super ambitious because some of my skills proved a bit dusty but quickly refreshed them.

5

Maybe, but I have no clue how to specially treat base64… I can put a prefix and treat the value specially but this is just error-prone, especially if you are scrambling to cancel and then retry an Oban job in a production k8s pod. :expressionless:

It’s a project from another person in the company and I am reasonably sure he will not change a thing to pander to my fragile little HTTP callbacks. :003:

But yeah, just switching from jsonb to json worked.