Terbium-135

Terbium-135

:erlang.term_to_binary and postgresql => invalid byte sequence for encoding "UTF8"

After reading this:

I tried to use term_to_binary together with postgresql:

Migration:

[...]
      add :owner, :text
      add :version, :text
      add :state, :bytea
[...]

Schema:

[...]
     field :state, :binary
[...]

Code:

update_or_insert(%State{owner: owner, state: :erlang.term_to_binary(state)})

  def update_or_insert(state) do
    state
    |> Repo.insert(
      on_conflict: {:replace_all_except, [:id, :owner, :created_at]},
      conflict_target: [:owner],
      set: [updated_at: DateTime.utc_now()]
    )
  end

Error:

07:11:53.186 [error] GenServer #PID<0.15770.0> terminating
** (Postgrex.Error) ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding "UTF8": 0x83

How to solve this?
Change encoding for the resulting binary?
Leave term_to_binary alone and use JSON encoding/decoding?

I was also looking at the docs for erlang term_to_binary but there are only compression levels mentioned

Marked As Solved

Terbium-135

Terbium-135

Thanks for chiming in @jhogberg and @kip!

This was interesting to solve.

The DB is set to UTF8:

While I was trying to narrow down the problem I got a different error message:

09:44:01.258 [error] GenServer #PID<0.7186.0> terminating
** (Postgrex.Error) ERROR 22023 (invalid_parameter_value) payload string too long
    (ecto_sql 3.11.0) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1

Which leads to the real problem
The table has a trigger attached - and that trigger is converting the updated/inserted table row to JSON to send a message using pg_notify => trying to convert the binary data…

And I was able to verify it as soon as I had a look at the postgresql log files

Never thought of this :crazy_face:

Also Liked

kip

kip

ex_cldr Core Team

My guess is that its resulting from an encoding error in either the owner or version fields since the Postgres text type maps to Elixir String.t type - which is UTF-8. Therefore one possible cause is that that either the Postgres database or the database connection string are set to LATIN-9 or some other non-UTF-8 encoding.

The bytea type maps to Elixir/erlang binary as you have configured.

jhogberg

jhogberg

Erlang Core Team

term_to_binary/1,2 produces a binary in the external term format, not a UTF-8-encoded binary, so this seems like a problem with the schema (by the looks of it on the database server side). Try looking into why field :state, :binary implies UTF-8 encoding.

Where Next?

Popular in Questions Top

vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 131117 1222
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 55125 245
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New

We're in Beta

About us Mission Statement