Terbium-135
After reading this:
I tried to use term_to_binary together with postgresql:
Migration:
[...]
add :owner, :text
add :version, :text
add :state, :bytea
[...]
Schema:
[...]
field :state, :binary
[...]
Code:
update_or_insert(%State{owner: owner, state: :erlang.term_to_binary(state)})
def update_or_insert(state) do
state
|> Repo.insert(
on_conflict: {:replace_all_except, [:id, :owner, :created_at]},
conflict_target: [:owner],
set: [updated_at: DateTime.utc_now()]
)
end
Error:
07:11:53.186 [error] GenServer #PID<0.15770.0> terminating
** (Postgrex.Error) ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding "UTF8": 0x83
How to solve this?
Change encoding for the resulting binary?
Leave term_to_binary alone and use JSON encoding/decoding?
I was also looking at the docs for erlang term_to_binary but there are only compression levels mentioned
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
**Building the future of software with AI, trust, and human connection.**
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 5- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
jhogberg
term_to_binary/1,2produces a binary in the external term format, not a UTF-8-encoded binary, so this seems like a problem with the schema (by the looks of it on the database server side). Try looking into whyfield :state, :binaryimplies UTF-8 encoding.kip
My guess is that its resulting from an encoding error in either the
ownerorversionfields since the Postgrestexttype maps to ElixirString.ttype - which is UTF-8. Therefore one possible cause is that that either the Postgres database or the database connection string are set to LATIN-9 or some other non-UTF-8 encoding.The
byteatype maps to Elixir/erlangbinaryas you have configured.rvirding
I still mainly do Erlang courses but I play with Elixir as well.
rvirding
Yes, the ETF is an internal format and is definitely not UTF-8 encoded.
Terbium-135
Thanks for chiming in @jhogberg and @kip!
This was interesting to solve.
The DB is set to UTF8:
While I was trying to narrow down the problem I got a different error message:
Which leads to the real problem
The table has a trigger attached - and that trigger is converting the updated/inserted table row to JSON to send a message using
pg_notify=> trying to convert the binary data…And I was able to verify it as soon as I had a look at the postgresql log files
Never thought of this