scouten
Converting a list of bytes from UTF-8 or ISO-8859-1 to Elixir string?
I have a list of bytes that might be in UTF-8 or might be in ISO-8859-1 or might be in some other code format. In some cases, I have a hint as to the encoding I’m decoding, but it’s not guaranteed. (I am not in control of the format I’m parsing … don’t judge, please!)
How would I design a function to decode this myriad of possibilities? Are there built-in libraries or third-party libraries that I should be examining?
I’d want to try decoding as UTF-8 first; if that fails, then follow the hint that I may or may not have (and if no hint, then fall back to ISO-8859-1).
Marked As Solved
bjorng
You can use the unicode module in OTP to test for valid UTF-8 encoding.
Here is an example:
iex(1)> latin1_list = [66,106,246,114,110]
[66, 106, 246, 114, 110]
iex(2)> utf8_list = [66,106,195,182,114,110]
[66, 106, 195, 182, 114, 110]
iex(3)> :unicode.characters_to_binary(:erlang.list_to_binary(latin1_list))
{:error, "Bj", <<246, 114, 110>>}
iex(4)> :unicode.characters_to_binary(:erlang.list_to_binary(utf8_list))
"Björn"
iex(5)>
The unicode:characters_to_binary/1 function accepts either a list or a binary as input. If the input is a list, it expects that each element in the list is a single Unicode codepoint. Since your input list is a list of bytes, it is not appropriate to use the input list directly. Instead, it must be converted to a binary. When given a binary, unicode:characters_to_binary/1 expects that the characters in the binary are encoded in UTF-8. If the binary is indeed encoded in UTF-8, the return value will be the same binary. If it is not properly encoded in UTF-8, an error tuple will be returned instead.
If the list was not encoded in UTF-8, here is how to convert it to a UTF-8 encoded binary, using unicode:characters_to_binary/2:
iex(5)> :unicode.characters_to_binary(:erlang.list_to_binary(latin1_list), :latin1)
"Björn"
Also Liked
bjorng
That blog post was written in 2015. At the time, the suggested solution might have been the best way to test for UTF-8 encoding.
Today, it’s easier to use the unicode module or binary matching:
def is_valid_utf8?(<<_ :: utf8, rest :: binary>>), do: is_valid_utf8?(rest)
def is_valid_utf8?(<<>>), do: :true
def is_valid_utf8?(<<_ :: binary>>), do: :false
(Using the unicode module as shown in my previous post is probably faster, but will build more garbage.)
ConnorRigby
Hey I’m all about using language built-ins. I learned something.
scouten
Thank you all. This implementation (slightly tuned from Björn’s answer) seems to do what I need it to do:
def decode(b) when is_list(b) do
raw = :erlang.list_to_binary(b)
case :unicode.characters_to_binary(raw) do
utf8 when is_binary(utf8) -> utf8
_ -> :unicode.characters_to_binary(raw, :latin1)
end
end
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"