Converting a list of bytes from UTF-8 or ISO-8859-1 to Elixir string?

I think Latin-1 is a subset of UTF-8, so there’s no way to differentiate UTF-8 from a bunch of Latin-1 characters.
Here is a nice decoder i found from NineNines that i converted to Elixir: Nine Nines: Validating UTF-8 binaries with Erlang

  # This function returns 0 on success, 1 on error, and 2..8 on incomplete data.
  def validate_utf8(<<>>, state), do: state
  def validate_utf8(<< c, rest :: bits >>, 0) when c < 128, do: validate_utf8(rest, 0)
  def validate_utf8(<< c, rest :: bits >>, 2) when c >= 128 when c < 144, do: validate_utf8(rest, 0)
  def validate_utf8(<< c, rest :: bits >>, 3) when c >= 128 when c < 144, do: validate_utf8(rest, 2)
  def validate_utf8(<< c, rest :: bits >>, 5) when c >= 128 when c < 144, do: validate_utf8(rest, 2)
  def validate_utf8(<< c, rest :: bits >>, 7) when c >= 128 when c < 144, do: validate_utf8(rest, 3)
  def validate_utf8(<< c, rest :: bits >>, 8) when c >= 128 when c < 144, do: validate_utf8(rest, 3)
  def validate_utf8(<< c, rest :: bits >>, 2) when c >= 144 when c < 160, do: validate_utf8(rest, 0)
  def validate_utf8(<< c, rest :: bits >>, 3) when c >= 144 when c < 160, do: validate_utf8(rest, 2)
  def validate_utf8(<< c, rest :: bits >>, 5) when c >= 144 when c < 160, do: validate_utf8(rest, 2)
  def validate_utf8(<< c, rest :: bits >>, 6) when c >= 144 when c < 160, do: validate_utf8(rest, 3)
  def validate_utf8(<< c, rest :: bits >>, 7) when c >= 144 when c < 160, do: validate_utf8(rest, 3)
  def validate_utf8(<< c, rest :: bits >>, 2) when c >= 160 when c < 192, do: validate_utf8(rest, 0)
  def validate_utf8(<< c, rest :: bits >>, 3) when c >= 160 when c < 192, do: validate_utf8(rest, 2)
  def validate_utf8(<< c, rest :: bits >>, 4) when c >= 160 when c < 192, do: validate_utf8(rest, 2)
  def validate_utf8(<< c, rest :: bits >>, 6) when c >= 160 when c < 192, do: validate_utf8(rest, 3)
  def validate_utf8(<< c, rest :: bits >>, 7) when c >= 160 when c < 192, do: validate_utf8(rest, 3)
  def validate_utf8(<< c, rest :: bits >>, 0) when c >= 194 when c < 224, do: validate_utf8(rest, 2)
  def validate_utf8(<< 224, rest :: bits >>, 0), do: validate_utf8(rest, 4)
  def validate_utf8(<< c, rest :: bits >>, 0) when c >= 225 when c < 237, do: validate_utf8(rest, 3)
  def validate_utf8(<< 237, rest :: bits >>, 0), do: validate_utf8(rest, 5)

  def validate_utf8(<< c, rest :: bits >>, 0) 
    when c === 238 
    when c === 239,
  do: validate_utf8(rest, 3)
  
  def validate_utf8(<< 240, rest :: bits >>, 0), do: validate_utf8(rest, 6)

  def validate_utf8(<< c, rest :: bits >>, 0) 
    when c === 241 
    when c === 242 
    when c === 243,
  do: validate_utf8(rest, 7);

  def validate_utf8(<< 244, rest :: bits >>, 0), do: validate_utf8(rest, 8)
  def validate_utf8(_, _), do: 1

so i think you can use that to determine if your string is valid utf8 (0), if its not valid utf8 (1) its not either of those (2..8) its neither

You can use the unicode module in OTP to test for valid UTF-8 encoding.

Here is an example:

iex(1)> latin1_list = [66,106,246,114,110]
[66, 106, 246, 114, 110]
iex(2)> utf8_list = [66,106,195,182,114,110]
[66, 106, 195, 182, 114, 110]
iex(3)> :unicode.characters_to_binary(:erlang.list_to_binary(latin1_list))
{:error, "Bj", <<246, 114, 110>>}
iex(4)> :unicode.characters_to_binary(:erlang.list_to_binary(utf8_list))
"Björn"
iex(5)>

The unicode:characters_to_binary/1 function accepts either a list or a binary as input. If the input is a list, it expects that each element in the list is a single Unicode codepoint. Since your input list is a list of bytes, it is not appropriate to use the input list directly. Instead, it must be converted to a binary. When given a binary, unicode:characters_to_binary/1 expects that the characters in the binary are encoded in UTF-8. If the binary is indeed encoded in UTF-8, the return value will be the same binary. If it is not properly encoded in UTF-8, an error tuple will be returned instead.

If the list was not encoded in UTF-8, here is how to convert it to a UTF-8 encoded binary, using unicode:characters_to_binary/2:

iex(5)> :unicode.characters_to_binary(:erlang.list_to_binary(latin1_list), :latin1)
"Björn"

It is impossible to infer the enccoding just from the bytes.

<<64, 65, 66>> is probably valid in any 8bit encoding that ever existed, you will never know if this is meant to be ASCII, latin, or EBCDIC without someone telling you! If its ASCII or latin1, you can both treat the same, but if its EBCDIC you are lost.

That blog post was written in 2015. At the time, the suggested solution might have been the best way to test for UTF-8 encoding.

Today, it’s easier to use the unicode module or binary matching:

  def is_valid_utf8?(<<_ :: utf8, rest :: binary>>), do: is_valid_utf8?(rest)
  def is_valid_utf8?(<<>>), do: :true
  def is_valid_utf8?(<<_ :: binary>>), do: :false

(Using the unicode module as shown in my previous post is probably faster, but will build more garbage.)

Hey I’m all about using language built-ins. I learned something.

Thank you all. This implementation (slightly tuned from Björn’s answer) seems to do what I need it to do:

def decode(b) when is_list(b) do
  raw = :erlang.list_to_binary(b)

  case :unicode.characters_to_binary(raw) do
    utf8 when is_binary(utf8) -> utf8
    _ -> :unicode.characters_to_binary(raw, :latin1)
  end
end

Latin-1 is not a subset of UTF-8. Only the ASCII parts match (0–127), the higher values do not.

Well, latin1 is a subset of UTF8 in the sense that every character that you can encode in latin1 is also encodable in UTF8.

That would make pretty much every charset a subset of UTF-8 since UTF-8 encodes Unicode and that includes all characters in the world.

Yupp, all I wanted to say is, that one needs to be careful when saying “is a subset of” or is not…