Why does String.replace output come in <<>> form when result contains non-ascii characters?

Questions / Help
1

How do we explain this?

josie@josie-Inspiron-580:~/exercism/elixir/parallel-letter-frequency$ iex
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
 
Interactive Elixir (1.6.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> String.valid?("Wo dein sanfter Flügel weilt")
true
iex(2)> String.replace("Wo dein sanfter Flügel weilt", ~r/\P{L}/, "")
<<87, 111, 100, 101, 105, 110, 115, 97, 110, 102, 116, 101, 114, 70, 108, 195,
  103, 101, 108, 119, 101, 105, 108, 116>>
iex(3)> String.replace("Wo dein sanfter Flügel weilt", ~r/\P{L}/, "") |> String.valid?
false
iex(4)> <<87, 111, 100>> |> String.valid?
true
iex(5)>
2

You haven’t specified the u option to your regex, therefore the regex considers the input beeing latin1 (if I recall correctly).

This means, the UTF8 encoded "üg" (encoded as <<195, 188, 103>>) is seen as üg.

¼ is not a letter obviously, so <<195, 103>> remains in the binary, which is not valid UTF8 anymore.

iex(1)> String.replace("üg", ~r/\PL/, "")
<<195, 103>>
iex(2)> String.replace("üg", ~r/\PL/u, "")
"üg"

This is because the underlying erlang module re uses latin1 as default encoding.

8 Likes