dogweather
Possible bug in String.split/1's handling of Unicode whitespace?
According to the docs, String.split/1…
Divides a string into substrings at each Unicode whitespace occurrence…
(emphasis mine)
But I found that it doesn’t do this for U+00A0 “NO-BREAK SPACE” (UTF-8 <<194, 160>>). Maybe others too? No-break space is unicode whitespace and it’s marked as such in the Elixir Unicode property list.
Here’s the unexpected behavior I see (Elixir 1.14):
iex(1)> nbsp = <<194, 160>>
" "
iex(2)> s = Enum.join ["one", "two", "three"], nbsp
"one two three"
iex(3)> String.split s
["one two three"]
I expected it to split on the nbsp, producing the input, ["one", "two", "three"].
I traced the code for String.split/1 to break() here: elixir/lib/elixir/unicode/unicode.ex at main · elixir-lang/elixir · GitHub. It looks ok to me, though.
Background: This caused problems for me parsing some text. I didn’t realize that this is valid Unicode because String.split/1 and Regex.run where failing on it (although for two different reasons).
Marked As Solved
brettbeatty
Also Liked
kip
The ‘break’ property and the ‘whitespace’ property are orthogonal in Unicode. Definitely significant overlap when looked at from the perspective of Latin scripts but definitely not universal - especially for languages that do not use whitespace between words (like Japanese, Chinese, Thai, Lao, Khmer, Myanmar scripts).
The elixir default for String.split/1 is “breaking whitespace”. By definition, “non breaking” characters should not break words, lines or sentences.
dogweather
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance










