dogweather

dogweather

Possible bug in String.split/1's handling of Unicode whitespace?

According to the docs, String.split/1

Divides a string into substrings at each Unicode whitespace occurrence…

(emphasis mine)

But I found that it doesn’t do this for U+00A0 “NO-BREAK SPACE” (UTF-8 <<194, 160>>). Maybe others too? No-break space is unicode whitespace and it’s marked as such in the Elixir Unicode property list.

Here’s the unexpected behavior I see (Elixir 1.14):

iex(1)> nbsp = <<194, 160>>
" "
iex(2)> s = Enum.join ["one", "two", "three"], nbsp
"one two three"
iex(3)> String.split s
["one two three"]

I expected it to split on the nbsp, producing the input, ["one", "two", "three"].

I traced the code for String.split/1 to break() here: elixir/lib/elixir/unicode/unicode.ex at main · elixir-lang/elixir · GitHub. It looks ok to me, though.

Background: This caused problems for me parsing some text. I didn’t realize that this is valid Unicode because String.split/1 and Regex.run where failing on it (although for two different reasons).

Marked As Solved

brettbeatty

brettbeatty

The String.split/1 docs also have this line:

Divisions do not occur on non-breaking whitespace.

Also Liked

kip

kip

ex_cldr Core Team

The ‘break’ property and the ‘whitespace’ property are orthogonal in Unicode. Definitely significant overlap when looked at from the perspective of Latin scripts but definitely not universal - especially for languages that do not use whitespace between words (like Japanese, Chinese, Thai, Lao, Khmer, Myanmar scripts).

The elixir default for String.split/1 is “breaking whitespace”. By definition, “non breaking” characters should not break words, lines or sentences.

dogweather

dogweather

Urp! Thank you, I didn’t catch that.

Where Next?

Popular in Questions Top

nobody
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New

Other popular topics Top

hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement