kip

kip

ex_cldr Core Team

i would challenge the “unicode is not usually needed” assumption. Unless you have complete confidence in your knowledge of the cultural context of your audience, Latin-1 is a poor assumption. And in an Elixir context, the String.t type is unicode so its not surprising that developers are caught out by the default behaviour of Regex being Latin-1 (even though I understand why the choice, I think its hard to justify as the default).

Latin-1 doesn’t even include the Euro symbol. @michalmuskala’s native language can’t be represented in Latin-1 (Polish is Latin-2). And given the global participation of this forum I’d warrant there are many members building applications for a global audience and therefore should not make limited assumptions on character input or output.

I suppose I may be an outlier on this (Australian, living in Singapore, and having worked in several Asian countries for quite a while). It even bugs me that we don’t consistently address José with the name his mother gave him :slight_smile:

Elixir is a Unicode-centric language. I think that the :re option "u" should be the default.

5
Post #11
michalmuskala

michalmuskala

I will agree that the elixir regexes should include the u flag by default and require explicit opt-out. We generally should prefer correctness over speed with ways to opt-in into faster, but potentially incorrect results.

Unfortunately, the soonest we could be able to change this would be 2.0 (and there are no plans for 2.0 at the moment). This is because that would be a breaking change since many functions would start returning different results.

7
Post #12
kip

kip

ex_cldr Core Team

Oh I never had any expectation on a breaking change - its just one of those things that, in hindsight, would probably have been a better default. With all the work on globalisation stuff I’m doing I still forget this flag. This conversation even reminded me of one place I hadn’t set it!

0
Post #13
tallakt

tallakt

The sooner its done the better

0
Post #14
mudasobwa

mudasobwa

Creator of Cure

You gotta be kidding. Ruby failed to convert case with String#upcase / String#downcase in stdlib till 2018 and simple unicode normalization with String#unicode_normalize happen to appear in 2.5 only.

Ruby literally had the worst unicode support among many languages till a while ago (maybe only python had the worse.)

1
Post #15
abtrapp

abtrapp OP

Please re-read the posts. I never said that the utf-8 handling in Ruby was perfect (has it’s quirks like in every language I know). Upcase/downcase was not supported because it is locale dependend, … and 3rd party gems did what Ruby did not - but ruby has useful default behavior, if something does not make sense in practice over time the default behavior has been changed, … - that’s what I like about Ruby (contrast: Java, where things are “deprecated” for decades), I really don’t want to discuss unicode support in Ruby or start a language war in this forum :slight_smile:

This topic should be about how to solve a problem (my initial posting) and maybe (the discussion that started) discussing how to make Elixir easier to work with for most people (at least I hope so). Elixir should be “fun”, just like Ruby - digging into Erlang is not fun for most users when dealing with problem that they get in > 80% of the projects.

I have a lot of respect for people who write languages/libraries/frameworks because it’s always easy to point out some errors/improvements, but it’s hard to get something done right from scratch. So, when I’m new to a language (like Elixir) I try to help by providing some thoughts as a newbie to that language to help to make the transition easier for other people that are following - because after a short time using that language I don’t even think about this stuff any more.

If my english has not been good enough to bring that message across, please accept my apology. And if you can point me to the misleading text that lead to the “I hate Ruby” bashing post I will try to rewrite it.

1
Post #16
mudasobwa

mudasobwa

Creator of Cure

I honestly do not see how the documented default behaviour that somehow differs from your expectations might make digging into language less fun.

Also, I am biased because I literally hate “convention over configuration” paradigm and I am positive Rails killed the great language because at some point it become too easy to jump into. I constantly ask the question “how foo(&:bar) syntax works in ruby in a nutshell” and I constantly receive a vast majority of answers saying “it’s a syntactic sugar.“

1
Post #17
abtrapp

abtrapp OP

Let’s say 2019 ~ 80-90% working with UTF-8. (no current numbers here, but I’m pretty sure that we are in that region):

  1. If I call a function that supports UTF-8 on an UTF-8 string I assume it is working with that UTF-8 string and delivers the correct behavior. However it does not to that, you need a special parameter/option to make the function call return the correct result. Especially when switching from an OO language (duck typing: that’s an UTF-8 string, it should handle it like that).
  2. If 80-90% of all modern web applications are dealing with UTF-8 (not ASCII/Latin) then something like this should be the default (imho).
  3. If everything should be explicit, there should be a mandatory parameter (this is a Latin string / this is UTF8)… which I would find very ugly
    Nothing that changes the world, but it just would make the entry and daily usage easier.

Everybody is biased :slight_smile: - I worked with Java when it had stubs and skels. Learned to love “convention over configuration” and DRY and started to have fun with my work after copy/pasting everything 10 times in Java :laughing: - wrote a MDSD template generator back then that saved people about 1-2 weeks of typing because they had everything configured (some stuff multiple times)

Imho Rails was that success because concepts like DRY and “convention over configuration” made the entry easy (and imho the entry can never be too easy) - but I agree that every decision has two sides. When it became so easy that people didn’t even think about what that “database thing” was doing it lead to a log of nice performance issues that I have in 9/10 projects where I have to get “startup code” to a more performant/stable state that is maintainenable.

However: all of the Rails programmers that I know that look into Elixir are looking because 1 reason only: performance - which I find a little bit sad because I really like some Elixir concepts and I think it makes you a better Rails programmer too. (Functional core / imperative shell as example) If somebody came from Java to Rails Elixir seems like a litte step back (configuration over convention, repeat yourself, the module a.b.c stuff can remind people of Java packages, …) - and things like Erlang naming convention, default values that made sense when Erlang was written, … do feel like I’m back in the Java world sometimes :smiley: - but that’s just my opinion.

1
Post #18
mudasobwa

mudasobwa

Creator of Cure

Eh. DRY in Elixir is way easier than in Ruby FWIW.

Eh. How ActiveRecord::Base is less reminding of fully qualified names than ActiveRecord.Base?

I never considered myself Rails programmer (I even have an explicit tagline “I like Ruby off the Rails” everywhere in the internets,) but I obviously have some experience with Rails. I can tell that I admire Elixir for its conciseness, clarity, expressiveness, resilience, fault-tolerance, and (the biggest one) AST on hand.

I am able to cook Ruby to be performant enough to suit my needs, so performance is a great bonus, but it definitely not what made me to switch.

2
Post #19
Nicd

Nicd

Technically all strings in Elixir are UTF-8, there is no way to handle other character sets in the String and Regex modules (disregarding Erlang functions and charlists).

2
Post #20