Sorting using locales

@LostKobrakai said it very well. The only thing for me to add is that I have been back researching implementation designs for a native Elixir collation implementation. It’s not a small effort to get right (space and time efficient). The Unicode Collation Algorithm is comprehensive, and has some tuning in the CLDR Collation implementation. I’m optimistic I’ll have something to ship this year but I can’t commit to that yet. Collaboration and contribution would be very warmly welcomed.

Many thanks for Your reply! Yes, it makes perfect sense.

There is no right or wrong here given a lack of context. Collations are locale specific defaults sort orders so given a lack of knowing the locale there’s no way to select a locale specific default sort order. One and the same list can be sorted differently depending on which local you sort for.

WITH test_data AS (
	SELECT val FROM (VALUES ('a'),('b'), ('c'), ('ch'), ('d'), ('e'), ('f'), ('g'), ('h'), ('i'), ('j')) AS t (val)
)
SELECT val FROM test_data ORDER BY val COLLATE "de_DE";
# a, b, c, ch, d, e, f, g, h, i, j

WITH test_data AS (
	SELECT val FROM (VALUES ('a'),('b'), ('c'), ('ch'), ('d'), ('e'), ('f'), ('g'), ('h'), ('i'), ('j')) AS t (val)
)
SELECT val FROM test_data ORDER BY val COLLATE "sk_SK";
# a, b, c, d, e, f, g, h, ch, i, j

As for the reasons why this is not part of elixir – same reason why elixir doesn’t ship a (full) timezone database – it doesn’t want to be dependant on the release cycles of the source for information needed, as well as needing to choose a source in the first place. Lots of elixirs TZ packages use IANA as their source, but IANA has different update cycles than elixir. They usually update a handful of times or more per year. Elixir does only twice. Instead Elixir exposes a config for users to provide a time zone database via the Calendar.TimeZoneDatabase behaviour. This also allows for packages to e.g. integrate with the tz handling of the underlying host OS, which is iirc what nerves does to not duplicate sources.

For sorting such an interface exists as well Enum.sort can take a module exposing a :compare/2 function as argument, which can define sorting order. There’s no behaviour, given the interface is so simple, but it’s documented here: Enum — Elixir v1.20.2

So it’s up to the community to build libraries, which allow sorting in a locale specific manner. ex_cldr_collattion is a first step in integrating with icu4c, but it’s limited atm:

This initial version uses only the “root” locale collator which is the CLDR DUCET collator.

Why hasn’t more happened yet? Probably because (as I did above) sorting often happens at the db level less so than at the application level and therefore the need might not be that large. I’m sure @kip would appreciate some help integrating some more locales though.

Yes, I know, that’s why I use Elixir! But I don’t understand why it isn’t part of Elixir and it’s not endorsed officially.

[“Łódź”, “Warszawa”, “Zegrze”] |> Enum.sort

Hey. The way I do it is using the CLDR Collation lib (all CLDR libs are pretty good! Thanks @kip!)

Could you give an example?

It’s eight years later and by default Elixir still sorts wrong. I think the expected behaviour would be to apply collation rules of the system locale. Does it mean it’s going to stay like this?

I have found the Linguist library but it does not seem to address collation And does not seem to be actively maintained anymore.

How strange! I see there is a i18n module in Erlang so maybe that’s something that can be done. Anyway, I think it’s a huge showstopper.