josevalim

josevalim

Creator of Elixir

Proposal: ICU/Unicode calendar/datetime formatting

NOTE: this is a focused thread, so we appreciate if everybody stayed on topic. Feel free to comment anything in regards to calendar formatting but avoid off-topic or loosely related topics. For example, if you would like to discuss or propose other Calendar/DateTime features, please use a separate thread.

Hi everyone,

This is a proposal for calendar/datetime formatting in Elixir. The formatting will use the markup specified Unicode’s Locale Date Markup Language.

Here is how the API will look like:

Calendar.format(date_or_time_or_datetime, "yyyy-MM-dd HH:mm:ss.SSSSSS")
#=> {:ok, "2018-11-29 13:19:41.032412"}

The Calendar.format/2 entry point accepts any calendar type, using structural typing. This means we will be able to format any map that has the fields being formatted.

We will also support a third argument, which is a formatter module, that can be used for translation of month names, eras, etc:

Calendar.format(date_or_time_or_datetime, "yyyy-MM-dd HH:mm:ss.SSSSSS", MyApp.PTBR)

Would you also be able to proxy all of this to Gettext or CLDR if you want to:

Calendar.format(date_or_time_or_datetime, "yyyy-MM-dd HH:mm:ss.SSSSSS", Gettext.Calendar)

In other words, the third argument supports translation but not localization. For example, CLDR specifies the short and long formats for dates and times that each region/locale uses but we won’t support those as we believe those can be trivially built on top of Elixir:

defmodule CLDR do
  def long_date(date_or_time_or_datetime) do
    Calendar.format(date_or_time_or_datetime, CLDR.Locale.long_date, CLDR.Locale)
  end
end

Note: this proposal was written by José Valim and Michał Muskała.

The ICU syntax

As described in the ICU page:

A date pattern is a string of characters, where specific strings of characters are replaced with date and time data from a calendar when formatting.

The Date Field Symbol Table below contains the characters used in patterns to show the appropriate formats for a given locale, such as yyyy for the year. Characters may be used multiple times. For example, if y is used for the year, ‘yy’ might produce ‘99’, whereas ‘yyyy’ produces ‘1999’. For most numerical fields, the number of characters specifies the field width. For example, if h is the hour, ‘h’ might produce ‘5’, but ‘hh’ produces ‘05’. For some characters, the count specifies whether an abbreviated or full form should be used, but may have other choices.

Two single quotes represents a literal single quote, either inside or outside single quotes. Text within single quotes is not interpreted in any way (except for two adjacent single quotes). Otherwise all ASCII letter from a to z and A to Z are reserved as syntax characters, and require quoting if they are to represent literal characters.

“Stand Alone” values refer to those designed to stand on their own, as opposed to being with other formatted values. “2nd quarter” would use the stand alone format (QQQQ), whereas “2nd quarter 2007” would use the regular format (qqqq yyyy).

The complete specification can be found here. Elixir will implement a subset of those formats, outlined below.

Format Description Examples Source
G abbreviated_era AD; BC Calendar.year_of_era/1 + Formatter.abbreviated_era/1
GG wide_era Anno Domini; Before Christ Calendar.year_of_era/1 + Formatter.wide_era/1
GGG narrow_era A; B Calendar.year_of_era/1 + Formatter.narrow_era/1
u+ year 2004 struct.year
yy two_digits_year_of_era 4, 14, 14, 14 Calendar.year_of_era/1
y+ year_of_era 4, 14, 214, 2014 Calendar.year_of_era/1
D+ day_of_year 189 Calendar.day_of_year/3
M, MM month 1, 01 struct.month
MMM abbreviated_month Nov struct.month + Formatter.abbreviated_month/1
MMMM wide_month November struct.month + Formatter.wide_month/1
MMMMM narrow_month N struct.month + Formatter.narrow_month/1
d+ day 1, 14, 31 struct.day
Q, QQ quarter 2, 02 Calendar.quarter_of_year/3
QQQ abbreviated_quarter Q2 Calendar.quarter_of_year/3 + Formatter.abbreviated_quarter/1
QQQQ wide_quarter 2nd Quarter Calendar.quarter_of_year/3 + Formatter.wide_quarter/1
QQQQQ narrow_quarter 2 Calendar.quarter_of_year/3 + Formatter.narrow_quarter/1
YY two_digits_week_based_year 4, 14, 14, 14 Calendar.week_in_year/3
Y+ week_based_year 4, 14, 214, 2014 Calendar.week_in_year/3
w+ week_in_year 1, 9, 13, 42 Calendar.week_in_year/3
W+ week_in_month 1, 9, 13, 42 Calendar.week_in_month/3
E abbreviated_day_of_week Tue Calendar.day_of_week/3 + Formatter.abbreviated_day_of_week/1
EE wide_day_of_week Tuesday Calendar.day_of_week/3 + Formatter.wide_day_of_week/1
EEE narrow_day_of_week T Calendar.day_of_week/3 + Formatter.narrow_day_of_week/1
H+ hour (0-23) 1, 01, 23 struct.hour
h+ am_pm_hour (1-12) 1, 01, 11 struct.hour
a am_pm AM, PM struct.hour + Formatter.am_pm/1
m+ minute 1, 11, 59 struct.minute
s+ second 1, 11, 59 struct.second
S+ fraction_of_second 1, 001, 123456 struct.microsecond
VV time_zone Brasil/Sao Paulo struct.time_zone
zz zone_abbr BRT struct.zone_abbr
x zone_offset_basic_optional -08, +0530, +00 struct.std_offset + struct.utc_offset
xx zone_offset_basic -0800, +0530, +0000 struct.std_offset + struct.utc_offset
xxx zone_offset_basic -08:00, +05:30, +00:00 struct.std_offset + struct.utc_offset
X zone_offset_basic_optional_with_z -08, +0530, Z struct.std_offset + struct.utc_offset
XX zone_offset_basic_with_z -0800, +0530, Z struct.std_offset + struct.utc_offset
XXX zone_offset_extended_with_z -08:00, +05:30, Z struct.std_offset + struct.utc_offset

Whenever there is a plus sign at the end, it means the number of entries specifies the minimum number of digits. The exceptions are yy and YY, which is always shown as two digits regardless of how many digits, and S, which truncates.

Besides all entries above, we also support the following stand-alone formats: L for months (same as M) and q for quarters (same as Q).

The source column is used as a reference for the implementation and it won’t be present in the final documentation.

Rationale

Last but not least, it is worth discussing the rationale for date/time formatting. If you have an application that works with calendar types, it is likely that you have to format them at some point. If your application mostly interfaces with other systems, then there is a chance the built-in ISO format is enough, but not always. For example, some HTTP headers use a different format than the recommended ISO one. Therefore adding formatting to the standard library feels like a natural next step to the existing functionality.

Of course, there are some downsides to adding this functionality. First of all, there are many syntaxes for date/time formatting, and they all feel unnatural to some extent. While we could easily argue the ICU/Unicode is the one that makes the most sense for Elixir, as Elixir follows the standard in many other occasions, it is hard to argue it is the best approach generally (or even if there is such thing as best). One approach, which is orthogonal to the one above, is to allow the format to also be given as a list of atoms instead of cryptic single-letter definitions.

Another discussion, which may or may not impact this one, is about parsing. The parsing specification is often the same as the formatting specification but we have explicitly decided to not support parsing in Elixir. First of all, it is really hard to support a general but efficient runtime date/time parsing strategy. If you expect certain formats, it is almost always better to define functions that parse specifically those formats. Things get trickier if we consider the fact we need to support internalization, which is trivial for formatting, but quite expensive for parsing. In other words, while we can provide a general and efficient implementation for formatting, we can’t do so for parsing. Since different trade-offs can be made here, ranging from performance to flexibility, we are not comfortable in picking one or another.

Roadmap

We don’t plan to add this functionality directly to Elixir. Instead we will develop it as a library and collect feedback. The complexity of the implementation will also dictate if this will become part of core or not, but we believe the implementation will be relatively simple.

The first step is validation of this proposal and then a library will be developed as part of The Elixir programming language · GitHub for futher validation and feedback.

Feedback

Your turn.

Most Liked

kip

kip

ex_cldr Core Team

The full table of supported codes for ICU is in the reference you supplied, more specifically at Unicode Locale Data Markup Language (LDML) Part 4: Dates

E, EE and EEE are week day formats like “Tuesday, Tue, T”
MMM, MMMM are month names like “September, Sept”

There are others that are primarily textual and therefore locale specific like a, b, B that I recognise are outside of the scope of this proposal.

I’m just trying to work out if they remain valid formatting codes for this proposal, but not supported by the default formatter. Or if they are considered invalid codes for this proposal.

On the locales front - yes, your API strategy makes a lot more sense. The core of what i was trying to express, badly, is that even when a package or a language doesn’t support multiple locales, it implicitly is supporting one which for Elixir is en-US. Today, each package that intends to help support multiple locales like Gettext, Cldr, Trans has its own notion of current locale and what constitutes a supported locale, or locale name format. A unifying strategy for current locale and locale name would be helpful. But outside the scope of this proposal of course.

josevalim

josevalim

Creator of Elixir

Thanks everyone for the feedback!

I am bringing the proposal down for now because the discussion has made it clear that, if we want to support this, then we should go ahead and fully support ICU (with all formats, variants and what not) or we don’t support it all.

After all, if we only support part of it, then the community will have to implement everything again, like in @kip’s great projects, or we will have to define many interfaces so libraries are able to continue the work we started. None of those options feel great and then I would rather make sure that the Calendar functionality provides everything @kip needs in his libs.

So here are the next steps: we will look into simpler formatting syntaxes, possibly strftime, and see if we can write a proposal with reduced scope and less pitfalls that provides a minimum set of functionality without overlapping with more complex use cases. In case we can’t find anything that fits those constraints then we will give up on having formatting in core for now.

josevalim

josevalim

Creator of Elixir

Those are very good questions. We generally want Elixir core to be lean. We want the community to rely on third party packages. In fact, I have talked with @bitwalker about breaking Timex apart (for example, parsing/formatting could be its own lib).

The only possible rationale for including this in Elixir is because it belongs to the current feature set already part of the language and that this feature is small enough to warrant its inclusion in the language but I agree it is a blurry line.

To avoid any debate, I will just follow whatever Unicode and/or CLDR recommends as default. You will be able to customize by passing a third argument to format.

I haven’t thought about this yet and to be honest, I don’t think so. A timezone database is meant to provide the same results. What would change between them is functionality such as live updates, performance, etc. So while it is global, it is not mean to cause a difference in behaviour.

Setting a different formatter will change behaviour. So for now I would keep it explicitly opt-in.

Where Next?

Popular in News Top

Elixir
1. Enhancements Elixir [Kernel] Raise when U+2028 and U+2029 characters are present in comments and strings to avoid line spoofing attac...
New
josevalim
It is a maintenance release, so nothing out of the ordinary. Please see the release notes for more information: Release v1.5.3 · elixir-...
New
josevalim
NOTE: this is a focused thread, so we appreciate if everybody stayed on topic. Feel free to comment anything in regards to calendar forma...
New
Elixir
Official announcement: Elixir v1.16 released - The Elixir programming language 1. Enhancements EEx [EEx] Include relative file informat...
New
josevalim
Hi! :wave: I hope everyone is well! The Elixir team releases new versions every 6 months, typically every January and July. However, si...
New
Elixir
Release: Release v1.11.1 · elixir-lang/elixir · GitHub 1. Bug fixes Elixir [Code] Ignore tracers if lexical tracker is dead or explicit...
New
Elixir
Code snippets in diagnostics Elixir v1.15 introduced a new compiler diagnostic format and the ability to print multiple error diagnostics...
New
josevalim
Official announcement: Elixir v1.4 released - The Elixir programming language
New
New
Elixir
1. Enhancements Elixir [Duration] Add Duration.to_iso8601/1 and Duration.from_iso8601/1 [Keyword] Add Keyword.intersect/2-3 to mirror th...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42920 311
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New

We're in Beta

About us Mission Statement