Calendars & Calendar Conversions

[alisinabh]

But instead of Date or DateTime everyone should change to something like Calendar.ISO.Date or Calendar.ISO.DateTime. Am i wrong?

[OvermindDL1]

Or alias it in of course. ^.^

[alisinabh]

That was what i meant when i said getting hands dirty

[OvermindDL1]

@alisinabh

That was what i meant when i said getting hands dirty

Heh. One nice side-effect, it clearly tells you what format your Date/DateTime is in. ^.^

[alisinabh]

@OvermindDL1

Heh. One nice side-effect, it clearly tells you what format your Date/DateTime is in. ^.^

I like it too, and since Elixir is a newborn language, maybe this change wont get anybody mad because it is really useful and really Elixir like. It is a nice feeling if Elixir handles it in such a way that there will be no calendar that doesn’t fit into its core structure.
I really like your idea.

[michalmuskala]

From a developers perspective:

We already had a lot of different Date/Time structures. Each library had it’s own - timex had it’s own, calendar had it’s own, ecto had it’s own, different calendar implementations had it’s own… And it was a horrorshow. The usability of those solutions was extremely poor. If you needed to do anything with dates, you found yourself in an endless pipeline of conversions from one format to another. Please, please, please let’s not go back.

[Qqwy]

@michalmuskala

From a developers perspective:

We already had a lot of different Date/Time structures. Each library had it’s own - timex had it’s own, calendar had it’s own, ecto had it’s own, different calendar implementations had it’s own… And it was a horrorshow. The usability of those solutions was extremely poor. If you needed to do anything with dates, you found yourself in an endless pipeline of conversions from one format to another. Please, please, please let’s not go back.

Well spoken.

I think it is important to realize that there are two separate problems that Elixir Core attempts to solve:

1. Ensure that different date/time libraries can communicate, by specifying a single underlying data type.
2. Ensure that different calendars can be represented and calculated with; that Elixir is not tied to a single calendar.

The first problem was solved by introducing the Date, Time, DateTime and NaiveDateTime structs. Since Timex, Calendar and Ecto all only worked with the ISO 8601 proleptic Gregorian calendar, it was very natural at that time to include struct fields that matched that calendar.

I think the choice of splitting off the NaiveDateTime was a very smart one that follows the explicit nature of the language.

I also want to praise the Elixir Core team that they had the foresight at that time to think about (2.), included a :calendar field in these structs and alter the way the structs are represented by dispatching to the Calendar behaviour. !

This was what has left room open for the work @kip is doing right now: creating a very nice way of converting between different calendars by using the Rata Die as intermediate representation. This will allow:

• Comparing between two possibly different calendars, even if they do not follow the :year, :month, :day, :hours, :minutes, :seconds fields in the way that Calendar.ISO does.
• Obtaining an interval in {days, fraction_of_day} between two dates in possibly different calendars.
• Adding a {days, fraction_of_day} interval to a date in any calendar.
• Enumerating over an interval between two timestamps (in days, and possibly also any other subdivision or superdivision days)

So: I am really happy with the changes that have been introduced in 1.3, and I am also very happy with the new changes that are being worked on!

[Qqwy]

What would be a really nice addition in the future by the way, is to add a %Period{} struct (Or maybe a %Date.Period{}, %TIme.Period{}, %DateTime.Period{} and %NaiveDateTime.Period{}?) whose fields have the same names as those of the normal structs, and that can be used to specify periodical events, such as ‘1 month and 3 days’.
These could then be combined with a starting date(time) to return the times of periodic events in the future. Other than with an exact interval (a {day, fraction_of_day}), this period is added per-field: in above example, first one month is added to the current date, ending up at the same day next month (or possibly the last day of the month if that month has less days). And then three days are added (maybe rolling over to the next month).

This idea is not new; Joda time uses it as well.

And of course, details of how this would work exactly is something to discuss .

[josevalim]

@OvermindDL1

Date/DateTime should not be a global structure, every calendar could potentially represent Date/DateTime differently and they should be calendar specific. You can use protocols or type_class’s or so to get generic information out of ‘any’ calendar of course.

There is absolutely nothing stopping this from happening today. Otherwise it is like saying you cannot implement another key-value data structure because Elixir already has a Map module. If you want key-value genericity, you can define the protocol and you can define new data types. The same for the calendars data structures.

Given the Gregorian Calendar is the international de facto and using it as a base is what makes sense for the huge majority of developers and the huge majority of times, it makes sense to prioritize its representation.

So while it is very interesting to see how far we can take the calendar structures we have today, I have absolutely no problem with saying we won’t support calendars X, Y and Z since the problem can be solved by a third party. In fact, we may even arrive to the conclusion that the :calendar field is unnecessary and any other calendar should be integrated via a protocol. All it takes is someone to define the protocols in a separate library, call it Zoolendar or whatever, which everyone can depend on when building new implementations.

[OvermindDL1]

@josevalim

Zoolendar

Hehe. ^.^

[Eiji]

I agree with protocols and separate time representations idea. Also agree with you that Gregorian should be default.
I can create simple demo structure that combines both notes today. I wws already think about it and want to ask too.

[Lau]

Is there a need to add new data formats to represent dates and times? There are a bunch already: Date, NaiveDateTime, DateTime structs in the ISO calendar. Additionally Erlang already has an integer representation of dates. E.g. {2017,1,8} |> :calendar.date_to_gregorian_days and also functions for “gregorian seconds”.

If we want to add a behaviour for alternative calendar modules, could it simply require functions for conversions to and from ISO structs? E.g. %DateTime{calendar: Calendar.ISO}, and the same for Date and NaiveDateTime.

ISO DateTime structs already support leap seconds. Although there are reasons for not always taking leap seconds into account when doing conversions.

You can know about leap seconds up to ~6 months into the future. The Tzdata library provides this information. The Calendar library (calendar on hex) uses the tzdata to make this list native Elixir 1.3+ DateTime structs. Example:

 iex> Calendar.TimeZoneData.leap_seconds |> List.last
 %DateTime{calendar: Calendar.ISO, day: 31, hour: 23, microsecond: 0,
    minute: 59, month: 12, second: 60, std_offset: 0, time_zone: "Etc/UTC",
    utc_offset: 0, year: 2016, zone_abbr: "UTC"}

This functionality is also used to validate DateTime structs where the second field has a value of 60.

Calendar.DateTime.from_erl({{2015, 12, 31}, {23, 59, 60}}, "Europe/London") returns an error because there was no leap second at 2015 December 31st.

Calendar.DateTime.from_erl({{2015, 12, 31}, {23, 59, 60}}, "Europe/London")
{:error, :invalid_datetime}

But in 2016 there was, so this is allowed provided the newest tzdata is available:

Calendar.DateTime.from_erl({{2016, 12, 31}, {23, 59, 60}}, "Europe/London")
{:ok, %DateTime{calendar: Calendar.ISO, day: 31, hour: 23, microsecond: {0, 0}, minute: 59, month: 12, second: 60, std_offset: 0, time_zone: "Europe/London", utc_offset: 0, year: 2016, zone_abbr: "GMT"}}

[Qqwy]

@Lau

Is there a need to add new data formats to represent dates and times?

I believe there is not. however, adding an :extra field to Date, DateTime and NaiveDateTime would be useful to make these existing types more suitable for storing information from other calendars.

@Lau

If we want to add a behaviour for alternative calendar modules, could it simply require functions for conversions to and from ISO structs?

As was mentioned in this topic in more detail earlier, a simple intermediate representation such as Rata Die:

1. makes conversions simpler; other calendars do not need to care about the Gregorian’s leap rules.
2. makes conversions between two non-ISO8601 calendars work for datetimestamps more than 6 months in the future, as no leap second information is needed.

[josevalim]

@Qqwy

(note: it theorerically is also possible that there are leap second deletions, in which case an UTC day might only have 86399 seconds. This has never happened during the past 45+ years that leap seconds are part of the internationally used calendar, but if we’d want to support it, we’d need 86399 * 86400 * 86401 as base.)

Going back into this, doesn’t it mean that, when encoding a given datetime to the tuple format, I would need to know if that particular day has leap seconds or not, since the presence of leap seconds changes how we will encode the seconds in a day into the whole fraction?

[Qqwy]

@josevalim

@Qqwy

(note: it theorerically is also possible that there are leap second deletions, in which case an UTC day might only have 86399 seconds. This has never happened during the past 45+ years that leap seconds are part of the internationally used calendar, but if we’d want to support it, we’d need 86399 * 86400 * 86401 as base.)

Going back into this, doesn’t it mean that, when encoding a given datetime to the tuple format, I would need to know if that particular day has leap seconds or not, since the presence of leap seconds changes how we will encode the seconds in a day into the whole fraction?

I thought earlier that it would be possible to ‘just pick a high enough denominator’ and then use this (implicit!) denominator for all conversion calculations, which would be precise enough.

Besides this being a bad idea because it is a weird, implicitly-defined constant, I found out that even when using a Superiour Highly Composite Number as fixed denominator, there would be many instances in which a loss of precision would occur. I’ll elaborate on these experiments if you want.

So, I made a very small proof-of-concept (using a map instead of a tuple for clarity) on GitHub that uses a {day, numerator, denominator} format.

When talking to @kip about this yesterday, he came with the very astute observation that it might very well be simplified to ‘just’ using a float: {integer_days, float_last_day_fraction}. While floats have a rounding error, the 16 decimals of guaranteed matissa precision are enough to work with times in sub-microsecond format.
I believe he is currently using this approach in his being-reworked Pull Request.

So, to address your question directly:

… doesn’t it mean that, when encoding a given datetime to the tuple format, I would need to know if that particular day has leap seconds or not, since the presence of leap seconds changes how we will encode the seconds in a day into the whole fraction?

Only if the to-be-encoded datetime is in UTC, i.e. Calendar.ISOor erlang-time-tuple format.
And then only if the to-be-encoded datetime is June 30th or December 31th.

If there should be a leap second in the current UTC day and we do not know about it (because, for instance, it is a date in the far future), we might be off by (86401 / 86400) - 1 = 0.000011574074074 seconds ≈ 11 microseconds for that particular date only. Any time lower than 23:59:60 will have even less error.

[josevalim]

@Qqwy

When talking to @kip about this yesterday, he came with the very astute observation that it might very well be simplified to ‘just’ using a float: {integer_days, float_last_day_fraction}. While floats have a rounding error, the 16 decimals of guaranteed matissa precision are enough to work with times in sub-microsecond format.

Ok, that was going to be my follow up point. It seems when encoding times between different calendars we would lose precision anyway because the days are slightly longer or shorter. So the “leap seconds base” is only useful when encoding in the same calendar. But, if it is the same calendar, we don’t need to encode the time.

@Qqwy

Only if the to-be-encoded datetime is in UTC, i.e. Calendar.ISOor erlang-time-tuple format.
And then only if the to-be-encoded datetime is June 30th or December 31th.

Perfect. This clarifies it. So would it make sense to rather add two functions to the Calendar behaviour? One that converts a Date to integer and another that converts the Time to a fraction? There is no need to convert the time if you are interested only on the date. /cc @kip

[Qqwy]

@josevalim

So would it make sense to rather add two functions to the Calendar behaviour? One that converts a Date to integer and another that converts the Time to a fraction? There is no need to convert the time if you are interested only on the date.

Almost. There is however, one important caveat to note: Some calendars start the next day at sunrise, some at noon, some at sunset and some at midnight. This means that conversion from one date to another might not be unambiguous, as a date in Calendar A might overlap with two dates of Calendar B.
Thus, I think that the date_to_rata_die should return an integer as well as a fraction.

I am not entirely sure yet how date (without time) conversion could be handled gracefully; we might want to return a list of possibilities. An alternate approach could be to restrict date (without time) conversions to calendars that have the same day-rollover point; meaning that if a developer wants to convert to another, he has to make a choice as for what time-of-day is used. This is similar to how NaiveDateTime is restricted w.r.t DateTime.

So I propose:

• @callback Calendar.datetime_to_rata_die(DateTime.t) :: {integer, float}
• @callback Calendar.time_to_day_fraction(Time.t, options) :: float. Returns a time divided by the total fraction of day. options is a keyword list that might be used to specify day properties that are not stored in the Time struct itself. (such as: should we treat the time as a time on a leap day).
• @callback Calendar.date_to_rata_die(Date.t) :: {integer, float}
• @callback Calendar.day_beginning_relative_to_midnight() :: float, returns 0.0 if the day rolls over begins at midnight, 0.25 if the day rolls over at sunrise, 0.5 if at noon, etc.

[josevalim]

@Qqwy

Almost. There is however, one important caveat to note: Some calendars start the next day at sunrise, some at noon, some at sunset and some at midnight. This means that conversion from one date to another might not be unambiguous, as a date in Calendar A might overlap with two dates of Calendar B.

Ooooooh, good point.

About the calendar functions, I will leave it up to @kip, but note the Calendar functions should not receive structs. Otherwise we have cyclic dependencies between those which, although they are fine in Elixir, it is usually not a good practice.

[alisinabh]

@josevalim another thing that caught my eye was there are callbacks like Calendar.date_to_string/3 and Calendar.datetime_to_string/11. Why there is no callbacks to get date from string? I mean, what to do to parse a string to a calendar implementation?

Sorry if this is irrelevant.

[josevalim]

Most of the questions to why Calendar in Elixir does not do something is likely because there was not a discussion about including such features. We could discuss parsing but that would be a separate discussion.