Calendars & Calendar Conversions

@Qqwy, thanks for starting this thread. And @alisinabh, apologies for hijacking your thread. From the earlier conversations and @qqwy’s summary above, i believe consensus so far is:

1. A calendar conversion mechanism in the Calendar behaviour is a good idea
2. That the conversation mechanism should cater for dates and times (ie. conversion of Date, NaiveDateTime and DateTime
3. Desirably the mechanism preserves monotonicity of time

The candidates I have seen in discussion so far are:

Julian Date (JD)
This is well understood, especially in astronomy circles. In incorporates an integral number of days since an epoch (epoch depends on which version of JD is chosen, but the original epoch of Julian day number 0 is assigned to the day starting at noon on January 1, 4713 BC, proleptic Julian calendar (November 24, 4714 BC, in the proleptic Gregorian calendar). The fractional part represents the fraction of a day since noon.

Plus: well understood, incorporates date and time, conversions have good precision
Minus: float format means time conversions are approximate and therefore whilst quite precise, cannot be assumed to be accurate in the way formal time systems are accurate.

Rata Die
This is the mechanism used by Derschowitz and Rheingold. This is essentially the same mechanism as Julian Day but with epoch of 1 January 0001. In their book Calendrical Calculations, RD is sometimes used as an integer meaning number of days since epoch - and this is what I originally proposed in order to avoid the issues of time conversion. An alternative form uses the same approach as JD, using a fractional part of a float to denote fraction of a day since noon.

Plus: Mostly the same as JD. According to Wikipedia this is used by Go, Rexx and .Net. Another benefit is that D&R have described and tested the algorithms for 23 calendar types.
Minus: Same as JD, and also it seems to be less familiar than JD

Unix timestamp
Common timestamp format so well understood. As long as its 64 bits (as Elixir is) then precision is enough to cater for all practical date/time ranges

Plus: well understood epoch of January 1, 1970 Gregorian; many other library functions that can work directly on Unix timestamps. Precise in that only integer arithmetic is required whereas JD and Rata Die require float arithmetic.

Minus: Not monotonic due to leap second issues

TAI (Temps Atomique International)
Monotonic clock which underpins UT1 time and is part of the UTC time standard means the format is standardised (several formats are available from accuracy of seconds down to nanoseconds depending on needs).

Plus: Monotonic
Minus: Difficult to compute, requires calculation of drift from UTC and a leap second table. Must adjust each time a leap second is declared.

This is clearly not a definitive list, nor a complete explanation. Given that the primary discussion here is about calendar conversion, I would propose JD as the basis of conversion since that is familiar to calendarises, allows calculation of both arithmetic and astronomical calendar conversions and is computationally quite easy to calculate. Clearly I’m also ok with Rata Die (using the float form to represent time) since there is a solid body of work to be leveraged but that is still true using JD. I think despite the benefits, TAI is too complicated to calculate and to maintain.

Could we use JD or RD but represent it as a tuple where the first element is the number of days and the second element is the number of microseconds elapsed in that day?

I think we could, as long as we define a very clear translation of day fraction ↔ microseconds. This might be as simple as saying ‘there are always 86_400_000_000 microseconds in a JD (or RD)’. (86_400_000_000 == 24 * 60 * 60 * 1_000_000)
This does mean that extra care needs to be taken when converting to/from UTC datetimes because of the afore-discussed leap second troubles:
On days at which UTC has a leap second, the conversion of an UTC microsecond is not 1:1 to a JD/RD microsecond, but rather 86401 : 86400 (an UTC day with a leap second contains 24 * 60 * 60 + 1 seconds). Most timestamps with microsecond integer precision will have to be rounded when doing this conversion.

As the gcd of 86400 and 86401 is 1, if we want to use an integer base in which no rounding would occur, we need to multiply it with the common base (86400 * 86401) == 7465046400 first.

So, if we want to have an exact (monotonically increasing) result, we’d need to store microseconds * 7465046400 in the second tuple field.

This does sound like a bit of hard work, but the nice thing about using a {day, in_day_amount} tuple is that the answers will always be exact and fast to convert to- and from UTC on any non-leap second days (Using for instance TAI as intermediate standard we would not be able to do this as in that case, every datetimestamp past a leap second needs to keep track of that leap second).
I believe we’d be able to claim microsecond exactness for all times on days except the June 30ths and December 31ths that are more than 6 months in the future; during those days, this calculation might at most be (amount_of_years_more_than_six_months_in_the_future * 2 * 86401) / 86400 seconds off.

(note: it theorerically is also possible that there are leap second deletions, in which case an UTC day might only have 86399 seconds. This has never happened during the past 45+ years that leap seconds are part of the internationally used calendar, but if we’d want to support it, we’d need 86399 * 86400 * 86401 as base.)

This kind of intra-day amount would also mean that conversions to- and from calendars that have a different (but well-defined) starting point for their days, as these would convert to static offsets in the intra-day amount.

Thanks for the reply. I guess that raises the question of, even if we account for leap seconds in the datetime representation, the only way to calculate the difference between two dates is by having in hand all of the leap seconds that have occurred between the two dates. And if we are going to keep this information in memory, we will also need a way to update it.

Does all calendars have a fixed amount of hours in a day, except by the leap second case, or do some calendars have shorter and longer days?

Yes and no. As we count in days and ‘day fractions’ (my previous post was an explanation of how such a day fraction could be stored in an integer format that would be precise enough to handle leap second conversions), to calculate the difference between two dates (in days + day fractions) will not need to handle leap seconds.

In pseudocode:

@common_base = 86399 * 86400 * 86401 
@milliseconds_times_common_base_per_day = 24 * 60 * 60 * 100_000 * @common_base
def difference({rdf1_day, rdf1_fract}, {rdf2_day, rdf2_fract}) do
  resulting_fract = rdf1_fract - rdf2_fract
  if resulting_fract < 0 do
    {rdf1_day - rdf2_day - 1, @milliseconds_times_common_base_per_day - resulting_fract}
  else
    {rdf1_day - rdf2_day, resulting_fract}
  end
end

When working with a POSIX time clock such as OS time, which ‘forgets’ that a leap second happened, conversions are accurate except during a leap second (during which different clock implementations (strict POSIX, NTP, et al.) will do different things such as repeating a second or ‘slowing down’ around it, neither of which can be recognized/handled at the level of our calendar implementation.) Lucky for us, Erlang already has done some work in this regard, having multiple different kinds of clocks (Erlang time documentation).

When converting to- or from an UTC datetime, we’ll need to consult the list of leap seconds for dates that are the 31st of December or the 30th of June, to check if that particular date is in there (but this thus is only necessary for those two days of the year!).
When converting to- or from TAI, you always need the list of leap seconds, as you’ll need to add/subtract all leap seconds that have happened before the timestamp under consideration.

Such a list therefore seems necessary. As to how handle updating the list: When working with POSIX times, the Network Time Protocol is often used (Erlang uses it) to keep the OS clock in sync and notify it about leap seconds. However, NTP only notifies you on the day itself that this is a day with a leap second. It therefore is unusable for working with dates in the future.

A simple solution would be to release new versions of the Calendar library each time a leap second is announced. I am not sure how easy it would be for systems that rely on hot-code reloading to update such an dependency while running.

In any case, it would seem fair to me to say that maintainers of applications that absolutely need the sub-second leap second precision should be considered responsible themselves to update their calendar library once every six months.

I believe the length of a day is earthwide the same; even if the length of daytime and nighttime differ depending on if you are on the poles or at the equator, the length of (daytime + nighttime) is the same*, which means that all calendars that have been made through the ages at the different parts of the globe have this same ‘day’ unit.

As for the subdivision of a day (in e.g. hours), this is something that calendars do differently.

(* unless we’d want to correct for relativistic time dilation differences, but that is definitely a precision and complexity we do not need for a general-purpose calendar library).

But to calculate the difference between two dates in seconds, we would need to know which days had leap seconds, correct?

Yes. This would be the same as converting both times to TAI timestamps first, and subtracting these.

Hi everyone.

Sorry if i’m bugging you, i just have to understand this problem About the problem with leap seconds. If we use UNIX since we are only dealing with date and NOT time, assuming even if every year from now will have a leap second, in year 86336 we will have a day misplaced and that is why we want to use JD. To support +84K years ahead (again, assuming every year will have a leap second). Correct?

It’s not always equal to a second. For more info see a leap second wikipedia article. How about create a simple API for Elixir to fetch a leap seconds?

elixir-lang.org/api/leap_seconds

We should know about add new second several months in advance, so update it is not a big problem .