To whom are you replying here? In any case, the point about UT1 and UTC not being equal (and TAI ↔ POSIX not being equal) because of the difference in leap seconds has been made in this topic before.
This will not work for the following reasons:
Not a problem! Calendars are hard, and it is important to ensure that we do it properly and that everyone understands what is going on.
We want to use Julian Date or Rata Die because they are really simple, unambiguous representations of the single common format that all calendars (created on our Earth) share: The period of a day.
How this day is subdivided (hours/minutes/seconds; the Hebrew calendar for instance uses 24 hours, each subdivided into 1080 ‘parts’), superdivided (how/if weeks/months/years are counted) and finally where the border between two days lies (noon, midnight, sunset) is something that greatly varies per calendar, however.
The JD/RD formats are able to store this information well, because they use days as unit of measurement.
The calendar library will be forced to deal both with dates and times, as when the next day starts (noon, midnight, sunset) varies per calendar, so if you do not specify a subdivision in the current day, there might be multiple possible dates in the target calendar.
Leap days (Julian/Gregorian and Islamic), Leap months (Hebrew) and Leap Seconds (UTC) were only added after the fact, when people found out that the way they subdivided/superdivided days in their calendars turned out to not be totally conform what happened in nature.
Leap days/Leap months are artefacts to ensure that the seasons do not drift across the dates in the calendar too much, as a (solar) year does not take e.g. 365 days but actually (approx.) 365.24219 days.
The leap second that was added to UTC was introduced because it turned out that the length of a day is also not exactly 86400 SI-defined seconds long, but rather dependent on the exact speed of earths rotation, which is not constant but fluctuating slightly. Leap seconds are peculiar in that they cannot be predicted but are artificial agreed-upon corrections of UTC towards the actual ‘natural’ day length, when expressed in SI-seconds.
It is possible for zero, one or two leap seconds to be added to UTC (and theoretically also for one or two leap seconds to be removed, although this has not happened yet during the ±45 years that leap seconds are used) per UTC year. This means that, if we forget applying leap seconds starting now, that it will take approx. 43227 UTC years to be off by more than a day.
So, to reiterate:
JD or RD are useful because they count in days, which is nice because the length of a day is the same in all calendars, regardless of if or when they use leap months/days/seconds or not. When using JD or RD as ‘base’ representation, these leap artefacts only need to be considered during conversion from the respective calendars to JD/RD or vice-versa, and not during any intermediate calculations.
I like José’s proposal for {RD/JD, milliseconds} a lot.
I wonder if we can cater for the leap second issue on conversion to this intermediate format. It seems easier and more intuitive to convert to milliseconds elapse including leap second offset at this stage. Then the calculation of date/time difference is simpler - albeit still have to watch out for a negative offset and adjust accordingly.
As far as I can tell we just need the leap_seconds to apply the relevant offset. It would also make for easier pattern matching for far forward and far past dates since leap seconds have a short history (and maybe a short future!).
If this becomes the agreed approach then is it better to separate out Calendar from core so that it can be updated in cycle with leap year announcements and out of cycle with Elixir core?
As for intra-day conversions - my understanding is that the leap second is always in inserted/removed at 23:59:59 UTC. Therefore there still remains a challenge of conversion for non-UTC times and therefore the time difference for non-UTC times/datetimes is somewhat problematic?
On the updating topic - given the intent of BEAM apps to be long-lived perhaps in addition to a leap_seconds table, an API to update via external service is also a good idea in addition to the table? And if thats a good idea, perhaps that makes a stronger case for Calendar being decoupled from core?
Sorry, I was not clear. I wrote a small reply (from my Nexus 5) to @alisinabh and I mean that not every year have count of leap seconds equal to 1. I also linked a wikipedia article where he can see a table about added leap seconds in years. We can’t assume that every year could have exactly one leap second. More: it’s also possible (but not yet happen) to have -1 leap second.
I have the feeling that some of my posts might have been too long, so to reformulate:
@kip, @josevalim I propose we use a {RD/JD, fraction} approach, as this would mean that calendars do not have to care about their internal details: For instance, when converting from the Islamic to the Hebrew calendar, leap seconds would not need to be considered at all!
The fraction part could either be an arbitrary-precision Decimal, a Rational number, or a fixed-precision integer representation that is precise enough to handle very small differences between day subdivisions. Using floats is of course out because of their rounding errors. Using decimals or rationals is out if we want to include the Calendarium in the core (or minimize dependencies for another reason).
If we want therefore to use a fixed-precision representation, we need to choose something that is precise enough to handle very small differences. The leap second differences in the UTC calendar are the smallest ones that we will need to handle for the forseeable future.
We know that we want to represent all of 1 / 86400 (one SI-second on a normal day) 1 / 86401 (one SI-second on a leap-second-day) and 1 / 86399 (one SI-second on a leap-second-removal-day). Milliseconds are not precise enough. Attoseconds are.
To whomever is confused by the concept of leap seconds:
Consider a straw.
This straw has a certain length.
I can put my ruler against it, and measure that it is ‘10.25’ centimeter.
I can also put a ruler in inches against it, and measure that it is ‘4.035’ inches.
Neither observation has altered the length of the straw.
If we want to count multiples of straws lengths in fixed, centimeter-size steps, we could say ‘well… lets assume they are exactly 10 centimeter long’.
But now here’s the problem: After counting four straws, we end up with 40 centimeter. But in reality, four straws are 41 centimeter. Oh snap! The more straws we count, the more our calculation is off.
The solution? Proclaim that every fourth straw is a leap straw which is not ten, but 11 centimeter long!
These proclamations have not altered the length of the actual straw in any way either!
The straw in this analogy is of course the day, whose length is the same across all calendars, because that’s simply how the earth rotates.
When we convert from Super Strawcounting With Leapcentimeters to Inches or vice-versa, it is useful to choose an intermediate representation that is based upon the length of the straw itself: This representation will never need leap straws, as ‘a quarter straw’ is always exactly a quarter of a straw, regardless of with what ruler it is measured.
And therefore, if IERS decides that another leap second is needed, this does not alter that ‘a day is a day’ (one full rotation of the earth remains one full rotation of the earth), but only how many SI-seconds fit in that given day. And therefore, counting with fractions of a day is more suited to calendrical conversions than choosing e.g. milliseconds as in-day unit.
@Qqwy i think there is still the need to take leap seconds into account if we are talking about a diff function for datetimes? there is such a function now for naivedatetime, which doesn’t consider leap seconds and i think probably that needs an update.
For purely conversion i can seee it’s not a consideration.
We need to take leap seconds into account when writing a function such as seconds_between.
We will not need to take leap seconds into account when writing a function such as period_between, where ‘period’ will be returned as full days + final day fraction.
i have closed my PR on conversion using UNIX.