How to support multiple week calendars in Elixir?

[josevalim]

@kip said:

Calendar module responsibilities

I think a calendar should encapsulate everything relevant to calculating days, weeks, months, years etc. That is, any configuration should be baked into the module that implements the Calendar behaviour.

This seems obvious when we talk about the iSO calendar versus the Balinese , or Ancient Egyptian or the Islamic calendars. But its less obvious if we consider the myriad of calendars derived from the Gregorian.

For example, in the US, a company may have a fiscal calendar that starts on an arbitrary day of the year. All months, quarters, weeks and days are calculated from this date. Cisco Systems has a financial year calendar that ends on the last Saturday of July. There are a whole class of calendars called 445 calendars and their cousins the 454 and 544 which are common in retail and are designed to make it better for comparing this year to prior year periods.

Although I started thinking this would be best done as configuration passed around in each structurally compatible Date , Time or DateTime struct or map, the variations are too many. I now feel it is better that this detail be backed into a module that implements the Calendar behaviour. This doesn’t make implementing a calendar any more complex. It does potentially have the side effect of having to generate calendar modules at runtime for some use cases.

And then @josevalim said:

I am not sure I agree with this.

The instants are the same and they are represented by the same year, month, day, hour, minute and second, so it does not feel to me they should be different calendars. To me, it is the same calendar, but it wants to be traversed or viewed in different ways. My suggestion would be to explore a WeekTraverser or WeekBuilder or some sort of behaviour that focuses on exploring a given calendar through different interpretations of its weeks. It may be even something that can be implemented on top of iso days (and therefore calendar agnostic)?

Which @kip finally replied:

Totally ok with not agreeing. I just can’t reconcile how to do what you suggest and still use the Date , Time and DateTime standard structs which have a :calendar field only. Also every calendar is basically just a different representation of the same moments in time so by extension aren’t you saying there is only one calendar and multiple different representations that model year, month, day in alternate ways?

I will continue the discussion below.

[josevalim]

Totally ok with not agreeing. I just can’t reconcile how to do what you suggest and still use the Date , Time and DateTime standard structs which have a :calendar field only.

That’s a good point. I can think of cases this would be an issue but I can’t come up with anything concrete. Do you have any? Shall we explore this a bit further?

Here is a slightly more complete proposal. Imagine you have a data-structure called %WeekDay{}. It has fields such as year, week, and day of week. It also has a “week_calendar” that knows how to convert it to iso days so you can convert it back to a Date or even a DateTime with timezone (you can embed timezones on the Weekday data structure if you want to or even have NaiveWeekDay without tz and WeekDay with tz).

Does this make your life better or worse? I guess though what you are implementing here is exactly a Date*, except you are using the month as week and the day as day of week. Although it may still be worth exploring if it makes computations easier or harder.

Also every calendar is basically just a different representation of the same moments in time so by extension aren’t you saying there is only one calendar

They all represent the same point in time but they are different representations. In your case, they are the same representation. So I guess my point here is that it is weird to change calendars when they have the exact same underlying representation.

[kip]

José, I appreciate the opportunity to work this through, it’s been vexing me for a while.

Each calendar as a concrete implementation (reasoning)

Convertability

One of the reasons I came to the idea that each variation is its own calendar is because conversion between calendars becomes straight forward using the existing Calendar module functions.

As an example, lets say I’m using a 445 calendar. It has exactly the same notion of year, month and day. Except that the start/end of the year is a different Gregorian date and the month numbers don’t align either. Neither do the week days. For financial reporting purposes I would use my 445 calendar. But of course I also want to know what those periods are in the Gregorian/ISO calendar. Using date_to_iso_days/1 and iso_days_to_date I can convert between the two in a very straight forward manner. I don’t need to use another api to manage representations.

Encapsulation

As we discussed earlier, but keeping all the calendar configuration in the relevant Calendar module, a calendar now easily conforms to the standard Date, Time and DateTime structs which is in line with a developers expectations. Although the date/time/datetime functions only require structural compatibility, i think the developer experience is simplified if the principle of one module == one calendar is kept.

Implementation details

No doubt that there is plenty of room to optimise implementation for several classes of calendars, including 445 style, and calendars that have different min_days and first_day_of_week.

My thinking was something like this:

Configuration

defmodule MyCalendars.SomeCalendar do
  use MyCalendars.Gregorian, min_days: 4, first_day: 1
end

defmodule MyCalendars.RetailCalendar do
  use MyCalendars.Calendar445, type: :454, ends: {:last, 7, 7}
end

By no means the exact API but it will be reasonably straight forward to encapsulate the generalised version of these typical calendar types in a concrete fashion.

Module generation

One additional challenge with this approach is runtime configuration. As one example, a locale_string can actually specify the desired calendar as defined in BCP47. So assume we have an accept-language header with a language-tag it might look like this:

Accept-Language: he-IL-u-ca-hebrew-tz-jeruslm

Represents Hebrew as spoken in Israel, using the traditional Hebrew calendar, and in the “Asia/Jerusalem” time zone as identified in the tz database.

The ca=hebrew is the calendar configuration and it could be retail or cisco or whatever is interpreted by the serving system. In this case, a module needs to exist and it might need to be generated at runtime. I am not sure the optimal way to do this.

[josevalim]

As an example, lets say I’m using a 445 calendar. It has exactly the same notion of year , month and day . Except that the start/end of the year is a different Gregorian date and the month numbers don’t align either.

Interesting! So the representation that you are keeping in Date/Time is actually the year of week and month of week? The other question is: what would those calendars return for week_of_year? Just the information in the struct?

EDIT: Meanwhile I have removed week_of_year from the Calendar callbacks because I am really uncertain it is the best way to go. If using calendars is the best way to model week-based calendars, then having week_of_year is like having japanese_era as a Calendar callback.

defmodule MyCalendars.SomeCalendar do
use MyCalendars.Gregorian, min_days: 4, first_day: 1
end

I do like this.

[Qqwy]

I haven’t had the time yet to read through this discussion in much detail, but I do want to say that I really like that we are having this discussion !

I will post an in-depth post later today or tomorrow.

[kip]

Interesting! So the representation that you are keeping in Date/Time is actually the year of week and month of week? The other question is: what would those calendars return for week_of_year ? Just the information in the struct?

I’ve gone backwards and forwards on this and concluded that for any given Calendar, the year, month and day are that calendars representation. This fits for Gregorian derivatives (like 445, or US fiscal) since the notion of year, month and day remain but are offset from the Gregorian.

Where it is uncomfortable is for week-based calendars, like ISOWeek. In this case I use month to mean week. year and day retail their normal meaning but within the ISOWeek context. Its not perfect but its workable.

Meanwhile I have removed week_of_year from the Calendar callbacks

Every calendar we have discussed has the concept of a week (French Revolutionary has 10 days, but there rest, afair are 7 days). Maybe I’m an outlier on this. I don’t see week_of_year as valuable to only week-based calendars like ISOWeek.

For Gregorian-based calendars, the first_day and min_days determine the start of the weekly cycle. Weeks of year (and also quarter and month) are used quite a lot for scheduling and reporting ("… my forecast for week 11 of quarter 3 is …"). I hope there may be a case to return the callback, but clearly it can be implemented outside of the behaviour.

[josevalim]

Where it is uncomfortable is for week-based calendars, like ISOWeek . In this case I use month to mean week . year and day retail their normal meaning but within the ISOWeek context. Its not perfect but its workable.

Why not use the week-based month here too? Since that is also specified by ISO? I am mostly curious.

[kip]

Why not use the week-based month here too? Since that is also specified by ISO? I am mostly curious.

Mostly because the standard for ISOWeek says:

The ISO standard does not define any association of weeks to months. A date is either expressed with a month and day-of-the-month, or with a week and day-of-the-week, never a mix.

[Qqwy]

I finally had time to catch up with the discussion.

---

In the earlier discussion we had when talking about implementing the calendar behaviour, I noted that the concept of a week is orthogonal to the concept of days/months ❦, and arrived at the same reasoning as we did here, that when calculating with weeks, this could be modeled by a separate calendar that uses the month-field to store the week number.

If I remember correctly, this was not done at that time and we ended up including day_of_week in the Calendar behaviour; the reasoning being that it was common enough for people to want to look up the week to include it in the calendar directly.

I think this reasoning is still valid, although I do not think that we should make the Calendar behaviour itself more complex because of it.

Currently, I like the %WeekDay{}-approach the best; I think it might look as follows:

• We introduce a new behaviour called WeekCalendar, which mainly works with %WeekDay{} (or maybe %WeekDate{}/%WeekDateTime{}-structs?)
• We might find that there is some overlap between the WeekCalendar and Calendar behaviours (maybe we can extract this to a separate behaviour?)
• Converting between a Calendar and a WeekCalendar is possible using the date_to_iso_days-helper in the middle.
• We decide on some way to configure (and provide defaults?) on conversions that happen from a Calendar to a compatible WeekCalendar.
• This conversion might be used under the hood in the implementation of e.g. day_of_week, and of course play a role in the new formatter-proposal we’re working on in a similar fashion.

I see the main advantage to introducing a new behaviour as twofold:

1. There is a clear separation between :month and :week, week-based calendars do not have to think about how the months work (and only implement behaviour callbacks that make sense for week-related datastructures).
2. People working with the results are able to pattern-match on the :week-field rather than a :month-field as well.

---

❦ Side note: Calendar-weeks are often based on a religious background, although there is research that suggests that there definitely is psychological merit for humans to having a work-leisure cycle of ~7-8 days. In any case, the relation between weeks and ‘months’ for any given calendar is either ‘very loose’ (when trying to evenly subdivide the synodic lunar month of ~29.5 days), or completely non-existent. @kip 7 day-weeks are the most common, but there are:

• 7-day week calendars with ‘leap weeks’ to make sure every new year starts at the same day of the week again.
• The old Roman 8-day week Nundiae (‘market week’) that was in use until the Julian calendar took over.
• Besides the French Revolution calendar, there are a couple of ancient Egyptian and Chinese calendars with 10-day weeks.
• A couple of places used a 5-day calendar (Iceland, Java, Korea)
• The Maya calendar has two variants, one dividing the 260-day year in 20 13-day weeks, and one to divide the 260-day year in 18 20-day ‘months’ that were each subdivideded in 5-day weeks, and having one extra ‘monthless’ 5 day week.
• The Soviet Union had a 5-day week (and later also a 6-day week) that was used alongside the traditional 7-day one for factory workers.

[kip]

Its been a while but I’m getting reasonable close to releasing ex_cldr_calendars. The focus has been on gregorian-based calendars that follow the gregorian month cycle with a variable starting month as well as week-based calendars that have a variable start or end day and a configurable week.

After a number of design experiments I am settled on an approach where a calendar module can be defined statically such as the calendar that is defined by the US National Retail Federation. Its quite simple:

defmodule Cldr.Calendar.NRF do
  use Cldr.Calendar.Base.Week,
    min_days: 4,
    anchor: :last,
    day: 6,
    month: 1
end

Which basically means the calendar year ends on the last saturday nearest the end of January. An example of a financial year calendar for a US corporation that elects to have its financial year finish on the last Saturday in July would be:

defmodule Cldr.Calendar.CSCO do
  use Cldr.Calendar.Base.Week,
    min_days: 5,
    anchor: :last,
    day: 6,
    month: 7
end

An example of a month-based calendar would be a financial year calendar for a country. In the US the government financial year starts in October so it is defined by:

defmodule Cldr.Calendar.US do
  use Cldr.Calendar.Base.Month,
    month: 10
end

(Australia would be month 7, the UK would be month 4).

Calendars can be generated at runtime by passing the configuration to Cldr.Calendar.get_or_create(calendar, config) or to Cldr.Calendar.get_or_create_for(territory) which will create a financial year calendar module configured for a given territory (country).

Thoughts on the Calendar behaviour

As I’ve gone through this process I have asked myself what I would want to add to the current Calendar behaviour and the answer is: very little.

1. I would like to propose that the Inspect protocol implementation delegate to an inspect/2 function on a calendar. This would be very useful when inspecting week-based calendars, or when using (as I do) a sigil_d/2 to define dates in different calendars. For an ISOWeek calendar for example, it would - in my implementation - return ~d[2019-W12-02]ISOWeek which is a lot more meaningful that the struct dump - especially since for week based calendars one has to hijack the month field and use it instead as week.

2. I would like to propose revising the Calendar.week_of_year/1 addition to the Calendar behaviour. Weeks are such a standard part of most modern calendars, including the ISO calendar. The behaviour today nearly every other date facet: era, year, quarter, month, day. Week is the notable exception. Reporting data based upon weeks is an intrinsic part of business activities; they are how we structure formatted calendars.

These are really the only two behaviour additions that after a few months of calendar hacking strike me as additions I’d like to see - which is a testament to the original design and the recent 1.8 additions.

Cldr.Calendar behaviour builds on the core Calendar

In my own Cldr.Calendar behaviour I definitely have other requirements that I don’t believe are relevant to the core Calendar module but may be of interest to other calendarises:

• year/1, quarter/1, month/1, week/1 that return a Date.Range
• week_in_year/3 that takes a date, :first_day and :min_days per the earlier discussion
• iso_week_in_year/1 that delegates to week_in_year/3 with first_day: 1, min_days: 4
• first_gregorian_day_in_year/2 and last_gregorian_day_in_year/2 to return what it says

I don’t see any of these belonging in the core Calendar module but they are very useful in the practical use of calendar building and calendar usage.

I still have to add spec, docs and more tests so cldr_calendars is definitely not ready for anyone but it will be in a couple of weeks or so. Then I can finally finish up version 2.0 of ex_cldr_dates_times which only has to focus on formatting (think of it as an CLDR-based formatting using the LDML formats in a localised context).

[josevalim]

Fantastic work @kip!

I would like to propose that the Inspect protocol implementation delegate to an inspect/2 function on a calendar. This would be very useful when inspecting week-based calendars, or when using (as I do) a sigil_d/2 to define dates in different calendars.

Yes, please do send a PR!

I would like to propose revising the Calendar.week_of_year/1 addition to the Calendar behaviour.

We added it to Elixir but removed it before v1.8 was released exactly because the understanding of the “week of year” is locale specific. The only other functionality that I believe to be locale specific in the current calendar is day_of_week. So it felt awkward to add week_of_year given that it would be limited in many cases.

However, my understanding is that week_of_year would help your CLDR because calendars will now be able to rely on this uniform callback. Is this correct?

year/1 , quarter/1 , month/1 , week/1 that return a Date.Range

Can those be implemented using the existing Calendar callbacks? Those look nice additions to Elixir but I think we would need to add at least days_in_year and days_in_week.

[kip]

I’ll work on the PR for the inspect part first, that seems the most straight forward.

The week_of_year is by no means a requirement for CLDR-related work. It’s that to me a calendar that doesn’t have the notion of weeks is incomplete since thats a cycle within which human activity is firmly rooted. I accept that thats hardly a rigorous argument though.

For the date ranges:

• year/2 can be defined as {year, 1, 1}..{year, months_in_year(year), days_in_month(year, months_in_year(year))}

• quarter/2 can be defined in terms of months_in_year(year) / 4

… and so on. If there is interest in a more detailed proposal or PR its not much work and I’m happy to do it.

[josevalim]

For the date ranges:

• year/2 can be defined as {year, 1, 1}..{year, months_in_year(year), days_in_month(year, months_in_year(year))}
• quarter/2 can be defined in terms of months_in_year(year) / 4

Ah nice!

I would like to see that, yes, but as Date.year_range/1, quarter_range/1, etc. What about week_range/1 though? We would need days_in_week, wouldn’t we?

[kip]

José, I’ve been prototyping a bit today in the way you suggested: implement in Date by leveraging the existing callbacks. For month based calendars thats working well. For week based calendars it may not be that easy. Work in progress.

Date.week_range/2 would require Date.week_of_year/2 I believe (the earlier conversation).

I’ll summarise my findings here after some more experimentation in a day or so.

[josevalim]

Another question: would the approach above work with week-based calendars since we use month to mean week?

[kip]

Well it was more than a day or so but I can summarise my findings now that ex_cldr_dates_times is published.

Week calendars are complicated

The interpretation of a week varies quite a lot, even in Gregorian calendars, based upon usage and expectation. For example:

1. How is a week interpreted in a month-based calendar like Gregorian (Calendar.ISO)?
2. How is a month interpreted in a week-based calendar (like ISO Week)? These kinds of calendars (excluding ISO Week) define a “month” based upon 13 week quarters split into one of either 4-4-5, 4-5-4 or 4-4-5 weeks.
3. If a calendar starts in July (like The Australian fiscal year calendar) is it referred to as the beginning year or the ending Gregorian year? That is, if the fiscal date is 2019-01-01 (meaning July 1st) what is the Gregorian date? Is it 2019-07-01 or is it 2020-07-01?

When does a week start?

There are both cultural and definitional differences.

• Different countries start the week on either Sunday or Monday as common practise
• Companies in countries that allow defined fiscal years can start their calendar on the “first or last day_of_week”. For example, Cisco’s fiscal year is defined as last Saturday in July.
• Fiscal calendars for countries can and do start in different Gregorian months of the year. For example, the UK fiscal year starts in April, Australia in July and the US in September.

Calendar configuration for week based calendars

In order to provide configuration for calendars implementing week-based behaviour I found I had to define the following configuration elements:

# Each quarter has three
# 'months` each of 13 weeks
# in either of a 4,4,5; 4,5,4
# of 5,4,4 layout
weeks_in_month: [4, 4, 5],

# Indicates if the anchor
# represents the beginning
# of the year or the end
begins_or_ends: :begins,

# Calendar begins on the
# :first, :last or :nearest
first_or_last: :first,

# First week of the year begins on this day.
# of the week.
# `day: :first` means that the first week starts
# on the first day of the year. `day: :first` only
# applies to month-based calendars and it
# will also result in a short week for the last
# week of a year
day: 1,

# Year begins in this month
month: 1,

# The year of the last_day or first_day
# is either the year with the :majority
# of months or the :beginning year
# or :ending year
year: :majority,

# First week has at least
# this many days in current
# year
min_days: 7

Callbacks

In my implementation I found I required the following callbacks to implement week-based calendars:

  @doc """
  Returns the `month` for a given `year`, `month` or `week`, and `day`
  for a a calendar. Required to provide for returning the notion of a month
  for a week-based calendar,

  The `month_of_year` is calculated based upon the calendar configuration.

  """
  @callback month_of_year(
              year :: Calendar.year(),
              month :: Calendar.month() | Cldr.Calendar.week(),
              day :: Calendar.day()
            ) ::
              Calendar.month()

  @doc """
  Returns a tuple of `{year, week_in_year}` for a given `year`, `month` or `week`, and `day`
  for a a calendar.

  The `week_in_year` is calculated based upon the calendar configuration.

  """
  @callback week_of_year(
              year :: Calendar.year(),
              month :: Calendar.month() | Cldr.Calendar.week(),
              day :: Calendar.day()
            ) ::
              {Calendar.year(), Calendar.week()} | {:error, :not_defined}

  @doc """
  Returns a tuple of `{year, week_in_year}` for a given `year`, `month` or `week`, and `day`
  for a a calendar.

  The `iso_week_of_year` is calculated based on the ISO calendar.

  """
  @callback iso_week_of_year(
              year :: Calendar.year(),
              month :: Calendar.month(),
              day :: Calendar.day()
            ) ::
              {Calendar.year(), Calendar.week()} | {:error, :not_defined}

  @doc """
  Returns a tuple of `{month, week_in_month}` for a given `year`, `month` or `week`, and `day`
  for a a calendar.

  The `week_in_month` is calculated based upon the calendar configuration.

  """
  @callback week_of_month(Calendar.year(), Cldr.Calendar.week(), Calendar.day()) ::
              {Cldr.Calendar.month(), Cldr.Calendar.week()}

  @doc """
  Returns the calendar basis.

  Returns either :week or :month
  """
  @callback calendar_base() :: :week | :month

  @doc """
  Returns the number of periods (which are
  months in a month calendar and weeks in a
  week calendar) in a year

  """
  @callback periods_in_year(year :: Calendar.year()) :: week() | Calendar.month()

  @doc """
  Returns the number of weeks in a year

  """
  @callback weeks_in_year(year :: Calendar.year()) :: week()

  @doc """
  Returns the number of days in a year

  """
  @callback days_in_year(year :: Calendar.year()) :: Calendar.day()

Calendar Interval callbacks

I use Interval.year/2, Interval.quarter/2 and so on. Would be renamed Calendar.year_range/2 based on the earlier conversation.

  @doc """
  Returns a date range representing the days in a
  calendar year.

  """
  @callback year(year :: Calendar.year()) :: Date.Range.t()

  @doc """
  Returns a date range representing the days in a
  given quarter for a calendar year.

  """
  @callback quarter(year :: Calendar.year(), quarter :: Cldr.Calendar.quarter()) :: Date.Range.t()

  @doc """
  Returns a date range representing the days in a
  given month for a calendar year.

  """
  @callback month(year :: Calendar.year(), month :: Calendar.month()) :: Date.Range.t()

  @doc """
  Returns a date range representing the days in a
  given week for a calendar year. 

  Returns an error tuple if the calendar has no 
  notion of a week as is the case with the Julian 
  calendar. 

  """
  @callback week(year :: Calendar.year(), week :: week()) ::
              Date.Range.t() | {:error, :not_defined}

Basic Calendar math

When incrementing or decrementing years, quarters, months, weeks and days - including date ranges - most of the heavy lifting can be done in a base calendar module.

However for incrementing (and decrementing) months and quarters a callback is required since the nature of a quarter and a month is calendar specific.

  @doc """
  Increments a `Date.t` or `Date.Range.t` by a specified positive
  or negative integer number of periods (year, quarter, month,
  week or day).

  Calendars need only implement this callback for `:months` and `:quarters`
  since all other date periods can be derived.

  """
  @callback plus(
              year :: Calendar.year(),
              month :: Calendar.month() | week(),
              day :: Calendar.day(),
              months_or_quarters :: :months | :quarters,
              increment :: integer,
              options :: Keyword.t()
            ) :: {Calendar.year(), Calendar.month(), Calendar.day()}