Timex.now(timezone)

I had that feeling

Ok this is very embarrassing… The original solution with the timezone did fix this issue. The continued issues we were seeing were due to a recent change in our deployments, we were just deploying an old release . We only deploy once a week or so otherwise I think we would have caught that sooner.

Sorry, I was asking how :at value is consumed, but you show the code that passes around the value, but not the code that really uses the value. :at is passed to start_missing_jobs!/2, but I still don’t know how the value is used.

More specifically (why I’m asking that), is I want to know what time zone is used, as you said your server is configured in UTC. So :at is 13pm (at: {13, 0, 0}) in what time zone?

and also code of do_format(date, @time_format)

Sorry @thojanssens1 I was handling some other things last week.

:at is consumed through a function in the orechestrator:

  defp to_entry_point_map(entry_point) do
    %{
      queue: Keyword.fetch!(entry_point, :queue),
      ...
      at: Keyword.get(entry_point, :at)
    }
  end

That function is called by the orchestrate function:

  def orchestrate(opts \\ []) do
    exq_pid = opts[:exq_pid] || ExqApi
    {:ok, scheduled_jobs} = ExqApi.scheduled(exq_pid)

    for entry_point <- entry_points() do
      entry_point
      |> to_entry_point_map()
      |> start_missing_jobs!(scheduled_jobs)
    end
  end

start_missing_jobs!/2 starts jobs that are scheduled based on the arguments consumed from to_entry_point_map/1

How is this :at option consumed?

Sure.
We have an ExqOrchestrator that is queueing jobs with Exq. The config file looks a little like this:

config :application, Application.Jobs.ExqOrchestrator,
 entry_points: [
    ...
 [
      queue: "report",
      worker: ReportWorker,
      task: "build_and_send_report",
      at: {13, 0, 0}
    ]
]

ReportWorker:

  def build_and_send_report(opts \\ []) do
   # enqueues job for tomorrow
      atrs = %{
        task: "build_and_send_report",
        args: %{},
        worker: @worker,
        queue: @queue
      }

      Jobs.perform_on(atrs, Timex.shift(Timex.now("America/Denver"), days: 1))

    # functions to actually send the report
    Reporting.send_report(opts)
  end

That’s Exq and the worker… Looking into the actual Reporting.send_report/1 the function starts to get a bit complicated because we are running a few database queries gathering information and then processing it but it eventually calls a deliver_email function which calls a ReportEmail module:

  def make_report(report_vals, opts \\ []) do
    new_email()
    |> from("email@example.com")
    |> put_recipients(opts)
    |> subject(dgettext("report",  "Daily Report "))
    |> put_layout({MailerView, :reporting})
    |> render(:daily_report, daily_report: report_vals)
  end

The layout in Mailerview contains the Timex.now() stamp that is shown on the email.

Can you show the code that shows how the report is sent at 13:00?

This report is sent by a worker which queues itself to run every 24 hours. There is a config file that starts several workers when the server is restarted with specific times each worker is to run. In the case of this report, 13:00 is the time the report gets sent, this works because we receive the report daily at 1.
The worker calls a function that actually builds the report. Inside that function, we assign an HTML template for the email.
The label in the email is this:

<%= dgettext("report", "Time: %{time}",
              time: format_date(Timex.now("America/Denver"), :time_only) 
       )
%>

format_date is

@time_format "{h12}:{m} {am}"
def format_date(date, :time_only), do: do_format(date, @time_format)`

So if Timex gives you the right time on the server for America/Denver (as you proved by opening an iex session on the server and show date with timex), then my next question would be:

• Who sends this report? How is the sending triggered and at what time (and which time zone)?
• Is the report pre-generated or generated at the same time it is sent?

I believe they are, isn’t RTC not being in UTC an evidence that they are? I’m not positive but that’s what I’m assuming.