I might also just start a plain IEx console and leave that running and see if the time drifts as that would point to it being M1/OS related. I wonder if when the laptop sleeps that could be causing the issue, that’s almost certainly the case each time I notice this. Of course I wasn’t taking detailed notes, but I can’t think of a time where i’ve started the server, and not had some reason that the laptop was sleeping for a few hours, and its usually after that that I notice it, but again this is just from vague recollection.

Agreed. The underlying issue is almost certainly something in OTP rather than Guardian, but I still think there’s value in switching from system_time to os_time in Guardian, no?

Also, interestingly, my IEx session has drifted back to 0.

iex(1)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
0
iex(2)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
196
iex(3)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
196
iex(4)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
197
iex(5)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
196
iex(6)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
197
iex(7)> 197 / 60
3.283333333333333
iex(8)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
195
iex(9)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
17
iex(10)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
17
iex(11)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
0

Yeah it’s a thing in Erlang because most runtimes are relatively ignorant and are not trying to provide quite the same guarantees around scheduling.

I think in this case I think the original time abstraction was simply insufficient and maintaining that abstraction in the face of precision time and wall clock warping led to some slow syncing / unwanted consequences.

But things are good now with the new time API AFAIKT.