Ash.Flow: (ArgumentError) :utc_datetime expects microseconds to be empty

A :create action involving a utc_datetime type attribute is causing the above error when executed via Ash.Flow. If I run the same action outside of Flow, it doesn’t give me the same error. This makes me wonder if the date format is somehow modified as Flow executes the action. Where could I look to figure this out?

task = %{
 name: "buy milk",
  date: ~U[2023-12-04 19:22:36Z],

    |> Ash.Changeset.for_create(
    |> Tasks.create()  ---> This works fine  ----> This issues the above error

defmodule MyApp.MyFlow do
use Ash.Flow, otp_app: :my_app

flow do
   api MyApp.Tasks
   argument :task, :map, allow_nil?: false


steps do
   create :create_task, Task, :create do
           upsert? true
           upsert_identify :unique_key
           input arg(:task)

Full stacktrace

[error] GenServer #PID<0.4375.0> terminating
** (Ash.Error.Unknown) Unknown Error

* ** (ArgumentError) :utc_datetime expects microseconds to be empty, got: ~U[2023-12-04 19:22:36.922000Z]

Use `DateTime.truncate(utc_datetime, :second)` (available in Elixir v1.6+) to remove microseconds.

    at create_task
  (ecto 3.10.3) lib/ecto/type.ex:1376: Ecto.Type.check_no_usec!/2
  (ecto 3.10.3) lib/ecto/type.ex:552: Ecto.Type.dump_utc_datetime/1
  (ash 2.17.7) lib/ash/type/type.ex:945: Ash.Type.UtcDatetime.EctoType.dump/3
  (ecto 3.10.3) lib/ecto/type.ex:931: Ecto.Type.process_dumpers/3
  (ecto 3.10.3) lib/ecto/repo/schema.ex:1015: Ecto.Repo.Schema.dump_field!/6
  (ecto 3.10.3) lib/ecto/repo/schema.ex:191: Ecto.Repo.Schema.extract_value/6
  (elixir 1.15.5) lib/enum.ex:1825: anonymous fn/3 in Enum.map_reduce/3
  (stdlib 5.0.2) maps.erl:416: :maps.fold_1/4
  (elixir 1.15.5) lib/enum.ex:2522: Enum.map_reduce/3
  (ecto 3.10.3) lib/ecto/repo/schema.ex:87: anonymous fn/5 in Ecto.Repo.Schema.extract_header_and_fields/8
  (elixir 1.15.5) lib/enum.ex:1819: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
  (ecto 3.10.3) lib/ecto/repo/schema.ex:86: Ecto.Repo.Schema.extract_header_and_fields/8
  (ecto 3.10.3) lib/ecto/repo/schema.ex:48: Ecto.Repo.Schema.do_insert_all/7
  (ash_postgres 1.3.52) lib/data_layer.ex:1188: AshPostgres.DataLayer.bulk_create/3
  (ash_postgres 1.3.52) lib/data_layer.ex:1781: AshPostgres.DataLayer.upsert/3
  (ash 2.17.7) lib/ash/actions/create/create.ex:464: anonymous fn/10 in Ash.Actions.Create.as_requests/5
  (ash 2.17.7) lib/ash/changeset/changeset.ex:2209: Ash.Changeset.run_around_actions/2
  (ash 2.17.7) lib/ash/changeset/changeset.ex:1808: anonymous fn/3 in Ash.Changeset.with_hooks/3
  (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:1352: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
  (db_connection 2.5.0) lib/db_connection.ex:1630: DBConnection.run_transaction/4
    (ash 2.17.7) lib/ash/flow/flow.ex:26:!/3

I think this may be an internal issue with Ash.Flow not using proper type constraints. Let me take a look.

Okay, more context, the problem is that it’s doing an upsert, and the AshPostgres logic for setting what we call “lazy defaults” isn’t properly casting the input. A lazy default is a default value given by a function, which is how timestamps work, and that is causing the issue. I’m adding a fix to ash_postgres, and will release shortly.

EDIT: Can you try 1.3.64 of ash_postgres?

1 Like

Same issue there.

:thinking: are you on the latest version of Ash? Could you try doing the upsert outside of the flow? i.e YourResource |> Ash.Changeset.for_create(...) |> Api.create!(upsert?: true, upsert_identify :unique_key)? That should also break. That would help cut out some of the complexity.

Now it’s consistently failing with or without upsert, and in or outside of Flow. And the issue seems to be due to the timestamps, rather than :frowning:
As I troubleshoot this issue, I updated dependencies including ecto (3.10.3->3.11.0)and ecto_sql (3.10.2-> 3.11.0), so it might be introducing new issues.

Let me look into it further and report back.

:thinking: was it the ash_postgres update that made it start consistently failing?

No, I downgraded ash_postgres from 1.3.64 to 52 which is what it was before the update, and saw the same issue.

I noticed Ash-HQ’s migration exs files have :utc_datetime_usec type assigned to all timestamp attributes. How did you make it do that? My autogenerated migrations all show :utc_datetime instead.

Can I see your resource? There are two utc datetime types. :utc_datetime and :utc_datetime_usec. Which one are you using?

My resource’s timestamp attribute setting is this. timestamps(type: :utc_datetime) so having the same in migrations is as expected. What I didn’t expect is Ash-hq having timestamps() and yet its migrations show :utc_datetime_usec as data type. Is that the default behavior? I thought timestamps() defaults to NaiveDateTime.

timestamps() default to Ash.Type.UtcDatetimeUsec

1 Like

The error disappeared after changing the datetime type to utc_datetime_usec in migrations.

So the issue is probably not related to Ash.Flow. Thanks for your help in any case.

are you using the migration generator? Or handwriting migrations?

I normally use (and love) the migration generator. In this case, however, I manually changed

  1. timestamp(type: :utc_datetime) to timestamp() in each resource module
  2. changed the existing migrations accordingly
  3. made timestamp type changes manually in the db

Wouldn’t changing timestamps() and regenerating migrations have made that change for you? Or is there an issue?

You are right. I think I tried that and decided to go the manual route because the generated migration didn’t show the expected datetime type changes. It’s possible that I took steps in a wrong order along the way. In fact, when I tried on another machine (before pulling yesterday’s changes) the generator worked as expected (ie. captured the type changes).

That said, I have been getting mixed results from the generator lately where it includes previously migrated changes which I then have to manually delete before using the generated migrations. I suspect it might have something to do with the fact that I switch between different machines to do the development and perhaps the snapshots on those machines went out of sync somehow.

Yeah, that sounds likely. There is a ticket for solving this in ash_postgres by switching from snapshots to operations.

1 Like