Can I upsert with cast_assoc?

Is it possible to use cast_assoc upsert relations? I did this:

changes = %{"id" => 1, "thing" => [%{"id" => 1}], "name" => "hi"}
Ecto.Changeset.cast(%MyStruct{}, changes, MyStruct.__schema__(:fields))
|> Ecto.Changeset.cast_assoc(:thing)
|> Repo.insert!(
  returning: true,
  on_conflict: {:replace_all_except, [:id]},
  conflict_target: [:id]

But I get a primary key constraint on :thing. I thought that cast_assoc would figure out it’s an update not an insert because I gave it a primary key.

Is there a way to tell the insert to replace on the nested relation?