Ecto: upsert with an embedded schema

I have the feeling that I’m missing something obvious…

While a dedicated create or update works I get an error message if I’m doing an upsert (which is understandable because I’m doing an insert with on_conflict).
How to do a correct upsert?

** (ArgumentError) got action :update in changeset for embedded Backend.Model.Location.OpeningPeriod while inserting
    (ecto 3.10.1) lib/ecto/embedded.ex:241: Ecto.Embedded.check_action!/3
    (ecto 3.10.1) lib/ecto/embedded.ex:173: anonymous fn/6 in Ecto.Embedded.prepare_each/5

The schema location contains an embed like:

    embeds_many :embedded_opening_periods, OpeningPeriod, on_replace: :delete do
      field :from, :time
      field :until, :time

The upsert code:

  def update_or_insert(location, attrs) do
    |> Location.changeset(attrs)
    |> Repo.insert(
      on_conflict: {:replace_all_except, [:id, :created_at]},
      conflict_target: [:id],
      set: [updated_at: DateTime.utc_now()]