How to perform an upsert without loading associations?

Assume an Author schema:

schema "authors" do
    field :name, :string
    has_many :posts, Post
  end

  def changeset(struct, attrs) do
    struct
    |> cast(attrs, [:name])
  end

I can perform an update:

author |> Author.changeset(%{name: "Jo"}) |> Blog.Repo.update!()    

but an upsert fails:

 author |> Author.changeset(%{name: "Jo"}) |> Blog.Repo.insert!(on_conflict: {:replace_all_except, [:id]})
** (RuntimeError) attempting to cast or change association `posts` from `Author` that was not loaded. Please preload your associations before manipulating them through changesets

Is it possible to perform an upsert without loading the associations?
Or is this by design?