Ecto changeset validations cause lag in displaying nested form fields

I have a form that I use to add users to an organization, that looks like this:

Screenshot 2025-01-20 at 17.22.532472×516 36.4 KB

The problem:
Clicking on the Add User button takes a few milliseconds (~500ms) before the input fields are displayed, and the delay compounds as I add more users. Note that clicking that button sends a validate event to the server.
Checking the server logs, validation happens for all of the already filled fields. I’m not sure if that’s expected, or I’m doing something wrong in the code. It feels redundant to re-validate all the fields; I expect validation to happen only for the field in focus, but maybe it’s how inputs_for/1 works.

This is my changeset function:

 def invite_changeset(organization, attrs) do
    organization
    |> cast(attrs, [])
    |> cast_assoc(:users,
      with: &User.invitation_changeset/2,
      sort_param: :users_sort,
      drop_param: :users_drop
    )
  end

What I have tried so far:
I am preloading :users that cast_assoc/3 is using. I have tried not running validations untill the form is submitted, but that doesn’t work with inputs_for/1; the form is not being rendered when I click on Add User button on the UI. Also tried providing a changeset that has no validations (just casting) when validating, but the delay is still there, even though cast_assoc/3 does no validation, just casting the params.

The question:
Honestly, I am not entirely sure what the cause of the delay is, but I suspect it is the changeset validation. How can I fix this delay?