Struggling with Inputs_for with has_many relations on a big LiveView form

Can someone help me with some nested Phoenix form associations - I have a large form that has a survey, each survey has multiple questions and each question has_many answers. I want one answer per user per question. I got the code working for submitting but the issue is once a new user wants to submit their answers it replaces.

I’ve changed my code to instead add a new changeset per question which I associate before passing down to the form, but now my form is rendering out an input for each answer (old and new).
I only want to render an input box for the new answers I want to gather but I am slightly lost on how to associate new answers and keep the struct the same.

Here’s my mounting code:

  def mount(_params, _session, socket) do
    survey = socket.assigns.reference.role.survey

    questions =
      Enum.map(survey.questions, fn question ->
        answer_changeset =
          %MyApp.Referees.FeedbackAnswer{}
          |> Referees.feedback_answer_changeset(%{})

        question_changeset = Ecto.Changeset.change(question)

        new_answer = %MyApp.Referees.FeedbackAnswer{} |> Referees.feedback_answer_changeset(%{})
        new_answers = question.answers ++ [new_answer]

        Ecto.Changeset.put_assoc(question_changeset, :answers, new_answers)
      end)

    changeset =
      %MyApp.Survey{id: survey.id}
      |> MyApp.Survey.changeset(%{})
      |> Ecto.Changeset.put_assoc(:questions, questions)

    socket =
      socket
      |> assign(:form, changeset |> to_form())

    {:ok, socket}
  end

Here’s my form:

  <.form :let={f} for={@form} phx-submit="save" class="flex flex-col gap-6 mt-8">
    <.inputs_for :let={qf} field={f[:questions]}>
      <.input type="hidden" field={qf[:question_id]} value={qf.data.id} />
      <div>
        <h2 class="text-xl font-medium"><%= qf.data.text %></h2>
        <%= case qf.data.type do %>
          <% "scale" -> %>
            <.inputs_for :let={af} field={qf[:answers]}>
              <div class="flex flex-row justify-between my-6">
                <div :for={value <- 1..7} class="mt-4">
                  <.input
                    type="radio-scale"
                    label_class="w-[45px] h-[45px] mx-[2px]"
                    field={af[:score]}
                    label={value}
                    option_value={qf.data.text}
                    value={value}
                  />
                </div>
              </div>
              <.input type="hidden" field={af[:user_id]} value={@reference.id} />
              <.input type="hidden" field={af[:question_id]} value={qf.data.id} />
            </.inputs_for>
          <% "select" -> %>
            <.inputs_for :let={af} field={qf[:answers]}>
              <div class="flex flex-col justify-between my-6">
                <div :for={value <- qf.data.question_options}>
                  <.input
                    type="radio-scale"
                    label_class="block text-center my-1"
                    field={af[:body]}
                    label={value}
                    option_value={qf.data.text}
                    value={value}
                  />
                </div>
              </div>
              <.input type="hidden" field={af[:user_id]} value={@reference.id} />
              <.input type="hidden" field={af[:question_id]} value={qf.data.id} />
            </.inputs_for>
          <% "text" -> %>
            <.inputs_for :let={af} field={qf[:answers]}>
              <.input type="textarea" field={af[:body]} placeholder="Your answer here" value="" />
              <.input type="hidden" field={af[:user_id]} value={@reference.id} />
              <.input type="hidden" field={af[:question_id]} value={qf.data.id} />
            </.inputs_for>
        <% end %>
      </div>
    </.inputs_for>
    <.button>Save</.button>
  </.form>