How to manage associations during creates/updates?

I have a schema Element with a has_many relationship to Photo. I want to manage this relationship when creating/updating an Element. I persist the order of the Photos in an order field in the join table. Right now I do all this manually with a lot more code than I’d like.

  1. Load Photos from changeset, to ensure they all exist and belong to the user (privileges)
  2. Create or update Element in an Ecto.Multi
  3. Manually create the ElementPhoto schemas after comparing with existing ElementPhotos

Is there a cleaner way to do this? I looked into Ecto.Changeset.put_assoc, but it 1) doesn’t generate IDs for my join table (I use UUIDs), and 2) doesn’t seem to provide a way to update the order field.

  def create_user_element(attrs, user_id) do
    changeset =
      %Element{user_id: user_id}
      |> Element.changeset(attrs)
      |> validate_element_photos(user_id)

    Multi.new()
    |> Multi.insert(:element, changeset)
    |> Multi.merge(fn %{element: element} ->
      new_photo_ids = Ecto.Changeset.get_field(changeset, :photo_ids)
      element_photos_multi(element.id, [], new_photo_ids)
    end)
    |> Repo.transaction()
    |> case do
      {:ok, %{element: element}} -> {:ok, element}
      {:error, _field, %Ecto.Changeset{} = changeset, _} -> {:error, changeset}
    end
  end

  defp validate_element_photos(changeset, user_id) do
    photo_ids = Ecto.Changeset.get_field(changeset, :photo_ids, [])

    photos =
      Photo
      |> where([photo], photo.id in ^photo_ids and photo.user_id == ^user_id)
      |> Repo.all()

    found_ids = Enum.map(photos, & &1.id)
    missing_ids = photo_ids -- found_ids

    case missing_ids do
      [] ->
        changeset

      missing_ids ->
        Enum.reduce(missing_ids, changeset, fn missing_id, changeset ->
          Ecto.Changeset.add_error(
            changeset,
            :photo_ids,
            "Photo #{missing_id} not found"
          )
        end)
    end
  end

  defp element_photos_multi(element_id, existing_photo_ids, new_photo_ids) do
    removed_photo_ids = existing_photo_ids -- new_photo_ids
    added_photo_ids = new_photo_ids -- existing_photo_ids

    multi = Multi.new()

    case {removed_photo_ids, added_photo_ids} do
      {[], []} ->
        # no-op, no changes
        multi

      _ ->
        # delete all previous items (easier than trying to rearrange them)
        multi =
          multi
          |> Multi.delete_all(
            :delete_photos,
            from(p in ElementPhoto,
              where: p.element_id == ^element_id
            )
          )

        # add all items in proper order
        new_photo_ids
        |> Enum.with_index()
        |> Enum.reduce(multi, fn {photo_id, index}, multi ->
          changeset =
            %ElementPhoto{}
            |> ElementPhoto.changeset(%{element_id: element_id, photo_id: photo_id, order: index})

          multi
          |> Multi.insert("photo #{Integer.to_string(index)}", changeset)
        end)
    end
  end

Thanks!

You could write the assocs in your schema like this:

has_many :element_photos, ElementPhoto
has_many :photos, through: [:element_photos, :photo]

and in your changeset/2 add cast_assoc(:element_photos) (also add cast_assoc for photo to your ElementPhoto).

Then you have to give the nested changes to your changeset function like so:

%{
  "element_photos" => [
    %{"order" => 1, "photo" => %{"filepath_or_something" => "/foo/bar"},
    %{"order" => 2, "photo" => %{"filepath_or_something" => "/foo/baz"},
  ]
}

Depending on what should happend on update or delete you should also set on_replace and on_delete for the associations.

Thanks, I think that’s gotten me really close!

My changes are failing though because there’s no :element_id in the ElementPhoto. Do I have to add that manually? This would break the process of using this in a create Changeset, since I don’t have the Element’s id yet.

If you are using cast_assoc/2 the element_id should get set automatically during the insert / update.

Hrm, strange. I’m doing that.

photo_elements: [#Ecto.Changeset<action: :insert, changes: 
%{trip_id: "caa440ad-e196-4d1a-803e-67cb9b91710e"},
errors: [element_id: {"can't be blank", [validation: :required]}], 
data: #Ido.Elements.PhotoElement<>, valid?: false>]

My PhotoElement has cast_assoc(:photo) and cast_assoc(:element), as well as belongs_to for both of those associations. My PhotoElement changeset does have validate_required([:element_id, :trip_id])…so maybe I have to validate either element_id OR element? But that error above doesn’t show the element in the changes, so it appears it wasn’t included either way.

Yes, validate_required for the foreign_keys doesn’t work with cast_assoc.

Take a look at the documentation.
You can specify required: true as an options in cast_assoc.

Ok, I replaced validate_required with my own validate_either function which ensures either the ID or association is present. The changeset is still failing because neither the element and element_id are present. Ecto doesn’t seem to be making that connection. Is there a step I missed to ensure it fills in that property?

Could you share your schemes / changeset functions?

Sure! Here are my updated data functions (I’m trying to add Trip and Photo associations):

  def create_user_element(attrs, user_id) do
    attrs = attrs |> add_element_trips() |> add_element_photos()

    %Element{user_id: user_id}
    |> Element.changeset(attrs)
    |> validate_element_photos(user_id)
    |> validate_element_trips(user_id)
    |> Repo.insert()
  end

  defp add_element_photos(attrs) do
    photo_ids = Map.get(attrs, :photo_ids, [])

    photo_elements =
      Photo
      |> where([photo], photo.id in ^photo_ids)
      |> Repo.all()
      |> Enum.with_index()
      |> Enum.map(fn {photo, index} -> %{"photo_id" => photo.id, "index" => index} end)

    Map.put(attrs, :element_photos, photo_elements)
  end

  defp add_element_trips(attrs) do
    trip_ids = Map.get(attrs, :trip_ids, [])

    trip_elements =
      Trip
      |> where([trip], trip.id in ^trip_ids)
      |> Repo.all()
      |> Enum.map(fn trip -> %{"trip_id" => trip.id} end)

    Map.put(attrs, :trip_elements, trip_elements)
  end

# Element Schema
.......
    field(:photo_ids, {:array, :string}, virtual: true, default: [])
    has_many(:element_photos, Ido.Elements.ElementPhoto)
    has_many(:photos, through: [:element_photos, :photo])

    field(:trip_ids, {:array, :string}, virtual: true, default: [])
    has_many(:trip_elements, Ido.Elements.TripElement)
    has_many(:trips, through: [:trip_elements, :trip])

  def changeset(element, attrs) do
    element
    |> cast(attrs, [
      :trip_ids,
      :photo_ids
    ])
    |> cast_assoc(:trip_elements)
    |> cast_assoc(:element_photos)
...


# ElementPhoto

defmodule Ido.Elements.ElementPhoto do
  @moduledoc false
  use Ido.Schema
  import Ecto.Changeset

  typed_schema "element_photos" do
    field(:order, :integer)
    belongs_to(:element, Ido.Elements.Element)
    belongs_to(:photo, Ido.Media.Photo)

    timestamps()
  end

  @doc false
  def changeset(element_photo, attrs) do
    element_photo
    |> cast(attrs, [:order, :element_id, :photo_id])
    |> cast_assoc(:photo)
    |> cast_assoc(:element)
    |> validate_required([:order])
    |> validate_either(:element_id, :element)
    |> validate_either(:photo_id, :photo)
    |> foreign_key_constraint(:element_id)
    |> foreign_key_constraint(:photo_id)
  end

  defp validate_either(%Ecto.Changeset{} = changeset, key1, key2) do
    case {Ecto.Changeset.get_change(changeset, key1), Ecto.Changeset.get_change(changeset, key2)} do
      {nil, nil} ->
        error_message = "#{key1} or #{key2} required"

        changeset
        |> Ecto.Changeset.add_error(key1, error_message)
        |> Ecto.Changeset.add_error(key2, error_message)

      _ ->
        changeset
    end
  end
end

Here’s the attrs that goes into the changeset function:

%{
  element_photos: [
    %{"index" => 0, "photo_id" => "b92ec3d4-3ecf-46e3-a1ca-b7365af9d10d"}
  ],
  photo_ids: ["b92ec3d4-3ecf-46e3-a1ca-b7365af9d10d"],
  trip_elements: [%{"trip_id" => "ab2ae592-985c-40c2-b0d6-09565f602302"}],
  trip_ids: ["ab2ae592-985c-40c2-b0d6-09565f602302"]
}

And the resulting changeset error:

#Ecto.Changeset<
  action: nil,
  changes: %{
    element_photos: [
      #Ecto.Changeset<
        action: :insert,
        changes: %{photo_id: "b92ec3d4-3ecf-46e3-a1ca-b7365af9d10d"},
        errors: [
          element: {"element_id or element required", []},
          element_id: {"element_id or element required", []},
          order: {"can't be blank", [validation: :required]}
        ],
        data: #Ido.Elements.ElementPhoto<>,
        valid?: false
      >
    ],
    photo_ids: ["b92ec3d4-3ecf-46e3-a1ca-b7365af9d10d"],
    trip_elements: [
      #Ecto.Changeset<
        action: :insert,
        changes: %{trip_id: "ab2ae592-985c-40c2-b0d6-09565f602302"},
        errors: [
          element: {"element_id or element required", []},
          element_id: {"element_id or element required", []}
        ],
        data: #Ido.Elements.TripElement<>,
        valid?: false
      >
    ],
    trip_ids: ["ab2ae592-985c-40c2-b0d6-09565f602302"],
  },
  errors: [],
  data: #Ido.Elements.Element<>,
  valid?: false
>

Yeah, that doesn’t work. The ElementPhoto changeset doesn’t have an Element or element_id when you create it via cast_assoc.

If you plan to create an ElementPhoto outside of the Element context I would suggest you to create a new changeset function without the Element validation and use that one with the has_many association, otherwise just remove the validation completly.

Yeah, exactly I was trying to understand how that could work. Since this is a changeset to create the Element, I don’t have it yet to put the ID in the association. That’s why I was using an Ecto.Multi before so I could get the element to use it’s ID for the children. So I guess there isn’t a cleaner way with Ecto to avoid this multi-step process I’m currently doing?

You just have to add e.g.:

def element_changeset(element_photo, attrs) do
  element_photo
    |> cast(attrs, [:order, :photo_id])
    |> cast_assoc(:photo)
    |> validate_required([:order])
    |> validate_either(:photo_id, :photo)
    |> foreign_key_constraint(:element_id)
    |> foreign_key_constraint(:photo_id)
end

and change your
cast_assoc(:element_photos)
to
cast_assoc(:element_photos, with: &ElementPhotos.element_changeset/2

and it should work like you want to.

I don’t follow. My Element is definitely using the ElementPhotos changeset already for the cast_assoc, as it’s showing the validate_either message. The problem is the Element (which doesn’t exist yet) isn’t being added to the changeset. I can’t do it, and Ecto isn’t doing it for me.

If you construct it via cast_assoc from an Element, the ElementPhoto doesn’t have an Element in the changeset.
The association is handled during insert / update of the Element and the correct ids for the associations getting set automatically.

Awesome, that got it! I actually ended up not needing to call the changeset explicitly, just cast_assoc(:element_photos) did the trick.

Thank you so much! This really helped me clean up that mess!