How much are nested changesets used in practice?

[marick]

TL;DR: Ecto has this nice feature where you can Repo.update a nested tree of changesets and Ecto figures out the appropriate SQL UPDATE and INSERT commands… or provides an error-annotated nested changeset.

I’ve been using that, and I keep encountering special cases. Am I doing something wrong? Do people use this feature or do they construct their own sequence of SQL commands?

---

Here’s the form in front of my recent code:

Submitting such a form requires (I think) a little massaging on the way into changeset-creation. Specifically, if the user didn’t attempt to fill in the “Add a gap” subform, that set of blank values has to be removed from the HTTP parameters. (They wouldn’t pass validation and, even if they did, they certainly shouldn’t be given to INSERT.)

But removing an empty “gap” subform has implications when there are errors elsewhere:

1. If there’s an error in only the top-level form (the animal), the animal’s changeset will have no entries for any gaps. So, having earlier removed the empty gap from the HTTP parameters, we have to make sure to create a new one for the “please fix these errors” version of the form.
2. If the user tried to create a new gap with validation failures, the user should not be shown an empty form for creating a new gap. The previous attempt to do that should appear, with the appropriate error annotations.
3. But if the validation failure was not in the “create a new gap” subform but rather in an update to one of the already-existing gaps, Changeset.get_change(:gaps) will have changesets for all of the existing gaps (the wrong one and all the others) - but we have to remember to provide the user with an empty form. (At least, that seems to me the most user-friendly course of action.)

The result is that I have code like this:

This level of special-case-ey-ness makes me think I must be doing something wrong.

[wolfiton]

Nested assoc and changesets are used when you don’t want to create relationships between entities. A good example is an online store. Also this post may explain more about your dilemma Embeds Vs. Associations

[josevalim]

It doesn’t feel like what you are doing is wrong but it also feels what you are doing is all UI concerns - which would explain (at least to me) why Ecto doesn’t take care of it. But Ecto does provide the groundwork for handling those scenarios.

For example, for handling an empty form, you can check if all params are empty and retturn :ignore by the cast_assoc changeset function.

Your function also works great but I would keep it as part of the UI functionality - if you don’t already do it. If memory serves me right, the functions in Phoenix.HTML for working with nested associations does support a prepend option, exactly for use case like yours, but I am unsure if it does exactly what you did.

[marick]

@wolfiton I don’t think I understand. Are you saying that nested assocs are meant to be used only with embedded structs, not structs related by a foreign key?

[marick]

@josevalim :ignore doesn’t solve the special cases. For example, consider:

    struct
    |> cast(attrs, required)
    |> validate_required(required)
    |> ToSpan.synthesize(attrs)
    |> cast_assoc(:service_gaps)
    |> IO.inspect(label: "after cast assoc")

I’ve made the changeset function for ServiceGap set an insertion changeset whose fields are all blank to :ignore. However, what seems to happen is that cast_assoc does the ignoring. That is, ServiceGap.changeset returns twice:

to ignore: #Ecto.Changeset<action: :ignore, changes: %{}, errors: [],
 data: #Crit.Usables.Schemas.ServiceGap<>, valid?: false>
to process: #Ecto.Changeset<
  action: nil,
  changes: %{out_of_service_datestring: "2020-01-05"},
  errors: [
    out_of_service_datestring: {"should not be before the start date", []}
  ],
  data: #Crit.Usables.Schemas.ServiceGap<>,
  valid?: false
>

… but: the containing changeset after cast_assoc only has the second:

    service_gaps: [
      #Ecto.Changeset<
        action: :update,
        changes: %{out_of_service_datestring: "2020-01-05"},
        errors: [
          out_of_service_datestring: {"should not be before the start date", []}
        ],
        data: #Crit.Usables.Schemas.ServiceGap<>,
        valid?: false
      >
    ]

So I still have to have special code to add back an empty ServiceGap changeset when one of the ones to be updated has an error.

I kick myself because I’d made a mental note to look whether an :ignore option existed, but I either forgot or somehow missed it. That plus :prepend (which I’d used in an earlier version) probably make the code better, but I’m not sure. (In addition to the above, there’s another special case when there are no existing ServiceGaps but one of the fields in the enclosing Animal is wrong. In that case, :prepend doesn’t add an empty ServiceGap because of the Animal error.)

[marick]

@josevalim. Re: “it also feels what you are doing is all UI concerns”. I agree. I’m bothered by how much my schemas “know” about where their data is coming from. When I’ve tried to push such concerns up out of the domain, it keeps seeming like I’m remando contra a maré - cutting against the grain of how Ecto wants me to organize things. I’d love to see examples of doing it differently/better.

[wolfiton]

Maybe this can help more Nested Forms and Associations in Phoenix 1.4.

Also from my point of view a separation of concerns can be a good thing and creates more readable code.

One question that is essential from my perspective is:

Is your code easy to manage and maintain?

If yes then continue using your method if not try something else.

[marick]

I’ve written up my solution here: https://www.crustofcode.com/an-example-of-nested-forms/ Possibly the most useful bit is I have a set of integration-style tests that can be used as a checklist for cases to cover.

[baldwindavid]

Thanks for taking the time to write this up and sharing. As you’ve illustrated it’s not often simple working with nested forms. Ecto schemas/changesets provide a nice api to work with, but there is often still work to do. To your original question, I tend to only use them for simple has_one relationships.

For forms that get more complicated, I typically use a custom module decoupled from the schema/persistence layer to represent the form with its own embedded schema and changesets.

To be fair, that is usually in cases where the form is not a direct representation of database tables. In your case, it seems like mostly CRUD and I’m assuming those fields in the form are the same as in the database.

Lately, I’ve found that using LiveView to manage pages like this is a lot easier to reason about than a big nested form with a lot of things that can go wrong. All the associated records can use separate updates, which removes most of the complexity. Additionally, it is no longer necessary to mark records for deletion with a checkbox. Just throwing that out there as another potential option.

[marick]

Do you have examples I can look at?

The forms aren’t exactly the same as in the database. For example, the forms have in_service_datestring and out_of_service_datestring values that need to be converted into a Postgres daterange type.

It does seem that I should be looking at LiveView rather than trying to make traditional HTTP solutions work. One thing that worries me is locking. A big submit button that changes everything atomically (or not) seems to me to be an easy conceptual model, whereas I don’t like the idea of two different users simultaneously editing different pieces of the same tree-of-structures.

But I have so little experience with frontend processing that I don’t know how to think usefully about such issues.

[baldwindavid]

@marick - I built a slim example app using some of the language of your app. There is a single commit that shows the diff between mix phx.new and my additions at… https://github.com/baldwindavid/crit/commit/4d1688aa7edaae180621a6ed01394224c874b28b

I skipped a lot of stuff, but this might give you a basic idea of the mechanics of using LiveView for this kind of thing.

You should be able to create an animal via a regular controller create and then be dropped into a LiveView for editing. On that page you can update the animal name and create, update, and delete service gaps. All of these are updates to single resources via separate events and without a page update.

The liveview is mounted within the edit template, but it could also have been mounted from the controller or router.

The animal and service gap forms all validate upon interacting with inputs and save upon clicking a button. There are a lot of ways this can be done and rate limiting can easily be added. It would also be trivial to change the forms to save upon editing and removing the submit buttons.

Most of the action takes place in CritWeb.Animal.EditLive and hopefully you’ll see that a good bit of it is boring, yet simple, repetitive code. For simplicity, the service gaps are queried upon just about any event to keep them up to date. Rate limiting or only updating upon submission would cut down on queries. The only slightly complex thing is replacing a changeset from the list of service gap changesets in some cases. Maybe this is a hacky way to do it, but seems to work… https://github.com/baldwindavid/crit/blob/4d1688aa7edaae180621a6ed01394224c874b28b/lib/crit_web/live/animal/edit_live.ex#L155

If you have multiple people editing at the same time, you might want to introduce some pubsub that live updates via actions that others make, but that was a step further than I went here.

I’m no expert in LiveView, but it is already providing me with the blocks to build some interfaces that I probably would have not done before because of the amount of JS involved. I am also one of those people that tries to avoid frontend work. I didn’t write any JS here. The only JS you’ll see in the diff is straight out of the installation docs… https://hexdocs.pm/phoenix_live_view/installation.html

Does that help? Let me know if you have any issues getting it running.

[marick]

This is awesome! Thank you. I’m in the middle of some things, but I will look at your solution as soon as I can.

I’ve put off learning LiveView because I wanted to learn a traditional web solution before investigating alternatives. But you encourage me to look at LiveView earlier than I had.