handling discrepancy between a form UI and what should be persisted in the db

[err0r500]

Hi,
I’m working on a form and I’m not sure how to handle a specific type of input so I’m looking for advices & best practices.

Here’s how it works : the (text) input has an optional numeric value attached to it (we can think of an article that may have an additional cost in a specific context).

The UI is supposed to look like this :

• first article : no additional field
• second article : additional field set to 14
• third article is being created and for now the user has just activated the additional field but not set it yet (so we’ve got something I don’t want in the db : “has_additional” true, but additional value is nil.

Here’s what I currently do :
I use a functional component for the UI with the state handled by the lv
I use an Ecto schema to model this and in order to get it to work I included in it an “has_additional” along with the value of the additional field which is a nullable number.

The problem I have is that I’m not sure how to be consistent :

• in the db : if the value is nil, the “has_additional” should be false (and the opposite)
• in the UI : the value may be nil and the “has_additional” true when the user just activated the toggle
  (also, I’m not fond of having this “has_additional” field : having the value to nil or not should be enough )

The solutions I see :

• I’m thinking of using 2 different changesets : one for the validation and one for the persistence but it’s deeply buried so (in my understanding) I’d have to create a new Changeset for every schema.
• I guess I may also use a live component to encapsulate the toggle state

Not sure if I’m clear.
How would you handle this ?
Best,
Matthieu

[tty]

Would you consider having a supplement model and the article has a 1-to-1 relationship to supplement.

You can drop the has_additional field entirely.

The supplement.changeset would only create a supplement model if the UI field has a valid number.

i.e. article.supplement != nil means there is a valid numeric supplement.

[cmo]

Are you using an embedded schema or a schemaless changeset for the UI, or are you using the same one that persists data to the db?

[rhcarvalho]

If I understood well, it should be possible to generate a changeset for form validation that doesn’t store the temporary state in the db.

The validation for the phx-change event can also be different than the one for the phx-submit event.

If you feel the boolean helps drive the UI but you don’t want it persisted, you can tell Ecto that the field is :virtual when you declare it.

[garrison]

Unfortunately you have stumbled upon one of the uncomfortable and unavoidable truths of application development, namely that your data model must pass through invalid states on its journey between valid states, and that any attempts to circumvent this inevitably lead to poor UX. Here is a very long article about this exact problem if you are so inclined. You may find it enlightening, or existentially sad.

I’m afraid there is no good solution to this problem - the root cause is SQL databases themselves, and we’re stuck with them for now. I know it feels weird, but the correct solution really is to keep both fields and then handle the logic on the other end when you read the rows back. To soften the blow, you could perhaps default the value to zero, which depending on your exact use case might make things easier. But just keep in mind, somewhere down the road you will run into a situation like this again, and there might not be an easy way out.

[err0r500]

thanks to all of you for your responses.

so to answer the questions :
@tty : I don’t feel comfortable creating another schema for this : the situations arises in quite a few places of the application and the data structures are already complex (/deep) enough right now… and, I mean, it’s just a button ! (backend developper speaking, here )

@cmo : I use the same schema for the UI and the persistence

@rhcarvalho : I tried with a virtual field, the issue I have is to set its value :

• db → UI (read) the value of the “has_supplement” is determined on if the “supplement” has a value
• UI interaction (the user clicks the toggle, “change” event) : it sets the “has_supplement” to true, even if the value of supplement is nil at first
• UI → db (write) : if the “has_supplement” is false → set the “supplement” to nil ; if it’s true, store the supplement whatever its value (nil or number)

so to sum up the issue : on the read side : “has_supplement” is just not is_nil(:supplement) while on the UI and write side, it’s the other way : if :has_supplement, do: :supplement, else: nil so actually, maybe using different change sets for phx-change and phx-submit would be a solution, the issue is that the field is buried behind 4 assoc levels

@garrison thanks for the article, I’ll read the full article because, after reading the beginning, I clearly recognize myself as being a Stanley (dynamic typing & frontend are actually challenges I chose in order to leave my comfort zone ! )

I’ll put a minimal example in my personal “phoenix forms demo” repo and see what I can do (without being disturbed by the complexity of the current form). maybe a drop of JS might help ?
For now I can live with sightly incoherent state.

Thank you very much to all of you !

[rhcarvalho]

Have a look at how phx.gen.auth works with different changesets depending on what operation is being done (signing up a new user, login, change password, etc), that might be a good and readily available example to draw inspiration from (you can use the generator in a new Phoenix project and inspect the code).

Even if buried down levels of assoc, you probably will have a conceptual action that applies to all schemas involved… I guess it will be something like change_product which calls the respective change_* for every assoc. On the cast_assoc side, use the :with option to use the right function to cast each level of assoc. Docs: Ecto.Changeset — Ecto v3.12.5.

[err0r500]

thanks for the suggestion, I’ll have a look
I just also realized I could set a specific logic based on the content of the changeset (the :action, for instance) and handle the fields accordingly, I’ll give a try !

[rhcarvalho]

Yes, that’s true about the action, though sometimes it is only set after the validations (e.g. Ecto will set the action to :insert after user-defined validations have already run).

[err0r500]

ok, I think I’ve something that works :
I tried hard to use a functional component but finally gave up and used a live component instead

defmodule ManageWeb.CustomComponents.ActivableInput do
  use ManageWeb, :live_component

  attr(:id, :string, required: true)
  attr(:field, :any, required: true)

  def render(assigns) do
    ~H"""
    <div>
      <.input
        type="toggle"
        name={@field.name<>"-toggle"}
        value={@toggle_on?}
        phx-click={
          JS.push("toggle") |>
          JS.dispatch("set_to_null", 
                       to: "#" <> @id, detail: @toggle_on? |> to_string()) 
          }
        phx-target={@myself}
      />

      <.input
        type="text"
        id={@id}
        field={@field}
        phx-hook="NullableInput"
        readonly={!@toggle_on?}
      />
    </div>
    """
  end

  @impl true
  def update(assigns, socket) do
    {:ok,
     socket
     |> assign_new(:toggle_on?, fn -> not is_nil(assigns.field.value) end)
     |> assign(:field, assigns.field)
     |> assign(:id, assigns.id)}
  end

  @impl true
  def handle_event("toggle", _params, socket) do
    {:noreply, assign(socket, :toggle_on?, !socket.assigns.toggle_on?)}
  end
end

with the hook looking like this :

Hooks.NullableInput = {
  mounted() {
    this.el.addEventListener("set_to_null", e => {
     // toggle state seems captured at invocation so 
     // we update the input when the toggle is on -> off
      if (!!e.detail) {
        this.el.value = null
        this.el.dispatchEvent(new Event("input", {bubbles: true}))
      }
    })
  }
}

Not sure it’s the cleanest way to achieve this but seems good enough for now (and I can get rid of the “has_supplement” field in the schema)

[garrison]

Note that this solution is now haunted by the sort of UX problem I was talking about earlier. If a user presses the toggle (by mistake, perhaps) and then toggles it back on (to rectify their mistake), they will have inadvertently cleared the field, and it will not be obvious to them why.

Of course all engineering has tradeoffs and you’re the only person who can decide if that’s acceptable for your use case, but there is no free lunch I’m afraid. The toggle switch’s field is not redundant - it really does store useful information about the state, independent of the nullable field.

[rhcarvalho]

@garrison

If a user presses the toggle (by mistake, perhaps) and then toggles it back on (to rectify their mistake), they will have inadvertently cleared the field, and it will not be obvious to them why.

Very good point. The link you shared earlier was a good read too, thanks!

[err0r500]

yeap, I kept iterating a bit (thanks to @garrison comment and after testing the LiveView with a simulated latency) and here’s what I came up with (the value is restored when the toggle is switched off/on and it minimizes the possible race condition between the client and the server ) :

defmodule DemoWeb.CustomComponents.ActivableInput do
  use DemoWeb, :live_component

  attr(:id, :string, required: true)
  attr(:field, :any, required: true)
  attr(:label, :string, default: "")

  def render(assigns) do
    ~H"""
    <div>
      <.input
        type="toggle"
        name={@field.name<>"-toggle"}
        value={@toggle_on?}
        phx-click={JS.push("toggle")}
        phx-target={@myself}
      />

      <label>
        {@label}
      </label>

      <.input
        type="text"
        id={@id}
        field={@field}
        phx-nullify={JS.dispatch("null_or_restore", detail: @toggle_on? |> to_string())}
        phx-hook="NullableInput"
        phx-previous={@previous_value}
        readonly={!@toggle_on?}
      />
    </div>
    """
  end

  @impl true
  def update(assigns, socket) do
    {:ok,
     socket
     |> assign_new(:toggle_on?, fn -> not is_nil(assigns.field.value) end)
     |> assign_new(:previous_value, fn -> assigns.field.value end)
     |> assign(:label, assigns.label)
     |> assign(:field, assigns.field)
     |> assign(:id, assigns.id)}
  end

  @impl true
  def handle_event("toggle", _params, socket) do
    will_be_on? = !socket.assigns.toggle_on?

    {:noreply,
     socket
     |> assign(:toggle_on?, will_be_on?)
     |> set_previous_when_toggled_off(will_be_on?)
     |> push_event("js-exec", %{
       to: "#" <> socket.assigns.id,
       attr: "phx-nullify",
       value: will_be_on?
     })}
  end

  defp set_previous_when_toggled_off(socket, will_be_on?) do
    if will_be_on? do
      socket
    else
      assign(
        socket,
        :previous,
        socket.assigns.field.value
      )
    end
  end
end

with the relevant js part looking like this

Hooks.NullableInput = {
  mounted() {
    this.el.addEventListener("null_or_restore", e => {
        if (e.detail === "true") {
            this.el.value = this.el.getAttribute("phx-previous")
        } else {
            this.el.value = null
        }
        this.el.dispatchEvent(new Event("input", {bubbles: true}))
    })
  }
}

window.addEventListener("phx:js-exec", ({detail}) => {
  document.querySelectorAll(detail.to).forEach(el => {
    liveSocket.execJS(el, el.getAttribute(detail.attr))
  })
})

[garrison]

I’m not going to critique your approach any further (seems like you’ve arrived at a solution that works for you!), but I want to try to explain why this approach will never quite be enough - not in your case, but in a broad sense.

Imagine you decide that instead of a form-oriented design, you want to make your app real-time and interactive (features that are, rightfully, considered to be a selling point for LiveView). So, instead of writing a new row to the database on form submission, you instead write the row when it’s created and then update it live, with some PubSub magic so that others can see it. All good, right?

Well not exactly, because now you’re back to square one with the toggle. When the user clicks the toggle, you have to write that to the DB, so it has to pass validation! And if you don’t store an extra field for the toggle state, you will still have the UX issue from before where it clobbers the value of the field.

I want to emphasize that this is not a hypothetical. I have a LiveView app I’m working on (an RSS reader, alpha soon™) which works exactly like I just described. In particular, I have a UI control which allows a user to add link redirects, and each link redirect has a toggle switch to enable/disable it as well as fields to store the from/to patterns. It’s actually quite remarkably similar to your case.

Anyway, here’s the thing: my entire app is almost completely free of forms. Everything works in real time and is persisted back to Postgres. And just to make things more fun, I built full undo/redo support over the entire UI.

These things - realtime, undo/redo - would not work correctly if I structured my components as you have here. A user would hit the toggle by mistake, press undo, and the fields from that row would be gone. You could attempt to write more and more code to rectify this with special cases, but all you would end up with is an unmaintanable spaghetti nightmare. It can’t be done.

This is the point that article is trying to make. Features like undo/redo should be standard in every webapp, but they’re not because tools like SQL make them so hard to build. Once you have to persist the in-between states (in order to support undo), you can no longer maintain such a strict validation policy. As I said before, that toggle field is not redundant! The only reason you can get away with removing it is that you are avoiding persisting the in-between states to the database, and that is the compromise which harms UX. There is no way out.

[linusdm]

@garrison

Here is a very long article about this exact problem if you are so inclined. You may find it enlightening, or existentially sad.

Wow, this is a good article! Thanks for sharing It makes a lot of uncomfortable situations I’ve encountered in the past regarding UX/front-end design very explicit. They were uncomfortable in the sense that I’ve always thought there should be a simpler solution. But now it turns out UI is messy very enlightening to read indeed.

The article is quite abstract, and doesn’t translate directly to code (especially because the author is referencing examples in a more traditional SPA-and-backend architecture). I think Phoenix applications already have good tools to separate “intent” from state by using Ecto changesets (a “changeset” is more or less a synonym for “intent” in this regard), and schemaless changesets if UI doesn’t exactly match the database structure. There is much more to it, and there is still the object(struct!)-rdbms impedance mismatch that doesn’t encourage you to store intermediate/invalid state.

I’m curious how you’re tackling things differently with this explicit “intent” in mind, using the existing building blocks the Elixir ecosystem is providing you to build the “live” RSS reader you’ve described. If this would take us too much off-topic we can split off the discussion. But I’m very interested in this topic, as it seems to uncover some problems I’ve run into again and again.

PS: I’m a Stanley too… (this is a reference to the article)

[garrison]

In order to support undo/redo across the UI, I funnel all actions through a set of structs called Ops (operations) which each encode a change (e.g. rename a feed). Each struct implements a behavio(u)r with a callback that executes that particular change (this is a thin wrapper over a normal Phoenix/Ecto context). This is often referred to as the command pattern.

This might sound like something you could implement on an existing Phoenix app, but it’s not. The magic happens after each “op” is executed: it returns its inverse. For example, %Rename{name: "bar", ...} for a feed currently named "foo" would return %Rename{name: "foo", ...}, which can then be added to an undo stack. In practice in order to do this you have to design every one of your context functions with this in mind. Ideally, you want to return the state which was previously stored in the database (rather than your potentially-stale Ecto struct) to avoid corrupting the undo state, which I can assure you is a colossal pain because Postgres’s consistency guarantees suck and it was not designed for this at all. But it can be done, you just have to be very careful and know exactly how the database works (which thankfully I mostly do). You also have to be very careful to soft-delete everything or things will break.

I should note that some of the operations are considerably more complicated, particularly because I (for some reason) felt the need to commit to drag/drop nested folders.

The realtime stuff is a lot more to explain but the TLDR is it’s a lot of Phoenix PubSub.

I will write more about this when I move to public alpha testing. This will be a commercial product but I plan to publish all code for transparency anyway so I’ll definitely post on here about it when I do!

[garrison]

@linusdm

I think Phoenix applications already have good tools to separate “intent” from state by using Ecto changesets

On this point I don’t think I agree. Changesets are mostly about providing a convenient wrapper around existing database validations. They exist to wrap the DB and make its errors more friendly to deal with in your Phoenix app. They’re an ORM feature, not literally, but in essence.

The problem is that in order to build truly great UX you have to invert this model. Changesets validate data before it goes into the DB, but what you would want, to follow the model discussed in the article, is to validate the data when it comes out of the DB. Obviously in practice you would probably want to do both.

The “validations” we’re talking about here are not the sorts of simple declarative validations Changesets provide, either. They are really an arbitrary function of the state.

To use the example that this thread is actually about, because it really is a great example, you might have rows in your DB with a boolean enabled state and an integer value. As we discussed earlier, you could attempt to fit this into one nullable integer, but in doing so you have corrupted the state: disabling the row should not erase the value.

So instead, in your changeset, you want to allow the boolean to be true/false, and you want to allow the integer to be whatever (including null if the user has not yet filled it out). Then, when you read back the rows, you would essentially do this:

def value(%Row{enabled: false}), do: 0
def value(%Row{enabled: true, value: value}), do: value || 0

In a more advanced example this function could be much more complicated! But what I’m trying to emphasize is that this really doesn’t resemble what Changesets are meant to do. I’m sure you could find some way to accomplish this with them if you really wanted to, but the result would probably be a mess.