Dynamic Embeds in Ecto

AndrewDryga

Hey guys. I know this is an old topic, but as I write more and more complex application with Elixir and Ecto I feel like we really need a way to let developers use dynamic embeds.

Few words about our use case, we have a Transport schema which stores various common fields and settings of a transport. But depending on transport type (eg. Twillio and Facebook Messenger) settings can be very different and also there are DB constraints that should be in place for those settings.

We do work around this issue with an application logic which takes params for the embedded schema (which defined as :map type on parent changeset) and validates/casts them if embedded changeset is valid or properly adds errors to the parent otherwise. Here are some code:

A function that shows how dynamic changeset works in our case:

  defp cast_provider_settings(changeset, provider_field, provider_settings_field) do
    with {:ok, provider} <- fetch_change(changeset, provider_field),
         {:ok, settings} <- fetch_change(changeset, provider_settings_field),
         provider_settings_changeset = Provider.settings_changeset(provider, settings),
         {:ok, valid_settings} <- Validator.fetch_valid_attrs(provider_settings_changeset) do
      put_change(changeset, provider_settings_field, valid_settings)
    else
      :error ->
        changeset

      {:error, :not_found} ->
        changeset

      {:error, %{valid?: false} = settings_changeset} ->
        put_embedded_error(changeset, provider_settings_field, settings_changeset)
    end
  end

Here is how you can add an error to an embedded changeset defined as map:

  defp put_embedded_error(changeset, embed_field, embedded_changeset) do
    embedded_type =
      {:embed,
       %Ecto.Embedded{
         cardinality: :one,
         field: embed_field,
         on_cast: nil,
         on_replace: :raise,
         owner: %{},
         related: Transport,
         unique: true
       }}

    %{
      changeset
      | changes: Map.put(changeset.changes, embed_field, embedded_changeset),
        types: Map.put(changeset.types, embed_field, embedded_type),
        valid?: false
    }
  end

(Notice that you can’t override types and leave embedded changeset in Ecto Schema where :map type was defined because you would get a cast error. Ecto.Changeset does use pre-compiled type information when insert happens so overriding only helps when you use functions like traverse_errors/2.)

And even if you do that, there is a lot of issues that persist here. The main one right now for us is constraints - they are lost when embedded schema turned into a map and moving them manually to parent doesn’t make sense (error field would point to a wrong direction).

Other ways to hack around:

Define multiple schemas per database entity (or even combine that with PostgreSQL table inheritance). This one looks weird for me because when I fetch data back from DB I do want to see only one kind of schema. Data that I want to put there should be exactly what I get back.
Do not use dynamic embeds. This option looks poor because there is sooo many use cases where dynamic embed makes perfect sense.

As a very raw suggestion how we can deal with that:

We might add a :changeset type for Ecto.Schema.
It’s application responsibility to actually implement logic how embedded changeset gets there, on which fields it’s resolved, etc. (I don’t think that Ecto needs to add any kind of magic here.)
Repo operations should take care of changesets in :changeset fields in a same way as they would do with usual embedded schema.

Make Ecto use type information from changeset (removing the calls to Schema.__*__ functions) which is not straightforward and would make changesets structs much bigger. (See this issue.)

24 comments

#ecto #discussion

3 5535 24

2020-06-02 01:15:56 UTC

First 10 of 24 Posts

wojtekmach

Hex Core Team

Can you talk a bit more about your use case; on the DB level is it e.g. transports table with a few columns, including e.g. provider_type (string) and provider_settings (json) columns?

And even if you do that, there is a lot of issues that persist here. The main one right now for us is constraints

What types of constraints, like CHECK constraints?

Post #1

AndrewDryga

@wojtekmach I guess migration would answer both questions. In short - yes, it’s 2 columns. Constraints can be very different, I can’t tell which we will use in future. Currently it’s unique index and CHECK’s.

create table(:transports, primary_key: false) do
  add(:id, :binary_id, primary_key: true)
  add(:title, :string, null: false)
  add(:provider, :string, null: false)
  add(:provider_settings, :map)
end

execute("""
CREATE INDEX transports_provider_settings_user_id_index ON transports
USING GIN ((provider_settings->'user_id'))
""")

execute("""
CREATE UNIQUE INDEX transports_facebook_provider_settings_page_id_index ON transports
USING btree (provider, (provider_settings->'page_id'))
WHERE provider = 'facebook_messenger'
""")

Post #2

wojtekmach

Hex Core Team

Hey @AndrewDryga, the migration is very helpful, thanks. The DB design looks good.

What do you think about validating provider settings with schemaless changesets and copying the errors to the parent? This would be similar to how constraint validations are handled, they’re used in the parent changeset and end up in parent changeset errors.

Post #3

AndrewDryga

@wojtekmach this is definitely possible, we do as you said: validate dynamic embed with changeset (it’s not schemaless but it doesn’t matter) and put errors to the parent if any. But now we also need to copy constraints and in the view layer add a hack that would map constraint error to look like it occurred in the structure from provider_settings embed.

Mapping is required because we want error for a client to appear where it’s logically should be and point to a correct field, in case front-end maps that errors back. Correct me if I’m wrong, but changeset struct after constraint violation would point to a field in the embed, not to field in the parent struct.

The question is should we do something and make Ecto support dynamic embeds without a lot of hacking and mapping everything back and forth? Because resulting code is pretty complex, duplicated and error prone.

Post #4

blatyo

Conduit Core Team

The core team tends to prefer building an extendable core and allowing the community to provide extensions. Is there something here that might prevent a library and require this to be in Ecto? What would Ecto support for dynamic embeds look like?

Post #5

AndrewDryga

Unfortunately, I don’t know a way to write a library that would change the fact that you can’t use Ecto.Changeset you built by yourself with Ecto.Repo operations. If you have ideas - I’m all ears. Maybe provide your own Repo implementation, but for a library it would be very hard to keep it up to date.

To support dynamic embeds (as far as I know):

We should allow to use pretty much any Ecto.Changeset on embedded schemas (but I’m not sure how the syntax would look like there; syntax may be not required if we allow the lib to override Changeset type information and it would be actually used).
(maybe) We should support constraints on embedded schemas, or at least on dynamic embeds.
Ecto.Repo should thread dynamic embeds like any other embeds and in case of errors return them in proper structure (errors occurred in embed should be in embedded changeset).

Post #6

drapermd

@AndrewDryga Do you have an open source tree that you can share to solve this problem?

Post #7

AndrewDryga

We have code that we use internally but noting ready for open source yet. Without Ecto support it’s just hacks.

Post #8

Adzz

Forgive me if I’m not understanding the problem correctly, but can you solve this problem with a custom ecto type?

Essentially the custom type decides how to build which struct based on the shape of the params it gets.

Post #9

AndrewDryga

Currently, this is not possible because a type implementation only has access to data inside one field, but our use case is when type is actually a separate field in a schema. If we can, somehow, make type to know about other field values - it would work.

Post #10

Load more posts (14 remaining)

Where Next?

View thread on forum (has 24 replies!)

ecto

discussion

Home Chat & Discussions>Discussions

8 5535 24

Last post

REPLY VIEW THREAD WITH 24 POSTS

AndrewDryga

First 10 of 24 Posts

wojtekmach

AndrewDryga

wojtekmach

AndrewDryga

blatyo

AndrewDryga

drapermd

AndrewDryga

Adzz

AndrewDryga

Where Next?

Trending in Discussions

What Elixir related stuff are you doing?

Phoenix `phx.new` generator and AGENTS.md

What took you way too long to figure out?

General AI Chat

Elixir-lang.org redesign

Stack Overflow Developer Survey 2026

Now that fly are pivoting, should I trust them for hosting?

Other Trending Topics

Dexter - A fast, full-featured Elixir LSP optimized for large codebases

Beam Bots - Resilient Robotics on the BEAM

Emerge & Solve - a GUI framework for Elixir

Emily - A new MLX-based backend for Nx

Programming Nerves (self-published)

OeditusCredo - a pack of supplemental Credo checks

Chat & Discussions>Discussions

Latest on Elixir Forum

Sponsor Spotlight

Our Sponsors

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

We're in Beta

Options