AndrewDryga
Hey guys. I know this is an old topic, but as I write more and more complex application with Elixir and Ecto I feel like we really need a way to let developers use dynamic embeds.
Few words about our use case, we have a Transport schema which stores various common fields and settings of a transport. But depending on transport type (eg. Twillio and Facebook Messenger) settings can be very different and also there are DB constraints that should be in place for those settings.
We do work around this issue with an application logic which takes params for the embedded schema (which defined as :map type on parent changeset) and validates/casts them if embedded changeset is valid or properly adds errors to the parent otherwise. Here are some code:
A function that shows how dynamic changeset works in our case:
defp cast_provider_settings(changeset, provider_field, provider_settings_field) do
with {:ok, provider} <- fetch_change(changeset, provider_field),
{:ok, settings} <- fetch_change(changeset, provider_settings_field),
provider_settings_changeset = Provider.settings_changeset(provider, settings),
{:ok, valid_settings} <- Validator.fetch_valid_attrs(provider_settings_changeset) do
put_change(changeset, provider_settings_field, valid_settings)
else
:error ->
changeset
{:error, :not_found} ->
changeset
{:error, %{valid?: false} = settings_changeset} ->
put_embedded_error(changeset, provider_settings_field, settings_changeset)
end
end
Here is how you can add an error to an embedded changeset defined as map:
defp put_embedded_error(changeset, embed_field, embedded_changeset) do
embedded_type =
{:embed,
%Ecto.Embedded{
cardinality: :one,
field: embed_field,
on_cast: nil,
on_replace: :raise,
owner: %{},
related: Transport,
unique: true
}}
%{
changeset
| changes: Map.put(changeset.changes, embed_field, embedded_changeset),
types: Map.put(changeset.types, embed_field, embedded_type),
valid?: false
}
end
(Notice that you can’t override types and leave embedded changeset in Ecto Schema where :map type was defined because you would get a cast error. Ecto.Changeset does use pre-compiled type information when insert happens so overriding only helps when you use functions like traverse_errors/2.)
And even if you do that, there is a lot of issues that persist here. The main one right now for us is constraints - they are lost when embedded schema turned into a map and moving them manually to parent doesn’t make sense (error field would point to a wrong direction).
Other ways to hack around:
- Define multiple schemas per database entity (or even combine that with PostgreSQL table inheritance). This one looks weird for me because when I fetch data back from DB I do want to see only one kind of schema. Data that I want to put there should be exactly what I get back.
- Do not use dynamic embeds. This option looks poor because there is sooo many use cases where dynamic embed makes perfect sense.
As a very raw suggestion how we can deal with that:
- We might add a
:changesettype for Ecto.Schema. - It’s application responsibility to actually implement logic how embedded changeset gets there, on which fields it’s resolved, etc. (I don’t think that Ecto needs to add any kind of magic here.)
- Repo operations should take care of changesets in
:changesetfields in a same way as they would do with usual embedded schema.
OR
- Make Ecto use type information from changeset (removing the calls to
Schema.__*__functions) which is not straightforward and would make changesets structs much bigger. (See this issue.)
Where Next?
Trending in Discussions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Elixir, Erlang & messaging conferences from Code Sync. Next up: ElixirConf US (10–11 Sep) & Code BEAM Europe (21–22 Oct), Haarlem + online.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 24 to 15- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
mathieuprog
I suggested to copy the type field, so you end up with the exact same structure in the DB, no difference for querying, i.e. a type and payload fields in the schema, in addition to a type in the payload to enable polymorphism. That small logic for copying the type would be done in the changeset.
Resorting to non documented, private internals such as
Ecto.Embeddedstruct and its fields (thus that might change and break your app) seems worse as a solution. Would you go for a minor weirdness, or a fragile hack?Indeed I don’t know about the limitations you faced back in 2018, as I wasn’t doing Elixir at that time.
I see however that ecto_poly was released end 2017, so there were some possibilities.
AndrewDryga
There are few reasons:
Ecto.Typewas not flexible enough to cover our needs either;typefor us is strictly field even whenpayloadis empty.mathieuprog
Why not just copy the
typefield into the payload? You could do that in the changeset, before calling cast of a custom type.AndrewDryga
Here is our implementation: Dynamic embeds with Ecto · GitHub. It’s used in production and did not cause any issues yet. It’s very hacky though.
Notice that we explicitly call
load_dynamic_embed/2function every time schema is loaded from DB. We also dump everything to string-keyed map if changeset is valid to mimic behaviour of writing and reading from DB (so that you can compareuser == inserted_userin test cases, otherwise string- and atom-keyed maps would be a problem).If embedded changeset has an error it looks very much like nested schema was defined by using embeds_one/many`.
AndrewDryga
Thank you, the library looks nice. It does not fit out needs though as if JSONb column has type information it’s trivial to make it polymorphic without any external libraries (just pattern match on load in specific type).
Our cast is different, in the raw Ecto schema we have a
typefield and thenpayloadfield which should be casted depending ontypefield value. Which Ecto does not allow as you can’t accesstypevalue while casting/dumpin/loadingpayloadfield.thojanssens1
I don’t see why Ecto could be unable to load data.
I tried playing with datetimes, dates, times, decimals, … to be stored in
:map, and all seem to be dumped and loaded back to the right data without issues with Postgres without these encodings/decodings.Also, do you know what other format can be given to
embedded_dump(type, value, format)other than:json?lukaszsamson
thojanssens1
@lukaszsamson two things
I think that the
Ecto.Type.embedded_dumpandEcto.Type.embedded_loadin thoseEnum.reduce_whileare unnecessary, and that represents a lot of code in your Ecto type.Because you know the schema based on the ‘type’ field (
def load(%{@type_field => module_string} = data)), so when loading the data from the DB, you can simply cast those values against the changeset of your schema. And the cast will convert the data to the right elixir data. Or did I miss something?I don’t understand why you say off-topic as the library above for example does exactly what you are doing, i.e. picking dynamically an embedded struct based on some ‘type’ field, and store it in some field.
lukaszsamson
I was working with Postgres so JSONB was there. The problem was ecto mapping a single JSON column to one of several structs depending on some other column. Using a simple map serialized to JSON I would lose all ecto goodness like decimals, dates, times, casting, loading, validatins etc.
thojanssens1
Were you working with a DB that has no JSONB support? Or why did you need to convert to JSON as seen in the code?