Design discussion - Ecto is your application

Sorc96

Hello, everyone! This should probably be a blog post, but I don’t have a blog, so here we go My thoughts and frustrations regarding domain modeling in the face of persistence, distilled into a few paragraphs. I’m interested in your opinions.

This is probably going to get a bit philosophical, but hopefully also stay practical. I’d like to talk about the age old problem of ORMs. And when it comes to the impedance mismatch problem, Ecto is in the exact same situation as any other ORM in other technologies. For that reason, I’m going to refer to Ecto as an ORM in this post.

The impedance mismatch

The so-called object-relational impedance mismatch is a misnomer (like so many things in computing). The same problem arises without objects, because the mismatch really exists between graphs and relations.

Graphs are incredibly useful for adding meaning to the data, allowing us to work effectively in a given context. On the other hand, the relational representation is removed from any context, which makes it a great choice for storing data. That way, we can easily add new behaviour even if it needs to shape the data very differently.

But this versatility is both a blessing and a curse. It is a Jack of all trades, but a master of none. Therefore, we either accept significantly worse design of the application code, or end up mapping the relational representation into graphs based on our current use case.

A practical example

I’m going to show a simple example to illustrate the problem of designing code in fear of persistence. The idea is inspired by a video about DDD I watched recently.

Let’s say we need to work with books that have an identifier, a title and an edition. We can ignore everything else to focus on the actual issue. An edition can either be ordinal, identified with a number, or seasonal, identified by a season and year.

Design without persistence

In a world ignorant of persistence, where we can use the full power of idiomatic Elixir, the given description could easily translate into something like this:

defmodule Bookstore.Book do
  defstruct [:id, :title, :edition]
end

defmodule Bookstore.Edition do
  @seasons [:spring, :summer, :autumn, :winter]

  def ordinal(n) when is_integer(n), do: {:ordinal, n}

  def seasonal(season, year) when season in @seasons and is_integer(year) do
    {:seasonal, season, year}
  end
end

The code clearly explains what is going on with the two types of editions, each type is unambiguously identified and contains only the relevant data. Great!

Design constrained by persistence

When tasked to implement this requirement in a real world application, the design thought process would likely be very different, though. We would most likely start by creating an Ecto migration, because after all, everything needs to conform to the database.

Since relational databases aren’t known for their excellent support of sum types, the migration would probably end up looking similar to this:

defmodule Bookstore.Repo.Migrations.CreateBooks do
  use Ecto.Migration

  def change do
    create table(:books) do
      add :title, :string, null: false
      add :edition_type, :integer, null: false
      add :edition_number, :integer
      add :edition_season, :string
      add :edition_year, :integer
    end
  end
end

Inevitably, there is going to be an Ecto.Schema coresponding to the database table.

The table could hold all kinds of invalid data, so we would of course attempt to contain the mess at the application level. We would use changeset validations and hopefully define an enum for edition_type and edition_season.

This would however result in all our application code knowing about these different fields, carefully checking the type and knowing which other fields are relevant based on the type.

Of course, I’m being generous. In many real world applications, there would be no enum for edition_type, the column probably wouldn’t exist at all. Instead, all the code would check which of the other fields are nil and dispatch logic based on that.

Even worse, it’s possible that somebody smart enough to do this, but not yet wise enough not to do this, would reuse the same column for edition_number and edition_year, since they both map to integers.

At this point, there is no easy way to use the data correctly and no intuitive way to understand what the invariants even are, because the code does not contain that information.

The worst of both worlds?

Let’s face it, this design with a bunch of nullable columns is both terrible application design and suboptimal database design. Yet, it is the design I see every day in the projects I work on. I think that’s because the tools we have make it the only easy option.

Improving the database design would involve normalizing the data and splitting it into multiple tables. Just imagine the nightmare of all those JOINs and Ecto.Schema associations we would need in our application. That is clearly not worth the extra complexity.

On the other hand, we could decide that the application design is the only thing that matters and simply serialize the edition as JSON. This would allow us to have the design we wanted with just a custom Ecto.Type. But we would be giving up on so many features of the database.

Custom mapping?

I have to admit that I am now entering a territory that I have not yet explored in a serious project, so there will be some speculation.

Considering the significant impact this trivial requirement has on the application design, it seems that custom mapping may be the best answer to anything beyond basic CRUD. In the case of books and editions, the mapping could look like this:

defmodule Bookstore.Ecto.Mappers.Books do
  alias Bookstore.Ecto.Schemas.Book, as: BookSchema
  alias Bookstore.Book
  alias Bookstore.Edition

  def to_domain(%BookSchema{} = data) do
    edition =
      case data.edition_type do
        :ordinal -> Edition.ordinal(data.number)
        :seasonal -> Edition.seasonal(data.season, data.year)
      end

    %Book{id: data.id, title: data.title, edition: edition}
  end

  def from_domain(%Book{} = book) do
    data = %BookSchema{id: book.id, title: book.title}

    case book.edition do
      {:ordinal, number} ->
        %{data | edition_type: :ordinal, edition_number: number}

      {:seasonal, season, year} ->
        %{data | edition_type: :seasonal, edition_season: season, edition_year: year}
    end
  end
end

Now we can design the application exactly how we want and choose any storage implementation we decide appropriate. The only thing that will need to change is the mapper. Of course, normalizing into multiple tables would require a larger change of the mapper, but the domain model would still stay the same.

Unfortunately, this approach has downsides as well. We need to write the mapping code on our own, but what’s worse, we lose important features of Ecto. Change tracking is now gone and we will need to perform even more mapping for data that comes from the outside, duplicating many of the fields.

Perhaps most importantly, enforcing these mappers and keeping their design consistent is going to be difficult and require some discipline from everyone involved in the codebase. After all, it is easier to follow design decisions set by a framework.

A possible compromise?

Mapping everything on our own is clearly a difficult task. Moreover, we are throwing away more than we would like. After all, what’s the point of ORMs if we need to do the mapping ourselves anyway? This is their job!

Using Ecto.Schemas as our domain data structures may be a good trade-off. But we need a way to model the domain without conforming everything to the database design. This includes nested data, sum types, mapping multiple columns into one field and probably a way to build one schema from multiple tables. Maybe then Ecto could be “good enough” as an ORM.

It’s entirely possible that this was of thinking leads directly into the trap described in The Vietnam of computer science. I may simply too inexperienced to see that. Maybe it’s not worth it to add all this extra complexity to Ecto. But in that case, custom mapping seems like the only option left.

Conclusion

Just like many other framworks and ORMs, Phoenix and Ecto present a devil’s bargain. As long as we are building a web interface for a database, where one form field maps to one column and the application does not need to do anything complicated with the data, everything is simple. But anything beyond that quickly starts to hurt.

Custom mapping may be a lot of work, but in order for Ecto to be good enough as an ORM, I’m afraid it would need to evolve way beyond what it is now. Assuming that an ORM can actually be good enough.

In the end, if we want to keep all the nice benefits that Ecto provides, the following will always remain true. Phoenix may not be your application, but Ecto is.

38 comments

#ecto #design

13 2343 38

2024-11-21 21:10:05 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

zachallaun

Interesting observations and certainly worth thinking about!

While I agree with you that the naive/“obvious” schema leaves a fair amount to be desired, I’m not sure that I agree with your final conclusion. In particular:

Could you clarify which specific features you’re referring to here? Assuming Postgres, you’ll find pretty rich JSON indexing and querying support.

Post #1

Sorc96

Thanks for reading! The JSON approach would surely be a good enough choice in many situations. It’s true that Postgres has great support for JSON.

We are still giving up something, though. The application is now completely responsible for ensuring that the data is in a valid shape, and as far as I know, constraints like foreign keys cannot be used on JSON fields.

I can’t shake the feeling that this is abusing the JSON type, but it could very well be a sufficiently practical implementation. On the other hand, now our application design is forcing the design of the database, since I would consider using JSON only because there is no simple way to map multiple columns to one Ecto.Schema field.

Post #2

al2o3cr

IMO this is fundamentally repeating the mapping that Ecto’s already doing to turn the list-of-lists response from the database into BookSchema structs.

It also sounds like you’re looking for a feature like ActiveRecord’s composed_of.

If I was building the system you’re describing, I’d reach for polymorphic_embed and have Bookstore.Edition.Seasonal and Bookstore.Edition.Ordinal embedded schemas. That’s just as straightforward to pattern-match as a tagged tuple, and can participate in things like protocols.

Post #3

Sorc96

Thanks for your input! I agree, repeating the mapping isn’t great.

ActiveRecord and composed_of is exactly one of the things I thought of. I guess something like that could be achieved with a virtual field. Although I’m not sure how to feel about the data being duplicated in the schema. Ideally, the raw values would not be exposed so code in other places cannot depend on them.

I have seen polymorphic_embed before and I really like the idea. In this case, it would be another option when choosing JSON as the underlying representation.

EDIT: There is actually a proposition for something like composed_of, but it doesn’t seem active anymore. Proposal: Derived fields by greg-rychlewski · Pull Request #4261 · elixir-ecto/ecto · GitHub

Post #4

sodapopcan

This is a nice write up. I very much do view Ecto as my application insofar as I generally push a non-trivial amount of business logic to the database. Mostly calculations and aggregations. I do this purely in Ecto, ie, no stored procedures or triggers.

Otherwise this write-up, especially your examples, has me thinking even more about designing functions-first, or verbs-first, or is it just “functional design”? I’m currently involved in yet another domain modeling exercise where all the database tables are being designed upfront. I have a nagging feeling that a classic mistake is being made for the N-millionth time. Does anyone have any goto resources they like for not falling into this trap? I remember watching a talk a few years ago which I’m having trouble finding now. Am I even making sense??

Post #5

ErikNaslund

Polymorphic embed looks cool, thanks for sharing! If I get it correctly it seems like it’s a nice way to handle your embedded schemas, and they’ll end up being persisted as JSON(B) or something in the database? Seems cool and I can definitely see a few use cases.

JSON fields in your DB

I’ve walked the path of using too many JSON fields in my tables before. I’d say the main downsides IMO are:

You’re putting more responsibility on the application logic to make sure the data is sane. The database won’t help you cascading deletes, updates and such like it does with a traditional relational design. You can use stuff like CHECK constraints with JSON functions to make sure the data doesn’t get “out of shape”, but it’s very easy to forget to add the right ones.
Data migrations get scarier, and bugs tend to pop up after a few years. Even though there is an embedded schema in the Elixir code this too will evolve over time. Even when writing JSON migrations it’s easy to mess up and get data in an unexpected state.
You’ll need some new GiN/GiST indexes for good JSON query performance. If it’s a high throughput table adding extra indexes might have a significant impact. Most, but not all queries, have good performance done over indexed JSON data.
The syntax for querying with JSON functions is there, but it’s not nearly as nice as classical SQL/Ecto imo.

I hope I don’t come across as being an absolutist against storing JSON in a relational database. There can certainly be a few cases where it makes sense. I just tend to avoid it as a default nowadays, unless I have a good reason to go for it. If you do use a JSON field, do add a “version” key to it though. Helps a lot when it comes to data migrations and backwards compatibility.

One of the few cases where JSON fields makes sense to me is where the data that’s stored is mostly treated like an “opaque blob” on the server side code. Imagine having a client side rich text editor which stores its content as JSON. The Elixir code would probably never really care what the content was, as long as it was valid JSON. The advantage compared to storing it as an actual blob is that you could easily do stuff like ad-hoc queries on the data using the JSON functions if you need to in the future. In that case you might even extract some part of the data from the JSON into a separate column.

Database table inheritance

Databases like Postgres actually do support table inheritance if you want to. It’s definitely not a solution for every kind of problem, but I’ve had it come in handy every now and then.

Python model ineritance and polymorphism

I’ve done Python for 10+ years before moving over to Elixir, and two prominent “solutions” to this problem there are Django model inheritance and SQLAlchemy polymorphic tables.

In a nutshell they abstract away the issue that you have different variations of the “edition” in your case. They do this by creating one or more extra DB tables under the hood.

Will you get more JOINs and will it complicate your life? A little bit, but not as much as you’d think. Those two options are pretty solid abstractions, and they do a LOT of the heavy lifting for you. I’m not saying that such a solution is a silver bullet, but I wouldn’t rule it out without trying it for your use case.

Do I know the best way to do the same in Elixir - no

Ash is kinda cool

Since you mentioned you’re into DDD-stuff I think you should check out Ash if you haven’t already. It has some really nice ideas, and its goal is to allow you to “model your domain, derive the rest”.

I’ve only played around with it a tiny bit. It was a bit mind-bending at first but there’s a lot of pretty clever stuff in there that I like. My next toy project will most likely use Ash to learn it a bit better.

Post #6

Sorc96

Thanks for your response, I’m glad I’m not the only one with these thoughts.

I definitely agree about designing the functions/behaviour first and just like you, when I hear people start talking about the database design first, I feel like we’re heading into a trap but don’t know how to change direction.

Post #7

Sorc96

Thanks for such a thorough reply. I certainly agree that using JSON columns for data with a known shape is a bit weird for all the reasons you mentioned.

I mostly have experience with Ruby and Rails, where polymorphic associations exist exist, although they are mostly managed on the application level.

I know that either option is usable, but neither seems optimal. We can have the application design we want at the cost of giving up on some features of the database, or have an appropriate database design at the cost of more application level complexity. The question is how much can this be improved with more features in Ecto?

I haven’t yet found an opportunity to play around with Ash, but it definitely looks very interesting.

Post #8

dimitarvp

I don’t disagree with yours and @sodapopcan’s premise but it also has to be said that usually code follows data i.e. the most important thing you will do is to design the data (well or not). Throughout my long career I’ve spotted that a lot of deficient code has been written just to accommodate bad data design.

Obviously this doesn’t mean an all-hands meetings where every minute schema detail is discussed. But it can’t be brushed away as not important either.

This also ties to the JSON columns: they are a good escape hatch so you don’t have to use a NoSQL DB or a plain KV store. Shove semi-unstructured data in there, write meticulous code to make use of them, observe and watch and when certain common usage patterns emerge, start plucking data out of the JSON columns into RDBMS columns proper.

Not easy by any stretch but I haven’t found a better way.

Post #9

krisleech

If you don’t need to query the schemaless data you could also dump it in to a TEXT field as an erlang binary:

  def serialize(term) do
    term
    |> :erlang.term_to_binary()
    |> Base.encode64()
  end

  # Plug.Crypto.non_executable_binary_to_term() is used as events may contain user input
  # and deserializing executable code, such as functions, is unsafe.
  #
  # For more information see: https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/serialisation.html
  #
  def deserialize(binary) do
    binary
    |> Base.decode64!()
    |> Plug.Crypto.non_executable_binary_to_term()

Unlike JSON you’ll get the exact struct (or other term) back, e.g. dates will be dates, not strings)

Post #10

Load more posts (28 remaining)

Where Next?

View thread on forum (has 38 replies!)

ecto

design

Home Chat & Discussions>Discussions

97 2156 38

Last post

REPLY VIEW THREAD WITH 38 POSTS

Design discussion - Ecto is your application

Sorc96

The impedance mismatch

A practical example

Design without persistence

Design constrained by persistence

The worst of both worlds?

Custom mapping?

A possible compromise?

Conclusion

Showing Posts 1 to 10 Show Best PostsShow All (oldest first)Show All (newest first)

zachallaun

Sorc96

al2o3cr

Sorc96

sodapopcan

ErikNaslund

JSON fields in your DB

Database table inheritance

Python model ineritance and polymorphism

Ash is kinda cool

Sorc96

Sorc96

dimitarvp

krisleech

Where Next?

Trending in Discussions

What Elixir related stuff are you doing?

Phoenix Storybook: a storybook-like UI interface for Phoenix components

Cure - a new language compiled to BEAM

Ergo - a framework for creating microservices using technologies and design patterns of Erlang/OTP in Golang

Security advisory: Decimal DoS vulnerability

Companies that hire candidates without prior Elixir experience

What IDE/Editor do you use for Elixir?

Other Trending Topics

Gust - A task orchestration system built in Elixir

Beam Bots - Resilient Robotics on the BEAM

Xamal - Deploy Elixir releases to bare metal servers via SSH

Corex - Accessible and unstyled UI Phoenix components

How much do you still code yourself?

Aludel - LLM prompt evaluation workbench

Chat & Discussions>Discussions

Latest on Elixir Forum

Sponsor Spotlight

Our Sponsors

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

We're in Beta

Options

Thread Display Mode

Thread Preview

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)