Depends on what you expect it to do. I still declares relationships e.g. to preload them, it still does all the ecto type stuff for converting between various representations of the types values, it still comes with all the reflection around schemas, …
LostKobrakai
katafrakt
Thanks for this post. It touches on a lot of things that have been on my mind for last few years, working on larger Phoenix applications in larger teams. Quite often it felt like following Ecto “defaults” is limiting and leads to database leaking everywhere. At the same time the community does not seem to have an alternative proposal.
I completely agree that “designing with persistence” is problematic. My perhaps favourite example is that we had a database table representing a Thing. Part of the data in this Thing was updated quite seldom in a long-running transactions, another part was updated frequently and pretty much atomically (think: views count on an article). The problem: when the long transaction was running, it locked the row and those “quick updates” kept piling up, ultimately timing out, flooding our error reporting and leading to data loss. The solution was to split the table into two tables - now slow transaction locked only part of data.
I’m sharing this example because I think it’s pretty convincing one for the fact, that storage has its quirks and ways of work that enforces some decisions there that make no sense from domain logic level. There two tables were modeling a single Thing, always queried together. But of course there were two Ecto schemas and preloading everywhere…
Custom mapping
I admit that in one project after long discussions we tried this approach. But it did not work well. Especially at the beginning it was seen as just an unnecessary boilerplate. And even worse, we decided to have Schemas.Order and Structs.Order, so every time you saw Order in the code, you had to check which one is aliased.
There was probably a lot of poor judgement and bad design involved, but this turned out to not really be beneficial.
So what else?
Two things come to mind as alternative approaches:
-
CQRS, not necessarily paired with event sourcing, to separate read modeling from write modeling. Unfortunately that it a hard sell for many teams, because the concept was paired with other concepts too often and people think of it as a package with Kafka, eventual consistency, poor transactionality etc.
-
Embrace repositories. Let schemas be just a thin wrapper over database tables. But all meaningful data reading and writing should go through repository modules. This leaves schemas as purely application layer. Repositories sit on the border between application and domain layer, accepting and emitting domain structs, internally working with Ecto schemas. Unfortunately you have to probably give up on Ecto changesets for that or limit their usage only for type coercing, which again makes it a hard sell. Validation in that case should probably be done with something like Drops or with schemaless changesets.
The second approach is also a bit difficult to explain to people because of Ecto.Repo module. I’ve heard sometimes that “we already have a repository, what are you talking about?”. And on top of that, repositories are not so easy to design and I haven’t found a lot of good resources about that.
Anyway, many thanks for starting this discussion here. A lot of great replies already. Perhaps if we have more of this kind of conversation, we’ll come up with some ideas.
7
Sorc96
Thinking about this now, you’re right. Schemas still keep most of their functionality. So I guess the only issue is that a schema like this does not quite feel like a first class citizen. Instead of Repo.all(Schema), the code would now need to look like Repo.all(Schema.base_query()), which may not actually be a real problem. As already mentioned, the insert, update and probably delete story would be more complicated anyway.
Thanks for your replies!
0
Sorc96
I have very similar experience. Mostly with Ruby on Rails, where Active Record really ends up dictating the design of everything, but I have noticed similar issues with Elixir and Ecto. Other frameworks in other languages seem to have the same problem.
I don’t have experience with CQRS, but I’m certainly interested in trying it to see how I like the result. If you know of a good learning resource for it (without the extra parts you mentioned), I’d love to take a look.
As for repositories, I think those would fit the custom mapping proposal, unless I’m mistaken? I don’t even necessarily mind the boilerplate (although convincing others may not be easy), but my main worry is the fact that the design of the repositories is arbitrary. They will undoubtedly end up designed inconsistently and change throughout the project’s lifetime.
So I’m still hopeful for better mapping capabilities as part of Ecto, which would allow us to keep the nice features like changesets and avoid custom mapping, while offering good enough freedom for domain design.
0
LostKobrakai
Yeah, it really isn’t. Schema in that place is just one of a few Ecto.Queryable implementations. Under the hood that’s basically turned into a query as well. If you want to be really sneaky you could even make __schema__/1 overridable and change the implemenation for __schema__(:query).
1
katafrakt
Yes and no. In your example and in the attempt I made before the mapping was still 1:1 between Ecto schema and a “domain struct”. While you can roll up few columns into nicer structure (as in edition in your example), you are still limited by your database design.
In my understanding repositories lift this limitation a bit, allowing to build domain structs from multiple schemas. Let’s say you have multilingual e-commerce.
defmodule App.Catalog.Repository do
def get_product(id, language) do
case Repo.get(ProductSchema, id) do
nil ->
nil
product ->
description = Repo.get_by(ProductDescriptionSchema, product_id: id, language: language)
price = Repo.one(from p in ProductPriceSchema, where: p.product_id == ^id and p.valid_since < DateTime.now(), order_by: [desc: p.valid_since]))
%App.Catalog.Product{name: product.name, price: price.value, description: description.body)
end
end
end
And in the App.Orders context you would probably query for the price valid at the moment when the order was placed instead of current price + you probably don’t care about the description at all. The App.Orders.Product struct will look a bit different.
I don’t, unfortunately, especially not for Elixir (maybe something for .NET, where I think it’s most popular). Maybe I’ll draft a quick example myself in the upcoming days, if time allows.
1
joaoevangelista
I’m getting back into Elixir, and after some time in the OO world, I’m shooting myself in the foot trying to model data, instead we should model functions. Pretty data won’t solve any problems. That being said, ORM is a tempting path to follow that will bite you when you try to do things differently, either by lack of support from the database or the ORM. They are good when the domain can be mapped into the database, the schema can be rendered, changed, and then saved without worrying about SQL and everything becomes easy.
About Ecto, the bad thing is that it is a multi-faceted library, the early split of the SQL module helped. You don’t need to use the SQL module and Repo functions if your database can’t be mapped in a way that makes sense. Ecto can sit at the edges of your application, validating incoming data, both from the web and from the database (yes the database, where there is always a null field that shouldn’t be and it wreaks the application). You can write your queries and give them to Ecto to execute/sanitize them. That is especially useful when dealing with a legacy database or a database first design, you get to load only the data you need to that given case, and you don’t subject yourself to mapping 30 joins on the schema. You can use Repo.load/2 to fill your schema for that particular query result (but I don’t think it will validate the data). You can wrap it all with a DAO Data Access Module, which is a repository from DDD but you don’t get into the argument “we already have a Repo”.
TLDR: Functions over Data. ORM sucks, write SQL
2
Sorc96
I’m a big fan of overriding the default query, although doing it this way feels way too hacky and unwieldy. I would love this becoming a callback on a potential Ecto.Schema behaviour. Then the documentation could show that overriding the query is possible and even expected in non-trivial situations.
You have shown that it is easy to use shcemas with custom queries already, but I think this could be the small step that makes the feature really feel intentional. Especially for refactoring the database design, the application code would only need to change in this one place.
0
lithium
I kind of disagree here, some JOINs don’t bother me that much. If they do, you can hide them behind a database view (which still makes inserts slightly more annoying and breaks the “code first” strategy somewhat) and/or have your database mapping layer return a neat denormalized list of maps/tuples/your-preference (like you did later in your post) and the rest of your application can happily ignore the relational details. In my experience unless your domain is changing constantly (which I have experienced, total nightmare) the data mapping/whatever layer won’t need to change that much if you’ve done a good job modelling your problem in the database.
100% agree with you there. That table design would get your pull request (gently) rejected at my job.
That is painful. I agree that it’s easier to just spit out some flat denormalized tables but it’s very rarely worth it. I don’t think it’s that hard to do proper normalization once you’ve done it a few times, even if you give up some ORM magic. Especially with LLMs around to type out boilerplate for you. I genuinely think you’d be better off having tables with an id and json column and treating them as document stores rather than a halfway designed relational schema that will contain invalid data anyway. There’s also Ecto embedded schemas if you want to have sub-entities in your code but keep everything denormalized into a single table in the database. IIRC you can add constraints that test json values in postgres too, but I don’t remember exactly how much they can do.
This is kind of why I don’t use them much on my personal projects. Or rather, I don’t use many of their complex/magical features. I view application domain objects and the database as FUNDAMENTALLY different things that will likely need manual mapping. The application domain objects exist to be used in application code, freely passed around, transformed, and deconstructed as needed. The relational database primarily (it facilitates efficient querying too of course, but so do NoSQL databases) exists to provide a repository where only correct, consistent data can live. You may be able to generate a Book struct without an ID, but if you try to send that data to the database it should reject it. It should also reject any entities that try to reference a book with an ID that doesn’t exist, for example, so it does more than just validate the shape of a single struct. I don’t think there can ever be a great automagical mapper between domain objects and relational table for the same reason you can’t blindly cast(not the right word but whatever) JSON from an HTTP request to domain objects: They are distinct entities that model the underlying reality in fundamentally different ways so that we can make different trade-offs.
I think Ecto is already a pretty good ORM. Granted, I’ve only really worked with .NET EF Core professionally, and don’t use many of its bells and whistles, but so far I have not missed anything while usng Ecto. I really love how you can insert SQL queries/query fragments everyhere too. My only complaint so far is that it doesn’t have out of the box 1st class support from some postgres features that I like, like uuidv7s or (IIRC) window functions, but that’s very minor and I understand why the team chose to make Ecto generic rather than a dedicated postgres API. Also obligatory shout-out to the Ecto developers for doing wonderful work with only a fraction of the resources that entities like Microsoft have.
Overall I do not think there is a clean, easy solution to the graph/relational mismatch (I like your rephrasing a lot, I will steal it if you don’t mind). Although sometimes I wonder what we could achieve by embedding postgres or SQLite directly into the runtime like Mnesia…
And please forgive me for anything I’ve said that is myopic or ignorant: I’ve written more SQL than anything else at this point (so I’m biased towards it) and I’ve only been a developer for 5ish years. I am also a C#/SQL Server developer at my dayjob and still learning the Elixir ecosystem.
5
lithium
This is what I heavily prefer. The database is not part of your application, it is an external system that you’re probably talking to over HTTP. Letting the database leak into your application will inevitably cause problems as it grows.*
*For those rare systems that don’t change much after the first-pass implementation, like a hacky proof of concept that miraculously manages to stay out of production, it doesn’t matter as much.
1