I’m currently involved in yet another domain modeling exercise where all the database tables are being designed upfront. I have a nagging feeling that a classic mistake is being made for the N-millionth time. Does anyone have any goto resources they like for not falling into this trap? I remember watching a talk a few years ago which I’m having trouble finding now. Am I even making sense??
You can try Event Storming, it will focus you on the domain not the the data.
Your comment made me realize something interesting. I think we’re encountering the same issue as the object-relational (or graph-relational) mismatch. When designing the database first, we end up conforming not just the application code to it (which can be acceptable to a certain degree), but also the way our software can be used easily.
And the users won’t be happy when using the app the way they want is awkward because of the decisions we made about the storage mechanism. A classic example would be when data can be entered into the system partially, like information about products being sold. But of course, in order for customers to buy the product, all of the information needs to be in the system.
When thinking about the process first, we would most likely realize the need to strictly separate the work-in-progress products from ones that can be sold. But in practice, there’s probably just one table for products with a column which marks that the product can be sold. Inevitably, products with incomplete information end up being sold, creating problems all over the place.
I’ve actually done example mapping before! It was on a project with a very small model, though, and the primarily domain modelling had already been done. It was a very UI-heavy app (a gantt chart) and I think there were a total of maybe 8 business models? I feel like it was less but this was a few years ago. Everything had already been modeled and there were far more changes to verbs than nouns in general.
I’m not arguing that data isn’t the most important thing but that modeling it based on how it’s stored is not ideal. Using behaviour to inform its design is (likely) better. This is what I understood @Sorc96’s premise to be. I think I agree but I just don’t have the experience of knowledge to back that up which is why I asked if anyone had any really good resources. I assume it’s limited as I don’t know of a web framework that doesn’t push you to model data based on its storage and it seems like one of those things that most people are going to say “It’s what everyone already knows and is good enough.”
I didn’t respond to this earlier but having composed_of in Ecto would be great! A friend of mine actually did a really good 2018 Rails Conf talk that includes composed_of. I feel a little dirty slipping a plug in here, but it’s a really good talk!
I wonder if this is really true though?
I think there can certainly be certain features of frameworks that make it more or less difficult to design storage agnostic models but at the end of the day there will always be tradeoffs for doing so. ActiveRecord’s philosophy was very clearly to help you forget about the DB and as a novice developer I drank deeply of that particular kool aid (even before NoSQL became a real buzzword) because I found SQL to be a boring, terrible API in comparison to Ruby, and as the runtime env for my application the latter felt much closer to the “life” of my application. But even AR had aspects that clearly were geared to relational dbs (hard to imagine “associations” outside the context of a relational db). And isn’t that because relational dbs have features that, even if they sometimes seem like purely technical, implementation details, are really part of the value you are offering your users? “Persistence” isn’t some weird accidental part of software, it’s a real world need with its own logic that most frameworks want to expose.
So I guess the answer is yes, frameworks do push that, but not in an essentially different way than they push you to validate inputs, etc.
I think this is a good thing honestly. I love that you can write your own queries in Ecto and you don’t have to rely on some higher-level abstractions that ORMs from languages like c#/java offer. The problem with those types of ORMs is that they are opinionated in a lot of the decisions, not flexible enough and not extensible.
I always tend to showcase how easily ecto queries can be extended:
defmacro st_transform(wkt, srid) do
quote do: fragment("ST_Transform(?, ?)", unquote(wkt), unquote(srid))
end
Good luck implementing that with an ORM that tries to abstract everything.
I think ecto has a hard time to be explained because by being a data mapping tool it is great at two edges of the system – the one where data comes in and the one where data goes out (usually for storage). In a lot of cases, where data is shaped to allow for it, it’s very easy to just not think of those as two edges, but just wire data right through. Therefore that’s a lot of what we see in code shared around elixir. Imo this is however just a lazy mans shortcut, fair to be used where possible, but which needs to be acknowledged as not the way forward where it doesn’t anymore. If there’s a need for richer domain modeling, which deviates from the constraints of storing the data, then that’s imo no longer ectos job, even if ecto might be part of implementing that. That’s one of the places where ecto hard deviates from what ORMs commonly do. Ecto wants to model your data stored in your db by embracing that the data needs to go into a db, instead of finding magic ways to make you do things dbs cannot do well like polymorphic relationships.
If one is fine with the lack of referential integrity of ref_id, ref_type columns then that can be built with ecto. Others might go with abstract tables polymorphism, which retains referential integrity. Or you might not need to have foreign keys, so you opt for polymorphic data in a json column. One might even use all three options depending on their distinct tradeoffs.
I have the same experience. It’s probably because of how easy it is to get started wit this approach. Most people, especially less experienced devs, don’t want to start thinking about mapping between UI, domain and persistence layers. Just define one model/schema and have it go from the html form all the way to the database and you’re done.
But as the application becomes more and more complex, this approach starts to take its toll. Suddenly, domain logic is doing way too many checks for underlying details of the persistence layout, database tables are designed suboptimally to avoid even more application level complexity, and custom mapping starts happening all over the place because the UI doesn’t match the database 1:1.
I’m not sure if I will ever be truly happy with any solution, but I’m currently at a point where I think more ORM features (like composed_of) could help a lot.
I just realized the same thing after writing the initial post. Ecto has amazing capabilities for working with the database and changesets work well for validating the data that comes in from the outside. But having these two concerns (and also domain logic) align is a happy accident that cannot be relied upon.
I think the weakest part is probably Ecto.Schema. It provides enough features that I don’t want to work without it, while also constraining design significantly. In order to make schemas work better without custom mapping to other data structures, they would need to support more mapping features like composed_of mentioned above.
I also think schemas would need more versatility in terms of the underlying representation. Maybe a schema should not be tied to a single table. Why not allow any queryable? That way, schemas could automatically be limited to certain rows by a where clause or combine data from multiple tables using join. This would allow schemas to be used with the full power of Ecto.Query. However, I am aware that this could also make insert/update impossible, but I still think the benefits to read operations would be worth it.
EDIT: To make my point clearer, I think the problem is that we have to either use Ecto.Schema and give up on the powerful features of Ecto.Query, or use the full potential of Ecto.Query in out custom mapping functions, but then we’re giving up the benefits of Ecto.Schema.
def schema_from_two_tables do
from a in "a",
join: b in "b",
on: a.id == b.a_id,
select: %Schema{id: a.id, from_b: b.data}
end
Nobody requires you to query based on the table defined on the schema macro. The name provided to the schema table is also never validated to be an actual valid table name. It is only required to be valid if you try to query by it. E.g. {"filtered_people", Person} is a queryable, to query values using a schema from a separately provided table.
So I’d argue you already have all the freedom to query schemas how you like, we’re just having a hard time communicating all of that to people I think. You are correct though that inserts/updates are a lot trickier that way. If you need inserts/updates you can look into updatable views with postgres, but also I’d argue CQS, where you have separate schemas/models for querying and writing is something reasonable to look at.
Is the schema doing anything at this point, though? Would there be any difference with a regular struct?