ElixirConf: ElixirConf 2023 - German Velasco - Using DDD concepts to create better Phoenix Contexts
Comments welcome! View the elixirconf tag for more ElixirConf talks!
11 Likes
I really enjoyed this presentation. Very inspiring.
One question that popped up: how do you deal with maintaining transactional consistency over different subdomains? Do you pass down an Ecto.Multi to the different subdomains, or do you have another strategy for that?
1 Like
I haven’t seen the presentation yet. I will watch it later.
From your explanation I guess that domains and bounded contexts are used interchangeably in this talk?
I’ll use bounded contexts (BC) in my further explanation.
A BC is a context where you define a ubiquitous language and consistent model.
By that definition you should not have transactions across differenent BCs. Because by definition if you need a transaction to keep things consistent across 2 BCs, you can’t keep the model consistent in 1 BC on it’s own.
That means if you need transactions to keep things consistent across BCs, you probably should think if you need to remodel your BCs, for example that the things that you want to keep in 1 transaction should be part of 1 BC, or maybe remodel so that you have a business process that does first 1 thing and then the second thing.
(This is just regular DDD advice without having seen the talk, I will watch the talk at some point and maybe add some more insights after :-))
4 Likes
Enjoyed the talk 
I enjoyed the talk.
Whist I think the “DDD” approach to using business language and concepts within a subdomain is extremely useful, I have used that approach for at least 30 years, before it was given a fancy name.
It doesn’t seem like DDD is an especially new concept and we have been building consistent database systems for many decades.
What does seem new, (and quite worrisome), is a willingness and appearant advocacy to sacrifice referential integrity.
It does not follow that just because we want to model business sub domains using business language that we simultaneously need to accept inconsistencies and weak referential integrity at the data layer.
Given the choice we should always prefer a system move from one valid global state to another valid global state vs an approach that indulges or promotes inconsistency.
Admittedly I am not a DDD-phile, nor have I seen a reference Elixir DDD implementation approach held up for scrutiny that provides transactional consistency across subdomains.
Is there a good example?
I use the Ash Framework and it appears somewhat aligned to the DDD philosophy but it also supports transactional semantics.
Maybe Ash an “impure” DDD approach? I don’t know, but lack of transactional semantics across sub domains seems like a stake through the heart for DDD, as we should avoid any approach that promotes weak referential integrity.
My position is that if the paradigm breaks the data layer consistency or requires softening the referential integrity guarantees and other constraints on our data model then it is fundamentally flawed and rotten at the core.
Perhaps we have to limit DDD to just modelling domain concepts, events and actions and that’s as far as we indulge it. We have to use a different implementation approach to actually get the job done in a SANE way that doesn’t corrupt the data layer.
I am by no means a DDD expert but have a bunch of experience with some peoples’ idea of it as well as my own idea of it. The whole notion of DDD is quite flexible. It’s not something like SCRUM where the author says if you aren’t following it to the letter then you aren’t really doing it. The DDD book is very large but it’s peppered with bold paragraphs and the intro claims that it’s fine to only read those paragraphs, learn the concepts that appeal to you, and you’ll be “doing DDD”. I imagine “fancy names for existing concepts” isn’t too far off, especially if you are someone coming from XP (though I don’t know because I wasn’t doing this level of development in the 90s) but I find a lot of that language useful. For example, you could just call an Anti-Corruption Layer what it essentially is: a big ol’ wrapper! But “anti-corruption layer” clearly communicates the intent about why that wrapper is necessary and DDD gives good advice on how to implement and maintain them (especially when it comes to wrapping junk code that’s part of your codebase).
So, my brain is often pretty cluttered and I miss the nuance in these discussions sometimes, but in my experience, database referential integrity is a non-issue. Bounded Contexts are not microservices, so it’s perfectly legit—and I would say ideal—that tables still point to each other across subdomains. In fact, it’s perfectly legit for subdomains to share tables with each subdomain owning different slices of the table—of course any shared columns should probably only have one writer. The intro of this talk says it pretty well with “Stop doing table-driven design” and “you can’t express every entity globally throughout your app.” Ecto makes this feel really natural by not pushing you into a 1:1 relationship of schema-to-table. That 1:1 relationship is pretty ingrained into the soul of most web developers, though.
Is that even what you mean? If I’m way off base then could you share an example of when referential transparency is a problem?
4 Likes
I agree with the idea of thinking beyond the table design and understanding that facets of say a product will mean different things in different contexts.
However in order to have a consistent data model we would like to minimise the possibility that a product is not missig a SKU, a price and a warehouse and a units on hand. We may even enforce this through contraints whether they all be jammed in one table or related tables. It should not be possible to have a product you can include in a shopping cart that we we cannot bill and cannot fulfil.
Yet these are 3 separate sub-domains need to be involved when a new product is listed. Based on what I am getting from DDD approach (and I do hope this is an incorrect assessement) suggests that we isolate the concerns and forgo multi domain transactional consistency and therfore we must either relax our constraints be they referential integrity or column/check constraints and basically allow more nulls. Nulls beget more problems and edge cases.
It seems to me that in principal, getting the benefits of DDD business thinking should not beget a fragmented inconsistent mess of a datalayer unless we take it too far. I can certainly see this is possible with microservices backend as you have to assume basically zero referential integrity across services but I don’t indulge designs that promote these kinds of problems. I have to say that without seeing a good example of DDD in practice it is starting to look awfully similar.
Does anyone have an exemplary example of DDD using Elixir/Ecto that doesn’t weaken datalayer integrity and includes cross domain transactional semantics?
If there isn’t one then I think I’ve reached the conclusion that you should only use DDD to understand the problem domain. For the solution domain go build a sound datamodel and ensure your contexts orchestrate cross domain concerns so that your datamodel goes from one consistent state to the next using a multi transactional approach.
1 Like
I too am interested in an exemplary example! If there are no replies in the next bit I can tell you what I would do, but no guarantees it would meet the exemplary criterion 
1 Like
I’m wanting to be enlightened…
Hoping that someone can point to a material example of good DDD that doesn’t suffer the problems I’ve outlined.
I’m going to try to explain my understanding as best as possible.
-
The context in which you work matters. There’s a difference between a small company with 1 to a few dev teams, vs bigger companies that have many dev teams. From a DDD perspective it doesn’t matter much if you have a monolith or use different services.
You can apply DDD principles/practices in both kinds of architectures.
Often from a team organisation and deployment perspective, a monolith vs multiple services does matter. -
If you have a monolith, keeping referential integretity is not that hard, given that you have a single database.
When you have multiple services, you have to rethink referential integretity anyway. (I make the assumption that a service has a separate database and you don’t want to deal with distributed transactions) -
There is a difference between a (sub)domain and a bounded context. Domains are part of the problem space, they are how the business see themselves, how the business is organised,… Domains often have fuzzy boundaries, fuzzy language, are not super well defined.
Bounded contexts on the other hand are solution space. We (dev) design them. Often they take inspiration from the domains, but they don’t have to perfectly align.
The definition I use for bounded context is: in some context we have a consistent ubiquitous language and consistent model with an explicit boundary around it.
So this means that we design different contexts (with inspiration from the domain) and build a ubuiquitous language and model and clearly state what is inside this BC and what is outside.
The most important thing here IMO is that we design bounded contexts. This means that often we’ll have multiple heuristics to try to define the BC. Some examples of these heuristics are:- things that change for the same reason, belong together,
- things that change for different reasons, don’t belong together,
- look at the language - similar language → together,
- natural domain boundaries,
- look at business value → high value vs low value might be splitted up
- …
One of the heuristics might be keep referential consistency.
As you can imagine, often some of the heuristics conflict - we call these competing heuristics - and thats what make software design hard, we’ll have to make decissions based on tradeoffs. There are no perfect solutions.
- In DDD we try to capture essential complexity and listen to domain experts to design our systems. I’ve seen cases where developers try to add consistency on top of a concept where domain experts say there is no such consistency. In this case developers remove essential domain complexity making some features hard or impossible to implement.
(of course the opposite also happens where there’s no consistency and there should be)
I just want to say that there is no 1 correct design.
Given all this, here is an example from a client that I consulted with that worked with warehouses/products/…
They had a pretty complex domain and software to match (and quite a bit of legacy).
For them adding consistency rule like a product needs a price would be a mistake, the price setting of products was very complicated and often only happened after customers already had products in their basket.
Some products have a price, other prices would be calculated based on the warehouse with most stock, or maybe calculated based on which warehouses we own vs which are partners of us vs how much of these products we already sold from our warehouses…
A very complex price setting.
Do we have an item in stock was also much more complicated than yes/no. Maybe we have it in a warehouse in the same country as the customers, maybe we have it in a different country. Some products are ordered and build on demand,…
All of this is to say that if business requires referential consistency, then please also implement it that way. But if business doens’t require it, find out why they don’t require it and make a decision based on that information.
I’m going to end with one last comment on design. There are a lot of businesses where a product is as simple as just a product. Maybe a Product is in draft DraftProduct, or it’s orderable OrderableProduct or it’s archived ArchivedProduct.
These can be 3 different entities in your software and maybe they even live in different bounded contexts. There’s going to be different consistency rules on a DraftProduct vs OrderableProduct (must be in stock, must have a price,…).
In that case you can have consistency rules for each different product type and guard the process on how these products move from one type to the other.
As for the question on example software, I’m don’t have examples in elixir. And the best examples are things that are closed source from clients. We do teach courses on tactical DDD patterns but not in Elixir (Domain-Driven Design in Typescript)
Hopefully this helps a bit, I can talk a lot more on this, but don’t have the time right now.
7 Likes
Thanks for the comprehensive reply.
I am interested if you have the time to reply if DDD deals at all with cross bounded context transactional concepts.
E.g. a new product is listed may result in a triggering an event to the promotions context to alert users who were looking at similar items. In Elixir this could hypothetically be adding something to an Oban worker queue for batch processing, or immediate upserts to a promotional feeds table and sending a notifications via Phoenix channels to the relevant connected users.
Is this kind of cross domain orchestration “forbidden” or “discouraged” by DDD? I ask because these kind of scenarios are common business requirements, and they are quite possibly occuring in different bounded contexts and the problem needs to be solved.
Does DDD genuinely accommodate or deal with this kind of cross bounded context orchestration (e.g. an action in one context may result in a notification or sending of events to another context)
Can this orchestration when required be achieved under a single transaction (assuming the simple case of a single database) or does that somehow violate a DDD principle that transactions by definition must always be scoped to a bounded context?
I’m quite pragmatic about these kinds of things. And I definitely don’t like the word forbidden. Discouraged is can agree with.
But if something is discouraged we also need to know why it’s discouraged, and not just because someone or some practice says so.
Expanding on the discussion of yesterday
So let’s first look at a simple, imaginary example, but with some inspiration from real world situation that I’ve seen.
Imagine after a discussion with 1 domain expert, we came up with the following design:
(so a product has exactly 1 warehouse)
We implemented this and have a database with some constraint that a product must have a warehouse id.
Now a bit later we go talk to an other domain expert who’s main expertise is adding new products in the system.
We show them this picture and explain what it means
Sidenode on technical diagrams
The above picture is trivial to explain, but often showing technical UML diagrams to domain experts is not the best way to engage with them, because they’re often not familiar with this notation. In DDD we have some other techniques like EventStorming or Domain Story Telling to discover the domain, talk with domain experts, …
An example of the same knowledge but captured with EventStorming might look like this:
(blue = command or action, pink = constraint, orange = event or decision)
This notation is really easy for anyone to understand, we can read it like a sentence: when we add a new product, we check that the product has exactly one warehouse and if that’s true, the product will be added in the system.
Back on track,
So we discuss this with the domain expert and they immediately say, that’s not true. They explain
Often when we need to add new products, we don’t know in which warehouse we’ll store it. For some products we do, but for others we don’t know that yet.
“That’s interesting, but I guess that these products can’t be sold on the website yet.” I reply.
Sometimes these products will be shown on the website. People can’t buy them yet, but they can pre-order them.
The conversation continues for a bit and we go back to our design.
As we’re not doing any DDD and we feel that a refactor of the database constraints would be hard we propose the following solution:
If we don't know the warehouse, we'll add 9999 as warehouse_id
A small conclusion
This is how we start to add accidental complexity in our code base. Sure if this is the only thing that we do, maybe it’s ok.
But if later on the rules change again (we now have products that will never have a warehouse as they’re being sold by 3rd parties - we add 9998 as warehouse_id in that case), you can see that our system can quickly start adding a lot of accidental complexity, because we didn’t refactor to the actual business needs.
Is this forbidden?
No, but I’d discourage a design like that for a couple of reasons.
First, the upside of this design: it’s really easy to add it in at right now.
But, the downsides long term are huge.
These kinds of “hacks” accumulate quickly and the more you add in, the harder they are to refactor out. I’ve seen a lot of legacy systems that have columns with magic values like that. Figuring out what they do as someone new is often hard work (hopefully the meaning of a magic value doesn’t depend on an other magic value in a different column 
)
Even for people who’re experts in the software, this design significantly increases the cognitive load for them, because everywhere they use the warehouse_id they need to take these things into account.
Refactor towards deeper insight
The above design is obviously flawed and it’s a great way to start building systems that after a while no one dares to touch anymore.
In DDD when we encounter such flaws in our model, we want to refactor towards deeper insights. The downside of this is that it takes time, but the upsides are that we now should have a model that aligns much closer with the actual business needs. Thus being able to serve the business better and faster in the future.
Back to the original questions: BC communication
Communication patterns between bounded contexts is something that’s discussed with Bounded Context maps and bounded context patterns. (a great talk on this topic is: https://www.youtube.com/watch?v=k5i4sP9q2Lk)
So in DDD I’d say we don’t forbid or discourage things necessarily, but we try to find out what patterns would work best.
We often try to limit the amount of knowledge 2 bounded contexts need to have about each other (both domain and technical knowledge).
How much knowledge depends on the business requirements (something we don’t have a lot of influence on) and how we design the system (something we can influence hugely).
Bounded context patterns define what kind of coupling relationship there are.
The hardest type of coupling is called a “shared kernel” in DDD, that’s for example 2 bounded context sharing a database, or sharing a domain software module.
It’s often said that this pattern is an anti-pattern in DDD, but it’s still a pattern that’s widely used. For me it’s important to that when you pick a pattern like that you know why you do it, and what the upsides and downsides are.
If you’re 1 team working on 2 bounded contexts that have a shared kernel, you’re not going to have a lot of downsides.
Once you split the 2 bounded contexts across 2 teams, you’ll need a lot of communication between the 2 teams to make sure that you don’t break each others code by changing stuff that impact the shared kernel. For example, you share the database I talked about earlier, if 1 team decides that warehouse_id 9999 means no warehouse, the other team needs to be informed up front about that because who knows what they rely on.
And it’s like that for a lot of choices you make.
Something else to take into account is how fast or slow is your software changing. It’s easy to keep a shared kernel in a slow changing part of the software, but a lot harder if it’s part of your core domain that you’re working on all the time.
And finally there are differences even when talking about a shared kernel. Sharing the full database scheme is obviously tighter coupled than just sharing 1 table for communication.
So, is a shared kernel bad? It depends, and it depends on some of the things I just described.
In your example:
Doing the upsert is an example of the shared kernel. They share a database.
Changing it to an oban worker triggering an api call (or public module function call) would reduce the coupling.
Sending notifications via pubsub, depending on how it’s implemented I would also call that a shared kernel. But promotions could also expose a public interface that you need to call when you want to notify someone, and that reduces the coupling again.
(there’s a lot more nuance, but hopefully you can see what I mean)
Like I just explained, yes it can, we’d call this a shared kernel. It certainly has upsides (like the simplicity of the single transaction), but there are downsides as well. It’s up to you to decide how they balance out.
To conclude. Nothing is forbidden, but some things are discouraged because we know that there are patterns that will make your software hard to maintain and to evolve over time.
If you’re building software as a small team, reducing coupling, splitting into BCs or services, also has a cost. It’s up to you to evaluate what’s important and how to design the software.
7 Likes
Brilliant post!
Thanks very much for such a considered response.
That’s a lot of hard-won experience packed into a forum post! Thanks for sharing it.
I think this distills a key part of the concept and I agree that we should seek to isolate mutual knowledge between BCs thereby reducing coupling.
I see events as a possible way to solve this, as we avoid a sprawling globally shared data model. We instead provide the minimal event data to a bounded context with its minimal data models, it does not know about any other BCs it only knows about its own domain model and the specific events and actions it handles and the events it emits, almost analogous to the isolation and boundary layer of an Elxir/Elrang genserver process.
I am however still unsure how cross business domain communication and coordination is reflected and embodied in BC’s given ideally BCs should be close to 1 to 1 with the business domain but often aren’t due to valid design tradeoffs.
We know that businesses do communicate across their domains though systems so how do we bridge the BC’s without them having knowledge of each other?
There still appears to be a need for some kind of an orchestration or coordinator role that mediates events between bounded contexts, eg as a result of listing a product this may need to trigger processing in other bounded contexts, such as in my example, a promotions BC would receive an event about a new product listing because we listed a product in one BC and some coordinator/orchestrater now must ensure an event or action is delivered to the promotions context.
In which BC should this coordination live?
Does DDD have some recognition of a cross BC coordinator function or is this some kind of “union” context that is used to bridge between two or more BCs?
I just struggle with the idea of bounded contexts being isolated when they never can be because there is always a real business need for business domains to co-ordinate and participate in cross domain workflows. Therefore solution space bounded contexts must also couple in some way. E.g we may use service management and ticketing systems to mediate these cross domain workflows, but hook or by crook every domain must interface with at least one other domain in the business either directly or via some intermediary system.
So it seems the crux is really about asking what is an appropriate coupling because BCs need to couple because business domains couple through workflow coordination whether it’s system supported or manual processes.
Therefore I think we have to ask what coupling approaches in our Elixir programs are minimal, reduce fragility and provide the maximal agility for least reengineering effort.
There’s already a lot of interesting discussion here.
In regards to the initial question of “why does DDD promote the a lack of referencial integrity”:
I think this starts at the wrong end. A more useful exploration would be: Does the business you’re working with have the need for referencial integrity between the contexts in question? Does the real world work with shared data or does information in the real world flow from one set of information to another set of information.
Consider a car company. There’s engineering designing cars and there’s sales selling cars. The “cars” those two departments think about are completely separate things. That’s what bounded contexts are about – separating engineerings “cars” from sales’ “cars”. Sure at some point e.g. an engineering sample might become a car being sold. But that’s generally better understood by “oh this is no longer a car engineering works with, but it’s now a car being sold” – a deletion in the engineering context and a creating in the sales context.
I often feel like people try to apply DDD at a scale where this kind of issue doesn’t come up in the first place. If a webshop can be dealt with in a single db with all the referencial integrity possible that’s fine. That however doesn’t mean that model scales to a company with many different departments all dealing with kind of the same core product, but all interacting with it in vastly different ways, contexts, lifecycles, …
5 Likes
That’s fine but they still need something to relate those different car “types” or facets, perhaps it’s a natural key or perhaps it’s a database key but there needs to be a strong binding to ensure that you are referring to the same car across contexts.
And right there we have a coupling because in reality you need that strong binding to trace a car sold for accounting, stock control, compliance, state vehicle registration, manufacturer warranty, maintenance and product recalls, retail finance and insurance.
The whole thing collapses without the coupling across domains and referential integrity is still essential.
I accept DDD as far as it goes for business expression and understanding the problem domain however when it comes to solution domain and bounded contexts we need to recognise what DDD trying to do vs what it can do with its notions of “good” and “not good”.
The only principle in DDD and one generally understood long before DDD is that we should aim for minimal coupling by ensuring bounded contexts or sub-systems know as little as possible and ideally nothing about each other except where they must.
Well that’s wonderful.
That necessarily begets an integration and orchestration layer that must have the coupling to provide the actual cross domain flow that happens within a business.
Every time I bought a new car the sales guy checks the stock inventory and he’s in the sales domain. Then when I order the car and pay my deposit the fulfillment/delivery/service team need to prepare the car, remove all the packaging that new cars come wrapped in and install any ordered accessories. Their delivery team don’t magically work this out, the sales guy allocates the stock and puts in a work order so the delivery team can prepare the car ready for the customer. The business needs to make damn sure it’s the same car I ordered across all their business functions. These are all cross domain flows that “break” the naive notions of BC “goodness” and coupling.
So I’m left with the conclusion that DDD is not really providing much value in the solution domain, perhaps other than some repackaged awareness of various design considerations and tradeoffs but it’s the same tradeoffs and design challenges we have always had before calling it DDD and it I can’t see that it has actually provided any profound advantage outside of describing the business problem domain.
That’s exactly why I spoke of engineering and sales, not not two departments involved in delivering one and the same physical car. Cars build by engineering (as in creating new models of cars) don’t really share much of anything with a car eventually being build for consumers to be sold. It might even be completely scraped.
Also while I never worked in such a domain from what I know about car sales I’d be surprised if they actually have a single system with referencial integrity over all the things you mentioned. There might be multiple distributed systems sharing common identifiers like serial numbers or manufacturing numbers, but those systems don’t use a single shared atomic and transactional database. They depend much more on compensation actions to handle issues than making sure everything is correct everywhere everytime.
E.g if somehow a car is sold twice they either build another one and change the order of one customer. If that delays delivery they appologise and maybe give you a discount.
Would a single database with referencial integrity be nice to have – sure. The company certainly would like to not need to give out those discounts. But there’s many tradeoffs involved in distributed systems and large enough companies are almost by definition running as distributed systems.
1 Like
I didn’t imply there was a single database but that a strong binding must exist between domains. Where things are in a single database we absolutely should use referential integrity and where we have disparate systems then we may use various auditing processes and data pipelines to aggregate data to validate and detect data quality issues and other anomalies.
A car dealership is hardly involved in manufacturing and engineering the vehicles they sell but they certainly may use multiple systems vs a single integrated system to run their business.
But so what, we are asking how does DDD help us build an Elxir application within a project scope, not solve every business problem.
When we are building a typcial elixir or phoenix app with a database I don’t see that DDD has a place beyond what I’ve already stated, it can help to understand the business problem domain using business language.
Bounded contexts don’t magically solve anything nor do they co-ordinate cross domain flows because ideally they are not supposed to couple. So the coupling flows must be either pushed up to an integration layer above the contexts, or you just say stuff bounded context ideals and build the necessary integration flows from the context that needs to drive the co-ordination.
I think there are some techniques we can use here to minimise the coupling from one context into another but I’m not seeing it coming from DDD as there hasn’t been a good example presented on how we can minimise the cascading footprint of change when something in one context needs to change whilst also meeting application cross domain flows and coordination.
So my view is for 95% of elixir database apps just go do the best you can and not get hung up on things like DDD, because ultimately that’s all you can do, DDD doesn’t provide a magic bullet and in most apps it looks like a huge time wasting distraction that may result in weaker database models and less referential integrity if you take bounded contexts thing too far.
When you have millions of paying customers you can then worry about reducing coupling because you will understand the problem much better you will also have multiple systems and you will have the flexibility and cash to navel gaze and get less done.
1 Like
-
Yes, DDD is overkill for a small/simple project. But today, even a simple indie SaaS project may use a dozen external services for generic and/or supporting domains. You can still use context mapping to map all of these dependencies.
-
Some people in the Elixir community use DDD strategic and tactical patterns simply to better organize their code. This makes it easier to maintain/change.
-
Communication between BCs is carried out through pivot events. It can be implemented using event sourcing or durable transactional messaging (e.g. outbox pattern + message broker).
-
You use orchestration for workflows within BCs, and use choreography for workflows between BCs.
1 Like
