Hi all, my first post here. Certainly not new here.
First of all, huge congrats on the team for 1.3. It’s amazing in so many ways that it has actually forced me to re-think the way I create applications. I found myself spending hours at coffee shops drawing diagrams, contexts, etc. and each time I do it, I discover something new about the business problem space that I didn’t think of before (as a programmer). I’m loving it so far! However, I’ve seem to hit a roadblock…
Before I dive into my question, I’d like to give you a heads up:
I’ve so far read all 3 of the major books on DDD
Watched Chris’s video of Phx 1.3
Searched the forum posts
So, now for the question.
I’m building an ECommerce portal based off Phoenix 1.3. Let’s assume I have just 2 contexts - CMS and Ecommerce.
- CMS
- E-Commerce
Now, it so happens that the CMS domain has a page entity. While the Ecommerce entity has a product entity. I want to create a category model within the CMS context as well as the Ecommerce context.
- CMS
--- Category
- E-Commerce
--- Category
However, when I run the generators for the category, they seem to conflict: mix phx.gen.html CMS Category categories name:string shortcode:string description:text mix phx.gen.html ECommerce Category categories name:string shortcode:string description:text
They seem to overwrite the controllers and the views with one another, since they are all generated inside the Web context:
* creating lib/marketer/web/controllers/category_controller.ex
lib/marketer/web/controllers/category_controller.ex already exists, overwrite? [Yn] y
* creating lib/marketer/web/templates/category/edit.html.eex
lib/marketer/web/templates/category/edit.html.eex already exists, overwrite? [Yn] y
So, community, kindly help me to understand:
A. What exactly is the web context for?
B. Why can’t my Ecommerce and CMS namespaces handle their own templates, views, etc, separately
C. The whole problem seems to arise because everything is being dumped into the Web context. I don’t understand, everything is technically a web-facing entity, right? Phoenix is a web application framework, so my CMS, Ecommerce by right are all web-based functionalities already. Why should I dump it all into the web folder?
D. How would you re-architect this, instead? Clearly, I don’t seem to understand the new structure, or I’m missing something basic. I would really appreciate if someone can point me in the right direction.
Your thoughts, advice or any sort of help is much appreciated
As I understand it, Contexts in Phoenix are meant to separate your model behaviors from the way those models are persisted (e.g. In a database). What you seem to want to do is separate elements in the view and controller layers using the same context concept. Seems like something you could legitimately want to do, but I just don’t think they Context concept from Phoenix 1.3 is meant to do that.
What make make more sense, in your case, is having separate applications (one for E-commerce and one for CMS) in an umbrella framework. Your two web front ends for each of those areas could share underlying logic in a third application for shared model level functionality.
Thank you for your quick response. Suddenly, Umbrella makes total sense. However, I have a rather (stupid) question - Umbrella apps can’t run on the same port, if I’m not mistaken. So, in this case, my users can’t edit CMS pages and Ecommerce products on the same port (4000) right? How do you deal with this scenario?
I’m concerned that when my port 4000 is tied to CMS, it would be something like:
URL 1: admin/cms/pages/new
And say, 4001 would be tied to Ecommerce, it would be on a url like:
URL 2: admin/ecommerce/products/new
So, my user simply/easily can’t navigate from URL 1 to URL 2, right?
Please correct me if I’m wrong.
I don’t think there is a right (or wrong) answer here, my 2 cents: I’d probably leave the web part in one app, but refactor the generated controllers (I always do that, no matter the phoenix version). Generators are meant to be the starting point, they don’t have to define your structure.
Here is an example from 1.2 application: I have admin area with crud etc and frontend with some lists for say products. These two are different in all but models - admin needs to have a separate login etc.
So I have stuff like this in router:
scope "/", App do
...
get "/products/:category", ProductController, :index
...
scope "/admin", App.Admin, as: :admin do
...
resources "/products", ProductController
and in controllers there is an App.ProductController for frontend as well as nested App.Admin.ProductController in the admin subdirectory. Same with views and templates.
In bigger projects I use further nesting of entities, it doesn’t have to be the way to do things but treating the generator’s results as a starting point works all the time
I foresee two scenarios:
A. I setup a separate app called “Auth” for handling authentication. But then every other app will need to go through this app to check if a given user is authenticated and is authorized to perform the requested action. (Eg. view a particular resource/page). This seems like a violation of boundaries in terms of DDD to me.
B. I setup an auth sub-module under each app, and the current_user is simply tossed around from one app to another. Honestly I have no idea how to get this done.
Additonal info:
I come from a Rails background, so I’m still trying to break free of the Rails mindset. While at the same time, I want to make sure I’m not making a spaghetti (code wrangling from different modules) in the process.
I use coherence for my app setup. I have no idea how to break auth and user since they seem to be tied together.
After the advice on this thread, I’m now using an umbrella structure.
All I want to do:
From my previous example, when I visit Ecommerce to CMS, I would still need to be logged in.
I remember vaguely @josevalim mentioned somewhere in this forum along the lines of
Don’t pretend there are separations when there isn’t, it will only make things complicated.
I believe this is exactly where I’m stuck.
Thank you for your time, your help will be much appreciated
I don’t see how this is a violation of boundaries—if you have an app whose purpose is to handle authentication and authorisation, you could write a clean interface there and it would absolutely conform to DDD. The Auth app does not necessarily need to “know” about users past e.g. their emails or IDs.
So you could have an interface that lets you, say at signup, say to the Auth app “the user with ID 78 is allowed to buy products”. On login, you could call into the Auth app with the credentials, and if correct, it could give you the user_id (to be used with the Users context) as well as perhaps some sort of identity blob. Later, e.g. in an ECommerce controller, you could have a plug which grabs this identity blob (representing a logged-in user) and ask the Auth app if the user is allowed to, say buy an item. If not, you take appropriate action.
Another thing to consider is whether you actually want to separate your frontends into two apps. It is totally valid to have three apps, one managing ECommerce logic, one managing CMS logic, and one Phoenix app being the frontend to both. I like to separate things into their own modules first, and only if it becomes clear I really am running two independent systems do I refactor those out into separate apps in the umbrella.
One big thing which I’m not sure has been mentioned, and which is really different from Rails, is that Phoenix is not your app. When people say “app” in the Elixir community, they often mean an OTP application, which is just a bunch of modules and processes running together, often under one supervision tree. The advice you’re being given is to create your business logic in separate apps without a web interface, i.e. you should be able to run your app from iex directly (also great for testability), and then you can put a Phoenix-powered web interface on top of this, where the controllers simply call the appropriate business logic functions.
Thank you for your excellent, detailed reply Matt. Much appreciated. That was a very thorough detailed reply.
My concerns regarding separation were mostly around the routes. How do you organize the routes?
For example, if Auth module is taking care of the authentication and authorization, surely it needs to be a part of every route within every other module right?
So, cms's route file should make a reference to this auth for every request, if I’m not wrong. My logic says it’s wise to go with a shared kernel (DDD), using a pipeline for auth that every route from every other domain can use, but I honestly don’t know how to go about it. (If you’ve got some sample code, it would be great too!)
I think you’re still thinking of your web interface and your actual app as the same thing. The Auth module doesn’t need to know anything about routes, or be “part” of any routes.
The Auth module is simply a piece of code you can ask whether a certain principal (e.g. user) can do a certain thing (e.g. buy a product).
Routes are only relevant in the web layer—each route will have some controller handling it, and that controller can ask the Auth module whether the current user is allowed to do the action they’re trying to, and if not, error out. If they are, the controller proceeds to call into the appropriate context module, e.g. CMS, to perform the desired action, and then renders out the results as JSON or HTML.
This means you decouple your web interface from what your app actually does—you should be able to use your app fully from the iex shell before your web layer is ready.
So, to give an example. Say you have a ticket-selling service.
Features:
Guest must be able to create a User account.
Guest must be able to be authenticated as a certain User.
User must be able to see upcoming shows.
User must be able to buy a ticket, iff the tickets for a certain show are not sold out and the show’s deadline is not yet passed.
The Admin must be able to create new shows with a certain limited number of tickets, the price per ticket, etc. (and some other settings).
Now, it might make sense to split this up as follows:
Authentication Domain
This part is used to check if a certain guest can convince the application that they are a certain User. It uses some kind of underwater dispatching (which is an implementation detail and might be changed at any time!) to check:
If a user with the given name exists
If this user can be authenticated with the given passprase
(possibility to enhance for multi-factor authentication, etc).
When successfull, it returns the symbolic User reference to the outside world, which other domains might use to know that certain actions are (attempted to be) taken by a certain user.
The Authentication Domain asks other parts of the application for certain information. For instance, the User Domain will be asked if the given hashed password matches the one in the user domain user’s hashed password field.
The Authentication Domain does not care about what users actually are or where/how they are stored. (and has no idea about what tickets are at all).
Authorization Domain
This part is used to check if a certain entity can perform a certain action. Some other parts of your application will ask the Authorization domain if the current entity is allowed to do a certain action. The only thing the Authorization Domain knows, is what entities (referenced with some kind of symbolic identifier) are and aren’t allowed to do certain things.
The Authorization Domain consists of some bridges between some of the other domains. To check if a user might alter some user details, we might check if these are details of the current user, or if the current user is an administrator, for instance.
The Authorization Domain does not care what users or tickets actually are.
User Domain
Keeps track of general settings of a user, like their e-mail address, name, phone number, etc. A user can alter this information, but only for their own account.
The User Domain uses ‘some form of persistence’ to store user information for later retrieval.
Ticket Domain
The Ticket Domain stores information about shows and the tickets that might still be available. The Ticket Domain knows all about tickets, but not about the rest of the world. The outside world can ask to buy a ticket, which is then linked to a user. The Administration Domain can ask the Ticket Domain to create a new show.
The Ticket Domain doesn’t know what a user is (other than that a user has zero or more tickets that can be returned given the user’s symbolic identifier).
The Ticket Domain uses ‘some form of persistence’ to store shows, and what tickets have already been bought.
Administration Domain
The Administration Domain wraps the Ticket Domain and the User Domain to show and alter information that only the administrator(s) have access to. Access checking happens using the Authorization Domain, of course.
Phoenix
Enter Phoenix. This is just the web representation. All your controller (and channel) actions should do simple ‘flow redirection’, and call one (maybe two) functions inside a domain application, whose result(s) are then forwarded to the view+template layer (or channel broadcasts).
To allow users to authenticate over the internet, a technology like JSON Web Tokens could be used. A separate domain could be set up that handles the wrapping/unwrapping of these tokens that are passed to/from the user session, after which the Authentication and/or Authorization domains are used to find the unwrapped symbolic user reference.
There is no need to make the other domains care about authentication (how a browser session maps to a user).
Persistence
You might have noticed that I wrote ‘some form of persistence’ in multiple places. Because of the separation of these different concerns/domains, it is completely possible to store the list of users in one kind of persistence, and the list of tickets in a completely different kind. Everything will continue to work, regardless of if you use Ecto, MongoDB, ElasticSearch, Erlang Term Storage, a flat serialized file, or an in-memory ephemeral GenServer or Agent for this, because the persistence layers are only implementation details.
If you want to make the persistence layer as swappable as possible, you can go one step further, and use the (functional variant of) the Dependency Inversion Principle; specifying (for each domain/context/application) a persistence-behaviour that a persistence-layer-provider should implement.
Consuming External APIs
Maybe it is a requirement that we facilitate payments using Stripe, PayPal, iDeal, etc. The only domain that would be hit by this is the Ticket Domain. We could block new tickets from being bought until a pending transaction becomes either successful or failed. The nice thing is that all other parts of the application do not need any changes.
Providing a public API
Because we have a nice separation of concerns, we already have a good definition of how a public API could look like, and most public Domain functions could be wrapped 1:1 by an API call (for the domains we want to expose, of course).
‘Symbolic user reference’
This could be anything, as long as the different sections of your app agree about what it is. The Authentication Domain uses it to authenticate a user, the User Domain to keep user details up-to-date, the Ticket Domain to associate zero or more tickets with it. This might be a database Identifier, an Unique Universal Identifier, a simple struct, or something else that’s up to you.
@mjadczak@Qqwy Thank you both of you for your excellent detailed responses. Loving the community support.
Sorry for my delayed response, was busy traveling. I wanted to patiently craft a detailed response otherwise it would do no justice to both of your time spent helping me out
I have actually taken the approach you said. So, I have a travel website called ABC.
ABC has the following business domains:
The actual users themselves.
Flights, which users can browse / compare.
Hotels where users can stay.
E-Commerce where users can purchase 2 & 3.
Backoffice - Admin for all the above.
So, method 1:
I classified my business problem into following domain models:
The user domain - Handles auth, user related stuff. (Note: auth is a sub-domain)
Travel - Handles browsing of flights / hotels. (Note: Flight / Hotel is a separate sub-domain)
E-Commerce - Handles Payment, Order management, etc.
The routes:
The part where I hit a snag was when I had to maintain authentication across all different domains.
First challenge was handling the routes (which is me recent question posted while I was working on). How do you maintain authenticated and non authenticated routes within each domain?
Because, take flights/hotels for example, I want the users to be able to look at only certain of them. If the price is too low, I would require them to login to view the price. Should I put the routes inside the auth module where it belongs? Or into each individual domain??
So, it quickly became complex to even mentally keep track of what belonged where.
Authentication
Next, the false sense of separation. I really took a weekend off, broke down my application into pieces, drew everything on pen and paper. And it quickly became apparent to me that authentication isn’t really isolated. Why? Because, in reality, session information isn’t isolated - It is persisted through cookies across domains. Whether you’re in CMS, backoffice, flights, hotels, you NEED to be checked for your identity at every point. So, this was also quite challenging.
Cross-domain queries
Another problem I faced is maintaining referential integrity. For example, if I put all the user information into a separate database, now I need to painstakingly write extra code to maintain the referential integrity of the relationship. Because, my other resource (say pages, from CMS) will have a foreign key called user_id and I need to make a query across two databases to make sure this user with user_id exists. I am not sure how you guys are handling this, though. But, I have started working on other domains for now to focus on this one later.
More context:
How I’ve built things for now:
In my current setup, every domain is a phoenix app. It has its own database, it has its own controllers inside of that app and every domain has its own views and routes. I love this setup even though it breaks DDD rules because it gives me complete isolation from a business perspective.
I’ve come across threads here where people even separate Ecto into a separate domain. Their views (as you’ve also suggested) into a separate domain. But, to me, it doesn’t sit well because each business entity:
is unique.
should be responsible for its own actions.
may have different performance requirements.
Uniqueness
Each business domain is unique. For example, my CMS has a live preview feature and uses lot of JS libraries to make that happen, whereas for my ecommerce domain, I don’t need them. That’s why I love my current per-app-phoenix approach. I can afford to install brunch with different requirements in the default folders without having to modify much. IF I had a single application handle all my view level requirements, then it becomes a total pain. Sure, I could still install brunch with different requirements, but now, I need to create ten other namespaced public folders, etc.
Responsibility
Every app should be responsible for its own actions. This means, CMS should know how to get all its pages, Ecommerce should know how to get all its pages and render it to the users. IF there is instead a single app handling all this, then I think it would be vastly difficult to maintain the controllers, glue code to talk to the other apps from a single application when all my business domains in the real world are separated.
Different performance requirements
For example, in my current application (which is based off of an existing application), flights and hotels have extremely high loads, especially during holiday seasons. But, if I put everything into one domain responsible for handling controllers, generating views, it becomes a single point of failure.
If users are searching for a lot of flights during Christmas, then performance degradations are cascaded across all other entities when it shouldn’t have to. For example, if CMS goes down, ecommerce also goes down because view rendering is handled by a single application (the Phoenix app from your example). So, currently, I decided each business unit will take care of rendering it’s own pages, to an extent where duplication is not enforced. So, instead of the wordpress approach where everything is a “page” and stuff is built on top of it, in my application, an ecommerce “product” is totally different from a page, although, they have the same fields (title, content, description, permalink, etc).
So, I’m not singing a “this is the best approach” tune, but this is what I have chosen for now, and I’m still too early in the stages to conclude anything meaningful other than, this definitely seems to work well for me, alas, at the cost of breaking DDD.
Would be interesting to know your thoughts around this.
You’ve written quite a detailed post, so forgive me if I skip over anything, but here are my impressions on the scenario you’ve proposed.
I don’t think that an admin panel / back office is a separate domain here. A domain isn’t a subsection of your website. A domain is a “subsection” of your system or business. This relates to what you said:
Routes don’t belong in a domain. Routes belong in your web interface, and your other domains shouldn’t really care about them. Your core business logic domains shouldn’t care if they are being accessed through a web interface, the command line, or a touchscreen kiosk somewhere. I think the separation of your actual application from the web interface is something you’re still missing in your mental model. This talk may be useful.
Neither. The routes belong in a web interface layer, built e.g. with Phoenix, which merely interfaces with the system using its API boundaries. All of the other domains shouldn’t know or care that they are being accessed through the web, and hence they will have no concept of routes.
This ties in with the previous point a bit. Something which I feel many people mistakenly believe about the type of model that is being advocated here is the notion that domains must be completely separate systems that share no data. That is not true. What they should not share is internal implementation details—instead they should communicate through their public interface.
It is true that in the web interface context, the different domains will need access to different types of information about the user. It is true that this information needs to be stored into a session or cookies or some other state by the module that interfaces with the web browser (e.g. Phoenix). However, this does not mean that Phoenix needs to be coupled to these domains.
Instead, it is ok for these domains to return things to be treated as opaque, to be stored and given back later. An example of this might be a User ID. It’s ok for the Phoenix layer to store a User ID somewhere in a session, and provide it to domain modules where needed. It is likewise ok for the Flight Booking domain to hold on to a User ID within its own internal flight booking data. The key here is that these domains never take that ID and directly interface with the database to get user data—instead, they use that ID to ask the Users domain to perform any operations.
Let’s take this particular example. We’re going have three “Domains” here, Authentication, Authorization and Flights, as well as the web interface, Web.
First, if a user logs in, we handle that at the Web layer by passing their credentials to Authentication and obtaining a user_id, which we store in session. Note that the way we get a User struct is by calling the Users domain—we don’t query the database ourselves.
def login(conn, %{"user" => username, "pass" => pass}) do
case Authentication.auth_with_credentials(username, pass) do
{:ok, user_id} ->
conn = put_session(conn, :user_id, user_id)
user = Users.get_user(user_id)
render conn, "login_success.html", user: user
{:error, reason} ->
render conn, "login_failure.html", reason: reason
end
end
Now suppose that a request has come in to the Web interface, and we are inside a Phoenix controller, wanting to handle it. The request is for a “view flights” page. To render that page, we obviously need to obtain the flight information. As you said, this information will be different for logged in and non-logged in users, but we don’t care about this at the Web layer, as long as we can render a “hidden” flight (e.g. by showing “sign up now to see this deal!”)
def show_flights(conn, params) do
# extract the fields we care about, in a format which `Flights` is expecting
flight_params = sanitize_flight_params(params)
# we stored user_id in our session if logged in, if not, we use the anonymous user
user_id = get_session(conn, :user_id) || Authentication.anonymous_user_id()
flights = Flights.get_flights(params, user_id)
render conn, "flights.html", flights: flights
end
Note that we don’t care at this layer even whether we are logged in or not. Even if we are, we simply pass the user_id that we were given from Authentication through to Flights. Note that at no point we’ve assumed what user_id actually is—it could be an integer, an atom, a string, and we don’t need to know!
Now, let’s take a look at the Flights domain itself. Its job is to return flights matching certain parameters. But, if a certain user (for example the anonymous user) is not allowed to see premium flights, they should be marked as hidden.
def get_flights(params, user_id) do
params
|> fetch_flight_data() # local helper function
|> Enum.map(fn flight ->
if premium_flight?(flight) && Authorization.cannot?(user_id, :see_premium_flights) do
%{flight | hidden: true}
else
flight
end
end)
end
Again, the Flights domain doesn’t have to care what users actually are or how they work. All it needs is some token for the user that we are returning flights for (user_id) so that it can ask the Authorization domain whether this user is allowed to see premium flights. Likewise, Authorization doesn’t need to know what the :see_premium_flights permission actually means—its only job is to store whether a particular user has that permission or not.
Hopefully that example illustrates what I mean. Also, notice that no domain apart from the web interface actually knows anything about sessions or request. You could easily execute the code inside that controller inside iex and obtain the same results.
Finally, this is only one way to split up the responsibilities across the domains. You may decide that actually Flights shouldn’t know about the fact that some users are not allowed to see premium flights, and instead move that code to the controller itself.
Indeed, if you spread data across two databases, you do have to solve that hard problem. I’m not sure why you would though or what it achieves here. It is ok to add referential integrity checks at the database layer, without coding them into the software layer. Suppose that your Authorization permissions lived in a separate table to main user data, and your user profiles were in a separate domain Users. It’s fine to have a foreign key in the auth table to users, because without a user, their authorizations are meaningless. However, at the software layer, you’ll likely model that field in an Ecto schema as field :user_id, :id instead of belongs_to, :user, Users.User, because the second violates your domain at that layer. As I explained before, it is absolutely fine for domains to be passing around and storing opaque tokens belonging to other domains, as long as they don’t peek inside.
This entire thing is a spectrum and as is often the case, extremes have many problems.
You don’t have one system. You have multiple systems, and you’re trying to integrate them as external systems and wondering why you aren’t getting the benefits of having one system. As I said, usually the web layer is just an interface to the system—it is not a separate domain.
Your CMS and ECommerce domains do not care whether they are being displayed with a live preview or JS libraries. For all they know, they could be called from the command line.
You can achieve different resources being loaded for different sections of the site in other ways, but the beauty of the model we are explaining is that you can make these decisions after the fact. Because your application is completely separate from your web interface, you can have all of the modules be accessible through one huge website, or run a few smaller websites as separate Phoenix apps, all calling into the appropriate domains. You can even have multiple Phoenix “apps” being handled by one Cowboy server with a properly-written plug.
Yes it should.
No, it shouldn’t. It should be able to return a list of products or store pages or categories as a bunch of Elixir data. It’s the job of the web interface to turn that data into HTML that users can see.
Indeed it would be difficult, this is the way that people tend to write apps in Rails and this is exactly what we are arguing against
Separating your application into domains is again the solution here. If lots of people are searching for flights, the bottleneck could be at your web interface (in which case spin up more nodes with the web interface running), or the bottleneck could be with your flights backend (sorting through and processing all those flights).
Suppose that to date your whole application has been running on a single server, and suddenly you find that it can’t handle the load of processing all those flights. Because all of Flight-related functionality is within a single domain, you can change its internal implementation to actually work with a cluster of nodes behind the scenes to spread out all the work, and as long as you keep the same public interface to Flights, you should be able to get away with little to no code changes anywhere else in your entire application.
The entire point here is decoupling where it makes sense. This includes decoupling access (web) from the system itself.
The default model “encouraged” by Phoenix 1.3 is not strict DDD. It may adopt some concepts, but the main point is Phoenix is not your app.. Build your applications as a bunch of modules and then add a web interface on top. Keep the modules self-contained where practical and feasible. You could do proper DDD in Phoenix, but just following these guidelines gets you really far without the overhead that applying a strict model entails.
Since you just revived a 2 year old thread, let me just update here that I no longer use umbrella apps anymore as it simply adds a lot of unwanted complexity. I’m saying this after trying out various patterns, many architectures, including micro-services and what not since the last 2 years and I’m absolutely convinced that starting with a single huge monolith app and breaking it as needed (which is rarely needed) is the best way to go. If you’re a small team, umbrella isn’t for you. I still use DDD within the monolith app though.
If umbrella works out for you, then good for you, but it didn’t for me and hence my comment
I’ve definitely experienced the same thing. Trying to use umbrella’s to break up domain concerns tends to lead towards increased complexity without any real gain. If you’re not using umbrellas for operational reasons (deploying different apps to different nodes, etc.) then your app is all one monolith anyway. Its just a monolith with extra steps and complexity.
That being said discovering optimal bounded contexts inside a monolith isn’t easy (though poorly bounded micro-services are no better) and maintaining the boundaries requires a lot of discipline often challenging whether the “additional effort” is worth it - though the “can I replace this completely in two weeks” heuristic may be a good starting point for choosing initial boundaries.
Absolutely. I came across this fowler article too late in my 2 year journey. But let me make this clear - investment of time reading into DDD is not wasted at all, for DDD != Micro services. DDD will help architect your application with sensible logic. There are many good books on DDD, but almost all of them are too verbose and over 300+ pages to even get started, the one book that I personally found useful was this. I tried almost all books on DDD and this is my personal recommendation.
I use www.draw.io to do all my DDD work. Not sure if there are better tools out there specifically for DDD. At least on Mac.
