Thanks for clarifying Dave! You said “offering contexts as part of Phoenix is wrong” which I read as “offering contexts as a concept in Phoenix” and it seems you meant “offering contexts directly in Phoenix and not in a separate app”. 
4 Likes
Would parallel applications still run in the same BEAM, and if not, would you then create a client module and a server module, where by the client modules simply make :rpc calles to known servers? And the client and server code live in the same application, but when you need to use only the client you do not “start” the server?
Thanks,
Troy
2 Likes
I agree with this. This is how I develop my apps which are a bit larger. For smaller apps which are more web only I just put everything in one app (but of course still try to use the correct interfaces between my modules)
I tend to create an umbrella application with my application as one app and the phoenix web layer as another app.
I think it is great that Phoenix 1.3 is pushing towards this direction. For me it feels natural but I guess if you are used to Rails or Django which doesn’t usually have as clean interfaces perhaps it may look odd. I really like erlang’s -export directive as you really need to think on what functions you are exporting and it also gives a quick overview when you just open the files. Technically it is the same with def/defp but quickly look at a module to see what the interface is.
The cons to this approach I think is the same as the pros and cons of strong type systems (haskell, ocaml, rust). You need to think about your problem and problem domain in advance to get the right abstraction. This takes more time up front but saves you lots of time later. With dynamic languages you can get visible results much quicker but may end up with messy code that gets harder to refactor later.
I feel contexts are in the same problem domain as this. If you think first and get your contexts sorted out you wont be the first one out of the box but you will win the race
3 Likes
Could you provide a link? I remember reading about it or watching a video about it somewhere but I can’t find it.
2 Likes
Thank you everyone for taking part in the conversation 
Discussions like this are incredibly useful for developers like myself who are transitioning from the old ‘monolith’ way of doing things and are on a quest for something better.
I believe this (and finding it in the Elixir/Phoenix world) is actually a fantastic selling point - while many people may come for the speed, distribution, concurrency and fault tolerance I reckon they might stay because the eco-system pushes them towards good solid ‘modern’ approaches to building software that is resilient, less frustrating and scalable in every sense of the word. All the while making it fun (easy?) and interesting 
With that said, back to the topic. Although it’s very early for me to judge at present as I’ve still got 20% of the course to go through and we sadly won’t be using Ecto in the course (so I can’t yet fully picture a typical app) I am very excited by what I’ve seen so far. Much of what Dave has said seems to answer the burning question I often had when creating apps in other languages (that surely there is a better way of doing things).
Having said that, I can see a few challenges that might need to be overcome. For instance, if everything is created as separate services, and Phoenix should only be used for communicating via the web, then we might lose functionality or conveniences that Phoenix provides that we might want to use in the non-Phoenix part of our (overall) app - so those might need to be somehow moved from Phoenix or reworked in some way. The obvious one is validations (though I believe these are in Ecto - so maybe not an issue after all).
There’s probably more that I’m not aware of. José mentioned operational concerns and I would love to read about those too.
With regards to optimising for on-boarding new folks, I wonder if this could be worked around by having a --simple flag on app creation and where the generators reflect that (monolith?) ‘style’. This would explicitly let people know that it’s a learning tool, or one for quick prototyping or for creating the simplest of sites/apps. But perhaps more importantly it will spearhead a brave new direction, ‘the Elixir way’ of doing things. (I would actually be super excited by this! As if I could get any more excited about Elixir and Phoenix that is!)
The reason I think this is worth investigating was actually commented by Chris himself in his keynote, when he referred to @sasajuric’s feedback of where an expert team were somewhat led down a not ideal path because they thought the Phoenix way was what it was telling you to do via generators.
One final thing I would like to say is that it’s a very exciting time to be a developer! And an Elixir developer at that! I’m extremely happy to see that José, Chris and everyone in the Elixir, Phoenix and Ecto teams have this pursuit of reflection > perfection - and equally heartened to see that experienced members of the community are part of that transition, by feeling able to have conversations and inspire discussions like this.
8 Likes
But yeah, overall I agree with pragdave. And honestly, as I’ve mentioned before, I’m not even a fan of umbrella apps. Umbrella’s encourage a lot more interconnections between sections then what should really exist. I instead make my parts as lots of little dependencies that I then bring into a main app that does all the configuring and more. If I need a couple sections communicate between then I setup a channel between them at this point (not like a phoenix channel, but a set of behaviours in a sub project that both depend on, then setup their communication modules in the config). I find it keeps it very very clean and maintainable.
This precisely, this is what I do.
And this, 1.3 became less coupled to ecto, but I still do not like the monolithic apps that hold many (to use phoenix terms) contexts when a context should be its own standalone application/dependency.
I pass the Repo one of my dependencies should access into its config, which it then bakes in to their calls at build time. ^.^
That is really how it should be done I’d say, usually I pass the Repo module around as an argument, but currently I’m just using things like @Repo unquote(Application.get_env(:this_app, :repo) || throw ":this_app does not have :repo defined") in more places than I probably should. I should probably make a library to simplify a lot of this repetition too… >.>
Ditto, make lots of little applications and bring them in as dependencies.
Precisely this! They should be standalone dependencies as I’ve stated in that huge Contexts thread. ^.^;
They all run on the same beam. Even right now your beam instance is running probably a few dozen or more applications. 
6 Likes
It’s possible that we are overthinking this and if I’ve got to choose between one of these approaches, I think Phoenix is getting it right by encouraging and enabling separation rather than forcing it.
Completely separate applications come with a lot of hidden complexity that doesn’t generally need to be introduced until you really require it. The whole microservice vs monolith discussion generally boils down to isolation of dependency trees and resource consumption at its root. If a monolith is built with an object oriented language, you end up with this horrifying blur of inheritance chaining that’s incredibly messy to untangle. If you go microservices from the start it’s good long term but you pay for it up front with potentially unwarranted complexity.
With the approach from Phoenix and Umbrellas you don’t have a tangled mess, thanks to the way Elixir is designed. You’ve got smaller parts in their own zones that are significantly easier to separate when you finally NEED to.
Learning curve and on-boarding are a huge deal for a language. Determining trade-offs that get people moving quickly in the right direction is a key and it’s one of those things that is hard because you’re always sacrificing something. It’s something programmers debate about constantly because business factors like turnover, training, learning curve and time to market are real factors that matter beyond ideal architecture. Ideal architecture seems a lot easier when those other factors aren’t included IMO.
In that regard and from a community standpoint, I really do think that Phoenix 1.3 has nailed that balance. Like anything else in programming, it’s always a question of trade-offs and hidden costs.
If we aim for architectural purity instead of that balance, we aren’t going to be much better off than the “OMG Benchmarks!” crowd. If productivity, learning curve, training time, etc aren’t a factor then why aren’t we just using Erlang?
18 Likes
Just to mention: When you designing contexts as separate umbrella app it forces you to think about deps and configs.
4 Likes
My comment which Chris cited should be regarded in the context of Contexts
In particular, I’m not at all convinced that we should enforce separate OTP apps. Contexts basically nudge us to move web-unrelated concerns to another function in another module. That’s a simple separation of concerns with a very little overhead. Separate apps lead to much more operational and mental overhead, and I’m not convinced this is a good starting option. Personally, I feel that with contexts, it should be fairly straightforward to split an app later on, if needed, as it should mostly boil down to moving some modules and some parts of config. So IMO contexts are a very good trade-off between approachability and some sane default decoupling.
17 Likes
I think the question is, WHY. Why do database calls need to be broken into other services. What is gained? What are the concrete advantages from a development, performance, and operational perspective? How does that weigh against the additional complexity introduced?
If the answer is “well, because. Because it’s not Rails”, then I don’t think that’s a sufficient answer. If the impetus is architectural purity (“that’s not how it should be done in Elixir!”), without regard for the implications in real projects and real development, I’m unconvinced.
I’m not saying there aren’t advantages to breaking everything into microservices. But most apps that have a data layer can have a data model that is by necessity a series of highly coupled dependencies. And efficiently querying that model often requires that those dependencies be maintained. Breaking that into microservices loses a lot, and again, I’d have to be very clearly convinced that what is gained makes up for it. I think there is so much talk about “X is good and Y is bad” without real world examples of why, and what the implications are then you lose what X gives you because Y is so great.
As others have said, I think that contexts do a good job of at least a first pass of separation of concerns (even though there may be data dependencies between contexts). I’m not sure that there’s much need to make all applications, even basic applications, require much more than that in terms of complexity.
At the end of the day, architectural purity loses to real world concerns every time, in my book. Unless that purity has real world advantages.
13 Likes
This is very good question, and the only question that matters. The answer is a cliche, but it’s “to build better, more maintainable software”.
I think contexts were introduced because some people (me included) complained about inadequate separation of business logic from the infrastructure provided by Phoenix. It is awesome that the team tried to solve the problem, I am not on the side of these that think it’s a good enough solution, however.
Like @pragdave, I strongly believe that the “web” or “ui” app should be merely an interface to the underlying application - or applications most likely. Now it’s the question of how thick or thin this layer should be.
Database access layer, in my humble opinion, does not belong to the web UI interface layer for sure. It is an implementation detail, that the business logic application would use internally and behind the scenes. Ideally, this can be done using Ecto schemas that are as dumb as possible. In my app, I do not put any validation-related logic there, the Ecto schemas define just schema and relationships between schemas. These Ecto schemas also would not leak to the UI directly.
The “core” application, that provides the business logic, most likely keeps the Ecto schemas internally. The responsibility is to respond to either queries or commands issued by user, and either return some stuff (for query) or update some stuff in database (command), sending e-mails, generating reports, updating external services in the process. You can have many “core” applications in fact, which is probably good practice in most cases except simple problem domains.
How does user interact with our “core” application? They submit forms. Or they click on a link that can be mapped to empty form submit. In general case they call some endpoint on the UI web app, passing 0 or more arguments along. This input needs to be validated, business logic checked against “core” application, and if such command is looking like something user can do, it’s been passed over to “core” application for execution. The question if the modules responsible for casting forms to commands, and initializing validation is responsibility of UI or back-end “core” app is open and will vary from case to case in my opinion.
So the above is the ideal breakdown in my humble opinion, for most real life cases. I do not believe, however, that all the work above has the same weight in terms of making the software maintainable, and predictable: the most important part is separating command/form validation from underlying schemas that persist / read things from database. By doing so you no longer think in terms of CRUD app, instead of commands issued by user. You can also avoid many permissions-related bugs in the code, such as mass-assignment related issues.
Now, how to achieve that? In Rails we used to import Virtus.model to our so called “form objects”. You can do exactly the same, with Ecto schemas that are not backed up by any table. This way you integrate well with default form builder, gettext, etc. If you have an API, that layer can be totally separated from Ecto library, and either use your own, or something like vex.
So while I appreciate effort to make separation of modules in Phoenix, I think it went slightly less important way. Separating form validation from actual schemas does seem like a more important task for me. And it’s true you can easily do it on your own, but it’s not what the generators teach users to do
9 Likes
It is important to say that nobody is advocating to “put everything in the same app, forever”. To me a great sign for splitting your Phoenix projects apart is exactly when you need to communicate to extra data sources, be it batch processing from CSV files or an API from a third party vendor.
Some have mentioned microservices in this conversation but what is being proposed, which is to split into multiple applications that depend on the same database, is against one of the core tenets of microservices/SOA which is operational isolation.
I understand some may still prefer to build their projects with multiple applications talking to the same database because it gives them more isolation at the application design level. But I also understand those who prefer to split things apart when you can actually guarantee they are fully decoupled. YMMV.
The only thing I am 100% sure is that if Phoenix generated multiple applications by default, we would be discussing how it is a terrible choice for most applications, how it is verbose, how it gives a false sense of security due to the operational coupling and how it is in general over-engineering. 
I am not saying I agree with any of those, just pointing out those discussions will happen regardless of choice.
That’s why I am glad we were able to grow Phoenix+Ecto to the point were all of those choices are possible. I like to say languages, frameworks and libraries should “enable rather than provide”. Languages more than frameworks, frameworks more than libraries.
And while we need to pick one approach for generators, the fact Phoenix just calls modules from the web layer, allows you to put your business logic anywhere: in the same app or different ones. And in there you can even get rid of the database altogether. It doesn’t care.
22 Likes
We can remove generators altogether too. I never use them. There’d be no preferred way to build stuff anymore. Every architecture is on the table then.
3 Likes
I don’t think removing the generators is an option. While I also don’t ever use them, I agree with Chris that they are mainly for the learning purposes.
If you’ll ever use them, really depends on your style of learning in my opinion. Some just like to build things from scratch, but others prefer to get something working as quick as possible and then play with things, change the code, experiment. The first group can of course chose not to use generators, but I think the community would loose a lot of potential developers who would get discouraged by having to type everything when they understand nothing.
In my opinion, having generators is a perfect middle ground. It just need to be emphasised that they are mainly there for the learning purposes, not for actual development, but Chris and the team see this and they are already making an amazing job in explaining this.
5 Likes
You may have missed the main point of my previous comment. If we remove generators, then we would have a huge conversation about how generators are important and how we are making it hard for users to learn and get started with Phoenix by not including them.
There is no perfect choice. They all have pros and cons and each of us value those differently based on our experiences. It is important to talk about those different experiences but folks should drop any expectation that Phoenix will change on this front. The Phoenix team has already made their choice based on their understanding of the trade-offs. Contexts and generators are here to stay.
12 Likes
I think this is one of the disconnects whenever we discuss this.
When you express concern with microservices sharing a database, I think it’s because you view it as a form of coupling, because it’s the same database.
But, say instead that the design rule was “every transaction has its own service”. Not every database table, but every transaction.
Given that, I think I suddenly gain the freedom to switch implementation schemes far more easily than I could with a single database wrapped in contexts.
Say, for example, I wanted to switch from immediate payment authorization to deferred authorization when placing an order. With everything in one database, accessed via a single layer, it would be a change that would be riskier, as you’d have to manually look through everything on the order side for interactions.
But if I simply had a “place order” service, then I’d know that every change would be localized. I could even trivially A/B test by switching between old and new services.
I certainly don’t lose anything by moving to a service-per-transaction model (except possibly some performance for very small-scale transactions, and that’s a trade I’d make for flexibility).
Dave
5 Likes
There’s an extra pattern here where you can explicitly implement in-application transactions using GenServer. This really helps in situations where you need to do multiple things, in atomic way, that is not limited to the database operations. In your example you could spawn a GenServer for each order, make sure it charges user only once, generate one PDF invoice etc. Just something I find very useful from time to time - rather than relying on database transactions that do not provide such locking mechanism easily, btw.
4 Likes
Generators
Personally I don’t think we should get rid of generators - they are a convenience far more than they are a learning tool Imo. I used them all the time in Rails, particularly for generating Models, Controllers and Migrations. I think I stopped using them as a learning tool (i.e to generate a Resource) after a few weeks.
But why not have a system where you can decide what kind of application you want to build and the generators reflect that? So on app creation a --simple flag for one (which would also be ideal as a learning tool) and then either have a default to reflect the direction you want to push people, or just more flags: --microservices etc? I think this would be an industry first 
and a way to cater to the needs of the most common approaches to app development in Elixir… while at the same time allow you to spearhead what the community thinks is the smartest way to develop applications*. There could even be an --experimental flag
*Something I think would elevate Elixir & Phoenix even higher above everything else out there.
Microservices - same database, or not?
With regards to Microservices using the same DB, from my understanding this is an option:
There are a few different ways to keep a service’s persistent data private. You do not need to provision a database server for each service. For example, if you are using a relational database then the options are:
- Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
- Schema-per-service – each service has a database schema that’s private to that service
- Database-server-per-service – each service has it’s own database server.
With regards to why use separate DBs (which I actually like the sound of when I picture the kind of apps I want to create and the scale at which they might operate) here’s what doing that enforces:
Services must be loosely coupled so that they can be developed, deployed and scaled independently
Some business transactions must enforce invariants that span multiple services. For example, the Place Order use case must verify that a new Order will not exceed the customer’s credit limit. Other business transactions, must update data owned by multiple services.
Some business transactions need to query data that is owned by multiple services. For example, the View Available Credit use must query the Customer to find the creditLimit and Orders to calculate the total amount of the open orders.
Some queries must join data that is owned by multiple services. For example, finding customers in a particular region and their recent orders requires a join between customers and orders.
Databases must sometimes be replicated and sharded in order to scale. See the Scale Cube.
Different services have different data storage requirements. For some services, a relational database is the best choice. Other services might need a NoSQL database such as MongoDB, which is good at storing complex, unstructured data, or Neo4J, which is designed to efficiently store and query graph data.
Source: Pattern: Database per service
5 Likes
The problem here is that someone would have to maintain those. And, as with Phoenix 1.3 rcs, you can see that the generators get updated last. It’s extra work for the team.
There is actually nothing stopping you or me from creating extra generators, that would not be part of Phoenix core.
2 Likes
Someone thinks that is a problem to start with context modules as Phoenix 1.3 encourages and as the application grows giving a better understanding of what modules needs to became full OTP applications, split it later? I think that start small and growing the application design is the right way, instead of over engineering the solutions just from beginning.
6 Likes
