Discussion: Don't add a database layer to your Phoenix application

[AstonJ]

I’ve just started the Phoenix part of the utterly brilliant online course by @pragdave. On generating the Phoenix app he uses the --no-ecto flag and explains why:

I don’t think anybody needs to add a database layer into their Phoenix application. Because Phoenix is purely concerned with communicating on the web, and anything you use a database for should be in its own separate service. So I don’t see any point at all in putting Ecto into any Phoenix application that you generate from scratch, you always want to put your logic into separate applications in the way we’ve done so far.

Do you agree? Is this how you build your Elixir/Phoenix apps? Can you think of pros/cons?

If you haven’t got Dave’s course I highly recommend it. There’s much, much more like this and I’ve learned such a massive amount from it already - particularly on HOW to approach modern development using Elixir, rather than ABOUT Elixir itself. It’s definitely one of the best online course I have ever done… and I really don’t say that lightly!

[chungwong]

@AstonJ

–no-ecto

Phoenix 1.3 doesn’t “tightly” integrate Ecto into it. Even with 1.2, I don’ t think it is really a big difference.

[gregvaughn]

I generally agree with @pragdave, but in this case perhaps I’m a bit more … pragmatic

People are going to visit Phoenix with a background in other “full stack” frameworks and expect some sort of a generator that gets them boilerplate code from html templates all the way to a database table. If Phoenix does not offer that, whining would occur. From an adoption and “quick wow factor” approach, having Ecto in there is a win.

On the other hand, I completely agree that in the long run, the code will be better designed if you keep web access separate from core business logic. However, that opinion comes from experience, and those with less or different experience won’t want anyone else to tell them how they have to do it. I also imagine some projects with simple requirements and/or short lifetimes really are better off with a “full stack” approach, though I’ve never in 20 years actually worked on one.

If we took even the option of using Ecto out of Phoenix, then someone would create a new hex package that wrapped them both together to offer that end-to-end out-of-the-box experience and we’ve just pushed the concern to another place.

[AstonJ]

I certainly agree with you Greg (and don’t think or am suggesting that Ecto should be removed from Phoenix). Like you said it’s currently the most common approach in the industry and so Phoenix probably needs to continue catering to that for the foreseeable future. Plus I imagine there are other benefits, such as for prototyping; when you need to get things done fast.

But it’s the other cases I’m interested in - the somewhat larger applications where this kind of approach might make sense and how common Dave’s approach actually is (I think he said that he’s in the minority in this (and some of his other approaches) at the moment).

I should also add that I’m really excited by this (and a few other things he’s mentioned) as they go back to the burning question I had when developing in another language, where I often thought that “surely there is a better way of doing things”. I know Elixir doesn’t hold exclusivity on these things, but I find it very heartening that it’s introduced to us (and perhaps so many of us) as a lot of the Elixir learning material also covers these ‘modern’ practices

[gregvaughn]

@AstonJ

I think he said that he’s in the minority in this

I’d venture to guess he’s in the majority among people with 20+ years experience writing software professionally. But also in the minority among those with <10 years experience. And because our industry skews young, he’s in the minority overall. I hate to turn this into a point about age/experience, but I am heartened that the BEAM community in general recognizes and respects experience. Dave has been someone I look up to since I first met him 17 years ago.

[AstonJ]

I think you might like the next discussion I post then (where he thinks he’s the only person in Elixir doing it his way ) but I’m holding off for now as it’s not quite as simple as this one and I would need his permission first (to post it in the way I want to anyway).

@gregvaughn

I am heartened that the BEAM community in general recognizes and respects experience. Dave has been someone I look up to since I first met him 17 years ago.

Again I’m in complete agreement with you. In fact one of the reasons I was attracted to Elixir and Phoenix is because I felt they push us towards a better way of doing things. As much as I like Rails, and loved how easy it was to get started in, I think I’m at that stage where I don’t want short-cuts or instant gratification, but the most sensible approaches that will lead to more resilient software. So I’m definitely interested in hearing what experienced people like you and Dave have got to say

[pragdave]

My concern is not that ecto is available in Phoenix. It’s rather that first 1.2 and the 1.3 actually tightened the coupling between the two. as a result, people will come into Elixir and assume that they should write monolithic applications. If instead phoenix.new generated a separate parallel database access application, alongside the phoenix app, then people would have the same capabilities but without the temptation to write “yet another Rails app”.

Sure, we can let people discover this over time. But if we do that, and if people start running into the same problems with Phoenix that drove them away from Rails, what have we gained? People will just move on to the next big thing.

Instead, if we explain decent coding principles, and live by them, then we can help people have a better long-term experience. This will be good for them, and good for the language.

Dave

[gregvaughn]

@AstonJ

I’m at that stage where I don’t want short-cuts or instant gratification, but the most sensible approaches that will lead to more resilient software.

This is what excites me most about the Elixir community. It’s not long yet, but there is a history in the Elixir community. At one point Phoenix appeared to be Rails++ and some of us encouraged @chrismccord to aim higher due to what the BEAM offers. I’m sure that was hard for him to hear at first, but the new contexts in Phoenix 1.3 demonstrates his agreement with the underlying design concerns. Even if Ecto is in the same BEAM application as your Phoenix web code, good discipline can keep them decoupled. Decoupling is the core concern @pragdave speaks of. Yes, it is easier to use BEAM applications to enforce that decoupling, and that’s my preference, but I won’t automatically judge some project that has Phoenix and Ecto in one app.

Perhaps that is self-serving because I currently work with some Phoenix 1.2 code that has them in one app. There’s plenty we could and should do to decouple them, but business needs have to be balanced with technical needs. But there’s still a decoupling mindset amongst the team that helps us to incrementally improve the code. And that ties back into your point about writing maintainable and resilient software – keep your concerns separate.

[gregvaughn]

@pragdave

If instead phoenix.new generated a separate parallel database access application, alongside the phoenix app

So, it’s primarily the generators that concern you?

Personally, I don’t use much beyond creating a new project and generating a new migration. I tend to code anything else generators can generate myself, by hand. I know there’s a separate audience that rely heavily on generators, but I’m still trying to understand that group. It’d be interesting to hear from them in this thread.

[AstonJ]

@pragdave

Sure, we can let people discover this over time. But if we do that, and if people start running into the same problems with Phoenix that drove them away from Rails, what have we gained? People will just move on to the next big thing.

Instead, if we explain decent coding principles, and live by them, then we can help people have a better long-term experience. This will be good for them, and good for the language.

I think this is a fantastic idea

I don’t think the extra initial ‘complexity’ will deter people either, on the contrary I think it will do wonders in attracting even more people because as both Greg and I have commented, people are coming to Elixir and Phoenix because we want a better way of doing things.

Btw, so far everything you have said in your course like this (microservices, single responsibility functions/modules etc) makes SO much sense to me - so much so that I am actually bursting with excitement and dying for everyone else to do your course and learn these amazing things as well!! THANK YOU DAVE <3

[AstonJ]

@gregvaughn

This is what excites me most about the Elixir community. It’s not long yet, but there is a history in the Elixir community. At one point Phoenix appeared to be Rails++ and some of us encouraged @chrismccord to aim higher due to what the BEAM offers. I’m sure that was hard for him to hear at first, but the new contexts in Phoenix 1.3 demonstrates his agreement with the underlying design concerns.

Yep, and the same with Ecto when that had a change in direction. I think it shows remarkable strength and I feel it’s been incredibly good for Elixir - because people are taking note that the developers aren’t afraid to make changes if it means resulting in a markedly better product

[chrismccord]

Our project generator defaults first and foremost have always optimized for on boarding folks getting into the ecosystem. History and support requests from slack/forums/irc tells us our defaults of including DB integration out of the box are what the vast majority of folks want when getting into Elixir and Phoenix. This same argument is why we include brunch by default – folks want a good story on “where do I place my js/css so I can style and script my web application”. Our defaults are driven by these facts, but as Dave points out in his course, there are switches to customize your new project once you are comfortable in the ecosystem.

With respect to decoupling and thinking in terms of larger application architecture, our --umbrella flag to the phx.new generator follows what dave lays out in his course. You have an apps/my_app_web which is the phoenix server, and a separate apps/my_app, which is your app, and contains Ecto/DB integration by default. There are also new phx.gen.ecto and phx.gen.web project generators for generating isolated applications for existing umbrella apps, so I believe we tick the boxes dave is after if you are using umbrella apps. That said, we don’t generate an umbrella project by default today because it places a larger burden on folks getting started – multiple config files, multiple mix.exs files to place deps in different places, etc. It’s also not clear that all project scopes are best served by umbrellas. What we do in Phoenix 1.3 to bridge this gap is a project structure that more easily lends itself to graduate to an umbrella app, hence the lib/my_app_web and lib/my_app folders with separate aliases.

@pragdave

It’s rather that first 1.2 and the 1.3 actually tightened the coupling between the two.

Dave, can you expand on this point? The push for contexts encourages folks to use discrete APIs exactly to decouple the two. Whether MyApp.Calendar.create_reminder talks to an Ecto repo, or starts a GenServer process with a timeout is an implementation detail from the caller’s perspective so I don’t agree with this point.

[kylethebaker]

In what context is ‘application’ being used here? I can think of a few that might apply. Does ‘phoenix application’ refer to the entire mix project generated by phx.new or does it refer to just the myapp_web portion? Or does it refer to the actual otp application that you get out of the box?

Phoenix 1.3 encourages the logical separation of your web concerns and your business concerns, with the data layer being lumped in with the business logic. Is the idea to pull that apart further into three separate pieces: [web, business, data] instead of [web, {business, data}]? So that inside of a Foo context, Foo.get_all_things() contains something like DataLayer.get_all_things() instead of Thing |> Repo.all(), where DataLayer is referencing a separate service, whether that be in another child of the main supervision tree or a completely separate otp application?

I already think of myapp and myapp_web as being separate concerns, with nothing phoenix specific touching anything from myapp. So in that sense the database layer has a ‘source code’ decoupling from the web stuff, but there is still a ‘project’ coupling in that they both live in the same mix project, and I suppose a sort of ‘run time’ coupling in that the ecto Repo and phoenix Endpoint are siblings under the same direct parent supervisor and run under the same otp application. Is the idea to move the database layer into it’s own otp application?

[pragdave]

The push for contexts encourages folks to use discrete APIs exactly to decouple the two.

To a point. But the code still lives in the Phoenix app, so the presence of contexts actually encourages the exact behavior that is long-term harmful.

It seems to me that contexts are simply a way of creating service apis.

If I were writing an application that had services, but I included those services in one of the endpoint API modules, that would generally be considered to be a bad design. Why, for example, would a user creation service be part of my web API module when I would also use it from my command line and from my batch processing?

So I really think that offering contexts as part of Phoenix is wrong. It says that the Elixir community supports this kind of coupling.

But the reality is that the Elixir ecosystem is one of the few that actually supports the proper (and efficient) splitting of these responsibilities.

I really don’t buy the “ease-of-entry based on new user expectations” argument. If I did, I’d write a PHP interpreter as part of Phoenix. Instead, I think we make the entry easier by making it simpler. Give people a good architecture, where everything has it’s place, and where decoupling is the norm. The aim isn’t to write a 10 minute blog—it’s to write a five year application.

Parallel (not umbrella) applications are trivial to create and manage, and have decoupling built in from the start.

@chrismccord

Whether MyApp.Calendar.create_reminder talks to an Ecto repo, or starts a GenServer process with a timeout is an implementation detail from the caller’s perspective so I don’t agree with this point.

Except… this assumes that the Calendar context/service is part of the Phoenix app (assuming MyApp is one), and this just doesn’t make sense. Phoenix is not the application. It’s a way of accessing the application.

Dave

[kylethebaker]

@pragdave

If I were writing an application that had services, but I included those services in one of the endpoint API modules, that would generally be considered to be a bad design. Why, for example, would a user creation service be part of my web API module when I would also use it from my command line and from my batch processing?

Isn’t this the reason for contexts though? Your application services aren’t included inside of your web api module, but rather inside of the appropriate context module. MyAppWeb.Controller.UserController.create calls into MyApp.Accounts.create_user. If you wanted to use the user creation service from the command line you could have Mix.Tasks.MyApp.CreateUser.run that also calls into MyApp.Accounts.create_user to do the actual work.

In that sense, the phoenix/mix edges of your application are essentially just i/o adapters that are translating inputs/ouputs from somewhere (http, stdin) into calls to the functional api that lives inside of the contexts. If you maintain strict segregation between your contexts and the i/o layers (which I think 1.3 encourages), then your application remains decoupled from phoenix.

That is the way that I think of it at least. I think of phoenix as just being a dependency for my elixir application that lives along side of it inside it’s own app_web world.

[michalmuskala]

@pragdave

Except… this assumes that the Calendar context/service is part of the Phoenix app (assuming MyApp is one), and this just doesn’t make sense. Phoenix is not the application. It’s a way of accessing the application.

All the web things are in MyAppWeb namespace, MyApp has no web components.

[pragdave]

The point still holds: it’s still one big application. It should be many.

[josevalim]

It is really hard to agree that Phoenix v1.3 became more coupled to the database compared to previous versions. It has gone from an application that accessed the database in your controllers, channels and models to one that puts this access behind isolated modules with discrete APIs. Removing the database access in a Phoenix v1.3 app should require localized changes to your lib/app instead of rewriting most of your web layer.

It also puzzles me when someone says contexts are wrong because contexts are literally modules. You can disagree with the placement of those modules (in the same application instead of separate ones) but using modules to define our APIs is pretty much our bread and butter. This is supported by the fact you can now generate the web code that access any module and then just fill in the blanks. The blanks can be filled with a module in the same or in a separate application. Or even with a module that does not reach the database at all.

I agree Phoenix v1.3 does not completely push in the direction @pragdave would like to but it does make those patterns possible which is something we cannot say at all about previous versions.

I also have strong reservations about having different applications that depend on the same database instance, as those are effectively coupled at the operation level and you will run into issues depending on how you coordinate things such as migrations. I would like to see both development and operation concerns being taken into account on those discussions. But since I have already phrased those concerns elsewhere, I won’t rehash them here.

[pragdave]

The fact they live in a different place doesn’t stop them being part of the same application codebase.

I agree that Phoenix is an I/O adapter. I don’t agree that the application code should share the same mix project (or even umbrella project). That would be like putting my text-based adventure game into the /dev/tty driver codebase…

[pragdave]

@josevalim

It also puzzles me when someone says contexts are wrong because contexts are literally modules.

I don’t say contexts are wrong. I do say that encouraging new developers to place them inside the same project as the view layer is not a good idea. So in that way, 1.3 created a good idea (contexts/services) but then encouraged it to be used in a way that can lead to lots of coupling, and to Monorails. (And, as we’re discussed, umbrella projects don’t fix it.)

Saying “developers don’t have to use it” is dissembling when it is the default (and hence recommended) way that Phoenix apps are created.

Don’t get me wrong. I’m in awe of the amount of work that’s been done, and I’m very grateful to everyone. But I’m simply worried that this effort might not realize its full potential unless newcomers are shown that Phoenix is not a web framework—it isn’t Rails. Instead it is the web-tier (or, really, the TCP tier) in a multi-tier overall application.

And with that, I’ll sign out, because I respect the developers too much to spread bad karma.

Dave