Discussion: Don't add a database layer to your Phoenix application

So, it’s primarily the generators that concern you?

Personally, I don’t use much beyond creating a new project and generating a new migration. I tend to code anything else generators can generate myself, by hand. I know there’s a separate audience that rely heavily on generators, but I’m still trying to understand that group. It’d be interesting to hear from them in this thread.

2 Likes

I think this is a fantastic idea :023:

I don’t think the extra initial ‘complexity’ will deter people either, on the contrary I think it will do wonders in attracting even more people because as both Greg and I have commented, people are coming to Elixir and Phoenix because we want a better way of doing things.

Btw, so far everything you have said in your course like this (microservices, single responsibility functions/modules etc) makes SO much sense to me - so much so that I am actually bursting with excitement and dying for everyone else to do your course and learn these amazing things as well!! THANK YOU DAVE <3

1 Like

Yep, and the same with Ecto when that had a change in direction. I think it shows remarkable strength and I feel it’s been incredibly good for Elixir - because people are taking note that the developers aren’t afraid to make changes if it means resulting in a markedly better product :slight_smile:

1 Like

Our project generator defaults first and foremost have always optimized for on boarding folks getting into the ecosystem. History and support requests from slack/forums/irc tells us our defaults of including DB integration out of the box are what the vast majority of folks want when getting into Elixir and Phoenix. This same argument is why we include brunch by default – folks want a good story on “where do I place my js/css so I can style and script my web application”. Our defaults are driven by these facts, but as Dave points out in his course, there are switches to customize your new project once you are comfortable in the ecosystem.

With respect to decoupling and thinking in terms of larger application architecture, our --umbrella flag to the phx.new generator follows what dave lays out in his course. You have an apps/my_app_web which is the phoenix server, and a separate apps/my_app, which is your app, and contains Ecto/DB integration by default. There are also new phx.gen.ecto and phx.gen.web project generators for generating isolated applications for existing umbrella apps, so I believe we tick the boxes dave is after if you are using umbrella apps. That said, we don’t generate an umbrella project by default today because it places a larger burden on folks getting started – multiple config files, multiple mix.exs files to place deps in different places, etc. It’s also not clear that all project scopes are best served by umbrellas. What we do in Phoenix 1.3 to bridge this gap is a project structure that more easily lends itself to graduate to an umbrella app, hence the lib/my_app_web and lib/my_app folders with separate aliases.

Dave, can you expand on this point? The push for contexts encourages folks to use discrete APIs exactly to decouple the two. Whether MyApp.Calendar.create_reminder talks to an Ecto repo, or starts a GenServer process with a timeout is an implementation detail from the caller’s perspective so I don’t agree with this point.

15 Likes

In what context is ‘application’ being used here? I can think of a few that might apply. Does ‘phoenix application’ refer to the entire mix project generated by phx.new or does it refer to just the myapp_web portion? Or does it refer to the actual otp application that you get out of the box?

Phoenix 1.3 encourages the logical separation of your web concerns and your business concerns, with the data layer being lumped in with the business logic. Is the idea to pull that apart further into three separate pieces: [web, business, data] instead of [web, {business, data}]? So that inside of a Foo context, Foo.get_all_things() contains something like DataLayer.get_all_things() instead of Thing |> Repo.all(), where DataLayer is referencing a separate service, whether that be in another child of the main supervision tree or a completely separate otp application?

I already think of myapp and myapp_web as being separate concerns, with nothing phoenix specific touching anything from myapp. So in that sense the database layer has a ‘source code’ decoupling from the web stuff, but there is still a ‘project’ coupling in that they both live in the same mix project, and I suppose a sort of ‘run time’ coupling in that the ecto Repo and phoenix Endpoint are siblings under the same direct parent supervisor and run under the same otp application. Is the idea to move the database layer into it’s own otp application?

5 Likes

The push for contexts encourages folks to use discrete APIs exactly to decouple the two.

To a point. But the code still lives in the Phoenix app, so the presence of contexts actually encourages the exact behavior that is long-term harmful.

It seems to me that contexts are simply a way of creating service apis.

If I were writing an application that had services, but I included those services in one of the endpoint API modules, that would generally be considered to be a bad design. Why, for example, would a user creation service be part of my web API module when I would also use it from my command line and from my batch processing?

So I really think that offering contexts as part of Phoenix is wrong. It says that the Elixir community supports this kind of coupling.

But the reality is that the Elixir ecosystem is one of the few that actually supports the proper (and efficient) splitting of these responsibilities.

I really don’t buy the “ease-of-entry based on new user expectations” argument. If I did, I’d write a PHP interpreter as part of Phoenix. Instead, I think we make the entry easier by making it simpler. Give people a good architecture, where everything has it’s place, and where decoupling is the norm. The aim isn’t to write a 10 minute blog—it’s to write a five year application.

Parallel (not umbrella) applications are trivial to create and manage, and have decoupling built in from the start.

Except… this assumes that the Calendar context/service is part of the Phoenix app (assuming MyApp is one), and this just doesn’t make sense. Phoenix is not the application. It’s a way of accessing the application.

Dave

10 Likes

Isn’t this the reason for contexts though? Your application services aren’t included inside of your web api module, but rather inside of the appropriate context module. MyAppWeb.Controller.UserController.create calls into MyApp.Accounts.create_user. If you wanted to use the user creation service from the command line you could have Mix.Tasks.MyApp.CreateUser.run that also calls into MyApp.Accounts.create_user to do the actual work.

In that sense, the phoenix/mix edges of your application are essentially just i/o adapters that are translating inputs/ouputs from somewhere (http, stdin) into calls to the functional api that lives inside of the contexts. If you maintain strict segregation between your contexts and the i/o layers (which I think 1.3 encourages), then your application remains decoupled from phoenix.

That is the way that I think of it at least. I think of phoenix as just being a dependency for my elixir application that lives along side of it inside it’s own app_web world.

5 Likes

All the web things are in MyAppWeb namespace, MyApp has no web components.

2 Likes

The point still holds: it’s still one big application. It should be many.

2 Likes

It is really hard to agree that Phoenix v1.3 became more coupled to the database compared to previous versions. It has gone from an application that accessed the database in your controllers, channels and models to one that puts this access behind isolated modules with discrete APIs. Removing the database access in a Phoenix v1.3 app should require localized changes to your lib/app instead of rewriting most of your web layer.

It also puzzles me when someone says contexts are wrong because contexts are literally modules. You can disagree with the placement of those modules (in the same application instead of separate ones) but using modules to define our APIs is pretty much our bread and butter. This is supported by the fact you can now generate the web code that access any module and then just fill in the blanks. The blanks can be filled with a module in the same or in a separate application. Or even with a module that does not reach the database at all.

I agree Phoenix v1.3 does not completely push in the direction @pragdave would like to but it does make those patterns possible which is something we cannot say at all about previous versions.

I also have strong reservations about having different applications that depend on the same database instance, as those are effectively coupled at the operation level and you will run into issues depending on how you coordinate things such as migrations. I would like to see both development and operation concerns being taken into account on those discussions. But since I have already phrased those concerns elsewhere, I won’t rehash them here.

11 Likes

The fact they live in a different place doesn’t stop them being part of the same application codebase.

I agree that Phoenix is an I/O adapter. I don’t agree that the application code should share the same mix project (or even umbrella project). That would be like putting my text-based adventure game into the /dev/tty driver codebase… :slight_smile:

4 Likes

I don’t say contexts are wrong. I do say that encouraging new developers to place them inside the same project as the view layer is not a good idea. So in that way, 1.3 created a good idea (contexts/services) but then encouraged it to be used in a way that can lead to lots of coupling, and to Monorails. (And, as we’re discussed, umbrella projects don’t fix it.)

Saying “developers don’t have to use it” is dissembling when it is the default (and hence recommended) way that Phoenix apps are created.

Don’t get me wrong. I’m in awe of the amount of work that’s been done, and I’m very grateful to everyone. But I’m simply worried that this effort might not realize its full potential unless newcomers are shown that Phoenix is not a web framework—it isn’t Rails. Instead it is the web-tier (or, really, the TCP tier) in a multi-tier overall application.

And with that, I’ll sign out, because I respect the developers too much to spread bad karma.

Dave

8 Likes

Thanks for clarifying Dave! You said “offering contexts as part of Phoenix is wrong” which I read as “offering contexts as a concept in Phoenix” and it seems you meant “offering contexts directly in Phoenix and not in a separate app”. :sweat_smile:

4 Likes

Would parallel applications still run in the same BEAM, and if not, would you then create a client module and a server module, where by the client modules simply make :rpc calles to known servers? And the client and server code live in the same application, but when you need to use only the client you do not “start” the server?

Thanks,

Troy

2 Likes

I agree with this. This is how I develop my apps which are a bit larger. For smaller apps which are more web only I just put everything in one app (but of course still try to use the correct interfaces between my modules)

I tend to create an umbrella application with my application as one app and the phoenix web layer as another app.

I think it is great that Phoenix 1.3 is pushing towards this direction. For me it feels natural but I guess if you are used to Rails or Django which doesn’t usually have as clean interfaces perhaps it may look odd. I really like erlang’s -export directive as you really need to think on what functions you are exporting and it also gives a quick overview when you just open the files. Technically it is the same with def/defp but quickly look at a module to see what the interface is.

The cons to this approach I think is the same as the pros and cons of strong type systems (haskell, ocaml, rust). You need to think about your problem and problem domain in advance to get the right abstraction. This takes more time up front but saves you lots of time later. With dynamic languages you can get visible results much quicker but may end up with messy code that gets harder to refactor later.

I feel contexts are in the same problem domain as this. If you think first and get your contexts sorted out you wont be the first one out of the box but you will win the race

3 Likes

Could you provide a link? I remember reading about it or watching a video about it somewhere but I can’t find it.

2 Likes

Thank you everyone for taking part in the conversation :purple_heart:

Discussions like this are incredibly useful for developers like myself who are transitioning from the old ‘monolith’ way of doing things and are on a quest for something better.

I believe this (and finding it in the Elixir/Phoenix world) is actually a fantastic selling point - while many people may come for the speed, distribution, concurrency and fault tolerance I reckon they might stay because the eco-system pushes them towards good solid ‘modern’ approaches to building software that is resilient, less frustrating and scalable in every sense of the word. All the while making it fun (easy?) and interesting :slight_smile:


With that said, back to the topic. Although it’s very early for me to judge at present as I’ve still got 20% of the course to go through and we sadly won’t be using Ecto in the course (so I can’t yet fully picture a typical app) I am very excited by what I’ve seen so far. Much of what Dave has said seems to answer the burning question I often had when creating apps in other languages (that surely there is a better way of doing things).

Having said that, I can see a few challenges that might need to be overcome. For instance, if everything is created as separate services, and Phoenix should only be used for communicating via the web, then we might lose functionality or conveniences that Phoenix provides that we might want to use in the non-Phoenix part of our (overall) app - so those might need to be somehow moved from Phoenix or reworked in some way. The obvious one is validations (though I believe these are in Ecto - so maybe not an issue after all).

There’s probably more that I’m not aware of. José mentioned operational concerns and I would love to read about those too.


With regards to optimising for on-boarding new folks, I wonder if this could be worked around by having a --simple flag on app creation and where the generators reflect that (monolith?) ‘style’. This would explicitly let people know that it’s a learning tool, or one for quick prototyping or for creating the simplest of sites/apps. But perhaps more importantly it will spearhead a brave new direction, ‘the Elixir way’ of doing things. (I would actually be super excited by this! As if I could get any more excited about Elixir and Phoenix that is!)

The reason I think this is worth investigating was actually commented by Chris himself in his keynote, when he referred to @sasajuric’s feedback of where an expert team were somewhat led down a not ideal path because they thought the Phoenix way was what it was telling you to do via generators.


One final thing I would like to say is that it’s a very exciting time to be a developer! And an Elixir developer at that! I’m extremely happy to see that José, Chris and everyone in the Elixir, Phoenix and Ecto teams have this pursuit of reflection > perfection - and equally heartened to see that experienced members of the community are part of that transition, by feeling able to have conversations and inspire discussions like this.

:purple_heart:

8 Likes

*cough*ephp*cough*

But yeah, overall I agree with pragdave. And honestly, as I’ve mentioned before, I’m not even a fan of umbrella apps. Umbrella’s encourage a lot more interconnections between sections then what should really exist. I instead make my parts as lots of little dependencies that I then bring into a main app that does all the configuring and more. If I need a couple sections communicate between then I setup a channel between them at this point (not like a phoenix channel, but a set of behaviours in a sub project that both depend on, then setup their communication modules in the config). I find it keeps it very very clean and maintainable.

This precisely, this is what I do.

And this, 1.3 became less coupled to ecto, but I still do not like the monolithic apps that hold many (to use phoenix terms) contexts when a context should be its own standalone application/dependency.

I pass the Repo one of my dependencies should access into its config, which it then bakes in to their calls at build time. ^.^

That is really how it should be done I’d say, usually I pass the Repo module around as an argument, but currently I’m just using things like @Repo unquote(Application.get_env(:this_app, :repo) || throw ":this_app does not have :repo defined") in more places than I probably should. I should probably make a library to simplify a lot of this repetition too… >.>

Ditto, make lots of little applications and bring them in as dependencies.

Precisely this! They should be standalone dependencies as I’ve stated in that huge Contexts thread. ^.^;

They all run on the same beam. Even right now your beam instance is running probably a few dozen or more applications. :slight_smile:

6 Likes

It’s possible that we are overthinking this and if I’ve got to choose between one of these approaches, I think Phoenix is getting it right by encouraging and enabling separation rather than forcing it.

Completely separate applications come with a lot of hidden complexity that doesn’t generally need to be introduced until you really require it. The whole microservice vs monolith discussion generally boils down to isolation of dependency trees and resource consumption at its root. If a monolith is built with an object oriented language, you end up with this horrifying blur of inheritance chaining that’s incredibly messy to untangle. If you go microservices from the start it’s good long term but you pay for it up front with potentially unwarranted complexity.

With the approach from Phoenix and Umbrellas you don’t have a tangled mess, thanks to the way Elixir is designed. You’ve got smaller parts in their own zones that are significantly easier to separate when you finally NEED to.

Learning curve and on-boarding are a huge deal for a language. Determining trade-offs that get people moving quickly in the right direction is a key and it’s one of those things that is hard because you’re always sacrificing something. It’s something programmers debate about constantly because business factors like turnover, training, learning curve and time to market are real factors that matter beyond ideal architecture. Ideal architecture seems a lot easier when those other factors aren’t included IMO.

In that regard and from a community standpoint, I really do think that Phoenix 1.3 has nailed that balance. Like anything else in programming, it’s always a question of trade-offs and hidden costs.

If we aim for architectural purity instead of that balance, we aren’t going to be much better off than the “OMG Benchmarks!” crowd. If productivity, learning curve, training time, etc aren’t a factor then why aren’t we just using Erlang?

18 Likes

Just to mention: When you designing contexts as separate umbrella app it forces you to think about deps and configs.

4 Likes