Phoenix >= 1.3 umbrella architecture

Today I watched @chrismccord’s talk about the changes coming to Phoenix starting with version 1.3 and I really, really love the direction that Phoenix is going.

I have worked with both monoliths and microservices, and both have their known pros and cons. Phoenix Umbrella seems to be going in the right direction of providing best of both worlds: a strongly modular system, without the overhead of separate deployment.

One thing that I like about the --umbrella flag is that it completely separates the business layer from the web / presentation. But here I’d like to go even further. I would love to see an even more radical separation. Let me explain.

I have a web application that you can think of as a SaaS offering. It has a front page, pricing page, a login / signup form. After logging in, you get to see a dashboard that is a JS-app that communicates with a JSON/REST api. The REST API is the only source of data for the application and is both used by customers and the application itself.

Having this, I have created a Phoenix umbrella application structured the following way:

apps/my_app -> the business logic layer (encapsulates and abstracts the communication with the db)
apps/my_app_web -> the presentation layer for the HTML stuff
apps/my_app_api -> the API business logic layer (mostly communicates with my_app, exposes the models)
apps/my_app_api_web -> the API presentation layer for the JSON stuff

So the dependency tree looks like this:

my_app <- my_app_web (renders the assets)
my_app <- my_app_api <- my_app_api_web (exposes the REST api)
my_app_api <- my_app_web (uses the REST api)

The only way to achieve this structure was by creating the my_app umbrella project first, then the my_app_api umbrella second, and moving both my_app_api and my_app_api_web under the /apps folder of the my_app umbrella. I had to change the port of my_app_api_web to not collide with my_app_web. Not very clean, but hey, it works!

I’d love to see Phoenix evolving into a framework that naturally supports this kind of separation of concerns. Going further, I’d like to allow to have multiple “web” layers (e.g. public front-page vs application for logged in users, api, etc). This way the routers always stay clean and focused. The system naturally avoids growing into a bloated monolith.

Having a separate application for the API exposers might sound like an overkill, but I want to keep the business logic concerned with abstracting the DB in a clean way, and have the API layer be built on top of it. As an example, I have a lot of Swagger definitions in the API application, which I think does not strictly belong to the business logic.

Thanks again for making it possible, I’ve been waiting for this for so long!

3 Likes

Phoenix 1.3 also includes two new project generators for running inside an umbrella’s apps/ directory: mix phx.new.web and mix phx.new.ecto. So in your example, you could use the --umbrella flag, then generate your api, api_web, and other apps with these two generates. I think we already support what you suggested :slight_smile: Glad you like the direction!

9 Likes

Wow, this is great stuff! Keep up the good job :smiley:

2 Likes

@chrismccord is it possible to run, let’s say “app_web” and “app_api” in the same port? Perhaps using a proxy? How can one achieve this, any example?

Thanks in advance.

I stumbled on this github repo which has an example of what I’m trying to achieve, using the master_proxy app.

Is that example suitable for development? I mean will it have any negative impact, such as loss of live reloading or anything else?

Because I’d have to run it like: (right?)

cd apps/master_proxy && MIX_ENV=dev mix run --no-halt`

Thanks once more!

I wrote the master_proxy application originally to solve an issue with running an umbrella application on Heroku, where only one port is available.

What you have suggested would work, however I find it much easier to run mix phoenix.server from the route of the umbrella and have both applications running on different ports. So I can go to http://localhost:4000 for the API, and http://localhost:5000 for the web application.

1 Like

Thanks for the quick answer @Gazler!

In that case, to separate web and API, I think you’re right, and it’s more practical as it will probably run in a sub-domain (each app proxied by NGINX).

But imagine I have, let’s say, the web (application per se) and admin, which I’d like to separate in code, but to run on same port.

I’ve achieved this behavior by creating a mix phx.new myapp_admin --module MyApp.Admin application in /apps directory, removing the mod param from applications function in myapp_admin/mix.exs, adding this as a dependency of myapp_web and then forward "/admin" MyApp.Admin.Router.

However with this “solution” I’ve lost live reloading for the myapp_admin code and assets compilation…

Three questions, if you don’t mind:

  1. Do you think this is a good solution for this problem?
  2. How could I restore live reloading and assets compilation with this solution?
  3. For this specific case, are there better solutions?

Thank you in advance!

Well, I was able to make live reloading and assets compilation work again for the myapp_admin app, buy updating the following files in the myapp_web app:

File: myapp_web/config/dev.exs

...

config :myapp_web, MyApp.Web.Endpoint,
  watchers: [
    node: ["node_modules/brunch/bin/brunch", "watch", "--stdin", cd: Path.expand("../assets", __DIR__)], # watch myapp_web
    node: ["node_modules/brunch/bin/brunch", "watch", "--stdin", cd: Path.expand("../../myapp_admin/assets", __DIR__)] # watch myapp_admin
  ]

...

config :myapp_web, MyApp.Web.Endpoint,
  live_reload: [
    patterns: [
      # myapp_web
      ~r{priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$},
      ~r{priv/gettext/.*(po)$},                                                                                            
      ~r{lib/myapp_web/views/.*(ex)$},                                                                      
      ~r{lib/myapp_web/templates/.*(eex)$},

      # myapp_admin
      ~r{priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$},
      ~r{priv/gettext/.*(po)$},                                                                                            
      ~r{lib/myapp_admin/views/.*(ex)$},                                                                      
      ~r{lib/myapp_admin/templates/.*(eex)$},
    ]
  ]

File: myapp_web/lib/myapp_web/endpoint.ex

...

  # myapp_web
  plug Plug.Static,
    at: "/", from: :myapp_web, gzip:false,
    only: ~w(css fonts images js favicon.ico robots.txt)

  # myapp_admin
  plug Plug.Static,
    at: "/", from: :myapp_admin, gzip:false,
    only: ~w(css fonts images js favicon.ico robots.txt)

...

Ofc, this doesn’t look like a very elegant solution, but it works!!

I was wondering if there’s another way to work this out or improve this solution… Any ideas?

Thank you very much!

I can see why you would want them running on the same port in production (with some sort of proxy in front of them). I’m not sure why you would want to do it in development. If there is some reason that they need to exist together then they may not be good candidates for separating into different apps.

You an always run them both on separate ports to allow you to run mix phoenix.server from the root for the code reloading. Then you can forward as you have done already and just ignore the fact that the admin section is running on a different port. This means that you don’t need a work around like you have, but you can also run both applications in isolation.

If you look at the Phoenix supervisor, you will see that both the watcher_children (for code reloading) and server_children workers use the same server?/1 function to see if they should be started https://github.com/phoenixframework/phoenix/blob/4ff5a3e21ed8e1ca892e3bb60d3511d4cf8a2a1b/lib/phoenix/endpoint/supervisor.ex#L106L113

1 Like

All of this comes from the will to separate code as much as possible so that I don’t end up with a monolithic application. That’s why I want backoffice and frontoffice code to live in two separate “apps” or “directories”, as I would if I was to add an API to this project as well.

I now see that it really doesn’t matter if apps are running on different ports as long as I forward the app-sepcific path to correspondent router. I believe that approach will work well instead of the messy workaround I previously had in place.

However, when using that approach, as I’m calling the App.Backoffice.Router instead of calling it’s App.Backoffice.Endpoint, assets won’t be available if I’m accessing it through the forward. I mean, in development I’ll access localhost:4001 for developing the backoffice and I can access the assets. But in production it won’t be able to reach the assets, right? (at least this is what I’m experiencing right now)…

I don’t believe there is any reason that they need to exist together unless (I haven’t tried this yet) regarding authentication… Given this case where I separate both frontoffice and backoffice, given your experience, do you anticipate any issue with having authentication setup in both apps? As they’ll eventually have separated code for serialization and authentication it shouldn’t be a problem, right? But in the session-level do you think they could collide or something?

Sorry for all the questions! Thanks!

Hi,

Is it possible to add to https://hexdocs.pm/phoenix/heroku.html, a chapter about master proxy ?

My app has multiple phoenix app (scaffolded from phoenix 1.3 / different PORT) under the same umbrella and everything work in dev. And because I have to use heroku, a master_proxy is needed.

I have found using this forum the following gists: https://gist.github.com/alanpeabody/4fae12b420fb50376af4 and https://gist.github.com/ktec/ed2303eb167bd54ed1e390cd9371cc41

But something more official/easy to find would be cool (one working with websocket).

Thanks

Thanks @Gazler for your responses. I’m adding a +1 here because I’ve been trying to achieve the same result. My use case is that I want to build multiple apps, imagine a portfolio of sample projects or maybe micro-services, but as a developer I don’t want to create a new Digital Ocean droplet or AWS server or new Heroku app or dockerfile for each app. I want each App to almost be like a package in bigger App. Do you get what I mean? Do you have another way of achieving this? Most tutorials or application demos assume that they’re either monolithic or the only thing running on the server so they default to binding to port 80/443 etc. or default application entry point. Imagine a hosted app with ports in the domain name like google.com/gmail:4567 most people just want to see gmail.google.com and the port routing is handled somewhere… bonus points to Alphabet because even gmail.com works too. :smile:

Why can’t we have multiple apps running on the same server at different ports and being routed internally on the network or app layer through nginx or plug? I’ve tried a few different solutions but I can’t seem to have assets, websockets and APIs all working at the same time. Using plug and the host value to route requests still means that the assets aren’t delivered add plug proxy for holaWeb.Endpoint · iampeterbanjo/hello-proxy@a00b684 · GitHub and using Nginx means that the websocket requests don’t work hello-proxy/nginx/nginx.conf at master · iampeterbanjo/hello-proxy · GitHub

With Elixir umbrella projects it seems that we are so close to making this work… so close :slight_smile:

I have an issue open for the Acme Bank demo app that seems to have some other devs interested in this use case
https://github.com/wojtekmach/acme_bank/issues/14

@iampeterbanjo Pretty sure you need some additional settings for WebSockets and nignx (taken from https://www.nginx.com/blog/websocket-nginx/):

proxy_pass http://127.0.0.1:4010;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";

Nginx is the way I would go for hosting multiple applications on the same server.

3 Likes

Thanks

Nginx is the way to go, but if you’re stuck with managed hosting you have to stop using umbrella to manage multiple phoenix app (one for API, one for admin backend, etc).

I can’t find a managed hosting with postgresql and nginx.

Thanks for your answers.

1 Like

I struggled with master_proxy but finally made it work on my Phoenix 1.4 / cowboy 2.5 app.

It supports HTTPS and websockets properly, here is a gist: https://gist.github.com/cblavier/0c2cf3f101d82c32503774f179a66a3f

This may not be specifically applicable to this thread but if anyone is looking for an alternative to nginx as a proxy, we have successfully set up Traefik as a proxy to a phoenix 1.4 app.

Traefik will manage letsencrypt renewals and can detect internal resource changes without a restart.