Should we Separate Ecto From Phoenix in Umbrella App?

An Elixirconf presentation Building an Umbrella Project (slides). Cant find the video yet. Seems like a interesting example:

The example repo

Talks from elixirconf 2016 will be up at https://confreaks.tv/events/elixirconf2016 when the videos are available.

2 Likes

I’m simplifying here, but it seems that (based on the example repo), that the files normally put in Phoenix’ models directory are extracted and put in separate umbrella app(s).

When looking at this part, personally I would order this more strictly. Maybe grouping the models and the functions in different folders. Mentally it seems hard to keep track what is doing what. Any thoughts?

I’m experimenting with this now, and liking the following setup:

  • mix new project --umbrella
  • cd project/apps
  • mix new db --sup --module Project.DB
  • mix new app --sup --module Project.App or what have you for other otp apps
  • mix phoenix.new site --module Project.Site
  • Remove postgrex from site deps and applications, leave phoenix_ecto
  • Add {:db, in_umbrella: true} to site and other app deps, also add :db to the application :applications config in their mix.exss
  • Add ecto and postgrex to db deps and applications
  • mv site/lib/site/repo.ex db/lib/db/repo.ex
  • Rename all occurrences of Project.Site.Repo to Project.DB.Repo
  • mv site/priv/repo db/priv
  • mv site/web/models db/lib/db
  • Mirror phoenix’s config/<env>.exs files setup in db
  • Copy the /priv/static/ and /config/prod.secret.exs rules from site/.gitignore to db/.gitignore
  • Move ecto_repos configuration to db/config.exs
  • Move the repo adapter config in each site config/<env>.exs file to db
  • Make sure the "phoenix" and "phoenix_html" filepaths in your site/package.json's "dependencies" reference "file:../../deps/<dep>"
  • Stop using phoenix model generators, use mix ecto.gen.migration -r Project.DB.Repo from the db app
  • Freely access models, queries, etc from your other apps as Project.DB.<Model>

I think that’s about everything necessary that I did.

Further optional conveniences:

  • Rename databases from "site_<env>" in these adapter configs to "project_<env>"
  • Delete def model... from site/web/web.ex
  • Make a db/lib/db/model.ex with a __using__ macro with the quote from your Project.Site.Web.model you just deleted
  • Replace all occurrences of use Project.Site.Web, :model to use Project.DB.Model in db/lib/db/models
  • Move site/mix.exs's ecto aliases to db/mix.exs (optionally under a new db namespace instead of ecto)
  • Move site/mix.exs's test alias to the umbrella app’s mix.exs
  • I removed all of the references to Project.DB.Repo in site/web/web.ex and other Ecto stuff and instead made it so that you can use Project.DB.<Model> to get aliases for both that model and the repo, then used that instead atop my controllers and channels. Ditto with my Project.DB.<Query>s.
  • I changed all my otp app names to be prefaced with “project”, which requires changes in mix.exss, config/ files, db/lib/db/repo.ex, site/lib/site/endpoint.ex, and site/web/gettext.ex. You also have to rename your folders in the apps folder to follow the otp name, prefix and all, to avoid weird umbrella app and phoenix hot code reloading issues.
  • I experimented with moving everything from site/web into site/lib/site. You have to remove "web" from the site’s "elixirc_paths" and rename references to the web dir with lib/site in brunch stuff, and change the root of your templates within your site/lib/web.ex directory. However, this destroys hot code reloading so I can’t say I recommend it.
14 Likes

I personally like the idea to separate the ecto from the web app a lot. But I am going in another direction, where ecto will not be (in any way) used in the phoenix controllers layer.

What means that I’m extracting all my app business logic to an OTP application, and this is the layer that knows everything about how to save to the DB. The phoenix layer have just my controllers and view logic, and they depend from the business layer, and that’s it. They don’t even really know that there is a database or whatever saves the entities on the business layer.

5 Likes

That would be close the perfect standard setup IMO. Having Ecto as a standalone for purposes of migrations would also benefit from this setup.

1 Like

I really like that idea. Separating ecto and phoenix is where I’ve started (in my notes I mention how to remove all direct ecto references from the phoenix app) but I’d love to evolve a business layer in between without indirectly invoking ecto through the models. I’ve already started on a separate Auth app to hold my guardian and comeonin dependencies and mediate between the two, but haven’t dived in to how to cleanly do that for other things.

2 Likes

For a newbie in Elixir and Phoenix in general, like me, what the real benefits to gain starting out separating phoenix and business logic. Is not the Phoenix’s conventions (project structure and etc) good enough? And with the releasing of Phoenix 1.3 close, is there some concerns for this practice ? Thanks in advance.

2 Likes

I would love to read about your auth app as this is what I’m aiming to do once I have finished building my own integrated auth as part of learning about Elixir/Phoenix - separate it out.

There appears to be a growing number of posts with people wanting to break their apps into ‘micro apps/services’ that can be stopped/started/hosted independently to take advantage of the benefits of the OTP and scalability of Elixir/Phoenix, with each post touching on a part of this - from Ecto to auth.

Would be great to see a wiki entry/post on the ‘ideal’ architecture for 1-3 scenarios eg an app that could be as popular as Twitter etc - how to build an umbrella app for this scalability while being easy to maintain/backup/etc.

4 Likes

I’m trying a similar setup where I have 4 apps

  • api (graphql)
  • backend (sort of internal api)
  • catalog (elasticsearch)
  • persistence (ecto)

The api uses backend to retrieve and write data.
The backend uses catalog and persistence to retrieve and write data.

I’m using the backend app as an internal router (WIP)

https://gist.github.com/hl/00559a4d5070002aeb9cb58ba2fb8c3a

ps the naming does need some work but first I want to see if this setup will work in our case.

3 Likes

Actually, the phoenix conventions are very different from the Rails conventions! Rails enforces it’s conventions a lot, there is even “the rails way”. Phoenix’s conventions are very simple and they are more hands off than rails.

There is actually a new generator planned mix phoenix.new project --umbrella that will generate an umbrella project pretty much like the one we described here.

There’s a lot of people that are following this path, and the community itself is always concerned about umbrella projects, I guess that we could even consider this one of the phoenix conventions. I know it’s not a convention, hope you know what I mean! :wink:

5 Likes

Thank you very much @kelvinst. I wasn’t aware of this generator. This will be good for beginners that are looking for the “best patterns”. :slight_smile:

1 Like

Update: so yeah we did end up separating Ecto from Phoenix, and it was easy to do and easy to work with. There’s only some mild annoyances when running generators and stuff like that.

Now building my second Elixir project the same way. This time it’s a GraphQL API (using Absinthe) so there’s not even an HTML layer or controllers, making Phoenix a very thin layer for organising plugs and GraphQL schemas in a standard way really.

Here’s my short cheat sheet for starting projects this way: http://noamswebsite.com/wiki-main/computers/phoenix_no_ecto/

8 Likes

Did you try config :phoenix, :generators, model: false? I am not sure if it will work, but it is worth trying.

5 Likes

That does it, thanks :slight_smile:

following this set up, does it mean that the ecto schema and changesets are in the DB application? Or are they in the main business logic

1 Like

I am currently working with an umbrella app (one for business logic including ecto and on for phoenix) and it works fine.

But now I am thinking that it would be nice to put the BL and ecto back to the phoenix app. There are two reasons

  1. I really like to work with my test cases and code coverage to find spots where I need more tests and possible find dead code that is not used anymore. With the umbrella app my controller or hound tests will not recorded for my business app. Thats really bad.
  2. Lots of my code is vuejs/frontend stuff living in apps/web_app/web/static/js/whatever. So the dir structure of the whole project is getting more complex than it has to be (personal opinion)

So is there a strong point in staying with the umbrella app?

1 Like

I would also like to know the answer for this, since changesets can be used in both domains - Phoenix for rendering validation errors to users and with Ecto for validating data when inserting.

Maybe I am thinking about this wrong, still in the mindset of the tradition MVC application.

1 Like

I would put any shared data into it’s own library (or application if you are using an umbrella, I prefer libraries as there is a more strict separation and file deps update in realtime anyway), then just have both the DB work and the main application depend on that.

1 Like

I very like your approach, but there is a technical issue.
Basically phoenix context is a very convenient thing, it gives you separation of DB schema and business logic.
Once we separate DB, business logic and phoenix into 3 apps, will you say we basically might want to phx.gen.context and then move files to appropriate apps?