Should we Separate Ecto From Phoenix in Umbrella App?

Hey, so we have an umbrella app, where the API is served by a Phoenix app, and then kick off a bunch of apps to perform application logic. The result from those calculations needs to be stored in the DB as well. So, for example:

  • User calls API and updates some information on his profile.
  • The data is stored in DB.
  • Another app in the umbrella is kicked off to perform some aync job (collect more data from external API, run some calculations, etc.)
  • Result from those calculations need to also be stored in the DB, but that module does not have the Repo as a dependency.

So I’m considering either:

  1. Separating Ecto from Phoenix, and have a “DB” app in the umbrella that handles all the db stuff, and then including that app in both the API and any other modules that need to talk to the DB as a dependency.

  2. Have the modules send a message back to Phoenix API that called it with its result, and then have the Phoenix app handle all db requests.

5 Likes

I am biased towards 1) but with an extra clause. I do not like to have my ORM modules/classes to have any logic in them. So they are basically definition of db schema + associations, with some generic changeset function. This extracts extremely well as this has no logic inside other than accessing/writing to the database.

As soon as you start putting some logic as in validations, custom changeset functions etc in those modules, you may be tying yourself up to individual applications that run on your umbrella app. I do not think that this is good practice, but I am more than expecting other people to disagree with me :D.

2 Likes

That is my preferred way of structuring apps as well. At least two apps in an umbrella - a domain one and an api one. First handles all the persistence and knows nothing of https, the second one calls plain function from the first one and knows nothing of ecto or any other persistence mechanism.

10 Likes

@chrismccord actually touched on this today at ElixirConf - he said that they’re doing away with the web folder entirely.

Ultimately lib will be your domain ecto etc. And inside you’ll have a web interface presenter. I’m sure he can expand on this much better than me.

But yeah web folder is kil

2 Likes

Very interesting. With the possibility of umbrella apps the current way of structuring an app felt somewhat wrong, but I didn’t wan’t to deviate too much from best practice :slight_smile:

Very excited about this new direction. For now, are there any sample gh repo’s that are already following this approach?

@sergio Are you sure that you interpreted him correctly? I can see with doing away with the web folder, but I’m not sure that implies moving away from umbrella apps. I would think it would seem more of doing away with the current (unnecessary?) complexity of the phoenix app folder structure. I just would find it hard to believe that he would be advocating for going against the umbrella app structure!

I didn’t mean to imply umbrella apps are not recommended anymore. I don’t see where I mention that…

To expand though, he said the web folder sent out the wrong message that you should put domain in your model modules. When that’s not the intent.

Your app, your secret sauce should be in the lib folder. And you’ll have a web visual interface that shows that stuff to end users.

1 Like

Ah, well I think it may have just been a little confusing, as the context of the thread is about separating ecto into a separate app in the umbrella, and as you stated:

But it sounds like what your saying is that any per-request compiled lib stuff that was in the web folder will be in the lib folder moving forward, i.e. controllers, phoenix-specific ecto, etc, and that the web folder will no longer have these?

Hi @michalmuskala It would be great to hear more about the way you separate your application this way.
I mean I looks very elegant solution for a very “presentation - meetup” problems.
I would like to know how/where you handle the validations, where you decide on the http error status. Because what I mean is that it’s nice to have your logic in separate app but
for me big part of API application it’s also the error handling, data validation, may be some request logging, sometimes taking in count the headers you receive in the request.
Thank you!

2 Likes

An Elixirconf presentation Building an Umbrella Project (slides). Cant find the video yet. Seems like a interesting example:

The example repo

Talks from elixirconf 2016 will be up at https://confreaks.tv/events/elixirconf2016 when the videos are available.

2 Likes

I’m simplifying here, but it seems that (based on the example repo), that the files normally put in Phoenix’ models directory are extracted and put in separate umbrella app(s).

When looking at this part, personally I would order this more strictly. Maybe grouping the models and the functions in different folders. Mentally it seems hard to keep track what is doing what. Any thoughts?

I’m experimenting with this now, and liking the following setup:

  • mix new project --umbrella
  • cd project/apps
  • mix new db --sup --module Project.DB
  • mix new app --sup --module Project.App or what have you for other otp apps
  • mix phoenix.new site --module Project.Site
  • Remove postgrex from site deps and applications, leave phoenix_ecto
  • Add {:db, in_umbrella: true} to site and other app deps, also add :db to the application :applications config in their mix.exss
  • Add ecto and postgrex to db deps and applications
  • mv site/lib/site/repo.ex db/lib/db/repo.ex
  • Rename all occurrences of Project.Site.Repo to Project.DB.Repo
  • mv site/priv/repo db/priv
  • mv site/web/models db/lib/db
  • Mirror phoenix’s config/<env>.exs files setup in db
  • Copy the /priv/static/ and /config/prod.secret.exs rules from site/.gitignore to db/.gitignore
  • Move ecto_repos configuration to db/config.exs
  • Move the repo adapter config in each site config/<env>.exs file to db
  • Make sure the "phoenix" and "phoenix_html" filepaths in your site/package.json's "dependencies" reference "file:../../deps/<dep>"
  • Stop using phoenix model generators, use mix ecto.gen.migration -r Project.DB.Repo from the db app
  • Freely access models, queries, etc from your other apps as Project.DB.<Model>

I think that’s about everything necessary that I did.

Further optional conveniences:

  • Rename databases from "site_<env>" in these adapter configs to "project_<env>"
  • Delete def model... from site/web/web.ex
  • Make a db/lib/db/model.ex with a __using__ macro with the quote from your Project.Site.Web.model you just deleted
  • Replace all occurrences of use Project.Site.Web, :model to use Project.DB.Model in db/lib/db/models
  • Move site/mix.exs's ecto aliases to db/mix.exs (optionally under a new db namespace instead of ecto)
  • Move site/mix.exs's test alias to the umbrella app’s mix.exs
  • I removed all of the references to Project.DB.Repo in site/web/web.ex and other Ecto stuff and instead made it so that you can use Project.DB.<Model> to get aliases for both that model and the repo, then used that instead atop my controllers and channels. Ditto with my Project.DB.<Query>s.
  • I changed all my otp app names to be prefaced with “project”, which requires changes in mix.exss, config/ files, db/lib/db/repo.ex, site/lib/site/endpoint.ex, and site/web/gettext.ex. You also have to rename your folders in the apps folder to follow the otp name, prefix and all, to avoid weird umbrella app and phoenix hot code reloading issues.
  • I experimented with moving everything from site/web into site/lib/site. You have to remove "web" from the site’s "elixirc_paths" and rename references to the web dir with lib/site in brunch stuff, and change the root of your templates within your site/lib/web.ex directory. However, this destroys hot code reloading so I can’t say I recommend it.
14 Likes

I personally like the idea to separate the ecto from the web app a lot. But I am going in another direction, where ecto will not be (in any way) used in the phoenix controllers layer.

What means that I’m extracting all my app business logic to an OTP application, and this is the layer that knows everything about how to save to the DB. The phoenix layer have just my controllers and view logic, and they depend from the business layer, and that’s it. They don’t even really know that there is a database or whatever saves the entities on the business layer.

5 Likes

That would be close the perfect standard setup IMO. Having Ecto as a standalone for purposes of migrations would also benefit from this setup.

1 Like

I really like that idea. Separating ecto and phoenix is where I’ve started (in my notes I mention how to remove all direct ecto references from the phoenix app) but I’d love to evolve a business layer in between without indirectly invoking ecto through the models. I’ve already started on a separate Auth app to hold my guardian and comeonin dependencies and mediate between the two, but haven’t dived in to how to cleanly do that for other things.

2 Likes

For a newbie in Elixir and Phoenix in general, like me, what the real benefits to gain starting out separating phoenix and business logic. Is not the Phoenix’s conventions (project structure and etc) good enough? And with the releasing of Phoenix 1.3 close, is there some concerns for this practice ? Thanks in advance.

2 Likes

I would love to read about your auth app as this is what I’m aiming to do once I have finished building my own integrated auth as part of learning about Elixir/Phoenix - separate it out.

There appears to be a growing number of posts with people wanting to break their apps into ‘micro apps/services’ that can be stopped/started/hosted independently to take advantage of the benefits of the OTP and scalability of Elixir/Phoenix, with each post touching on a part of this - from Ecto to auth.

Would be great to see a wiki entry/post on the ‘ideal’ architecture for 1-3 scenarios eg an app that could be as popular as Twitter etc - how to build an umbrella app for this scalability while being easy to maintain/backup/etc.

4 Likes

I’m trying a similar setup where I have 4 apps

  • api (graphql)
  • backend (sort of internal api)
  • catalog (elasticsearch)
  • persistence (ecto)

The api uses backend to retrieve and write data.
The backend uses catalog and persistence to retrieve and write data.

I’m using the backend app as an internal router (WIP)

https://gist.github.com/hl/00559a4d5070002aeb9cb58ba2fb8c3a

ps the naming does need some work but first I want to see if this setup will work in our case.

3 Likes

Actually, the phoenix conventions are very different from the Rails conventions! Rails enforces it’s conventions a lot, there is even “the rails way”. Phoenix’s conventions are very simple and they are more hands off than rails.

There is actually a new generator planned mix phoenix.new project --umbrella that will generate an umbrella project pretty much like the one we described here.

There’s a lot of people that are following this path, and the community itself is always concerned about umbrella projects, I guess that we could even consider this one of the phoenix conventions. I know it’s not a convention, hope you know what I mean! :wink:

5 Likes