Hey, so we have an umbrella app, where the API is served by a Phoenix app, and then kick off a bunch of apps to perform application logic. The result from those calculations needs to be stored in the DB as well. So, for example:
User calls API and updates some information on his profile.
The data is stored in DB.
Another app in the umbrella is kicked off to perform some aync job (collect more data from external API, run some calculations, etc.)
Result from those calculations need to also be stored in the DB, but that module does not have the Repo as a dependency.
So Iâm considering either:
Separating Ecto from Phoenix, and have a âDBâ app in the umbrella that handles all the db stuff, and then including that app in both the API and any other modules that need to talk to the DB as a dependency.
Have the modules send a message back to Phoenix API that called it with its result, and then have the Phoenix app handle all db requests.
I am biased towards 1) but with an extra clause. I do not like to have my ORM modules/classes to have any logic in them. So they are basically definition of db schema + associations, with some generic changeset function. This extracts extremely well as this has no logic inside other than accessing/writing to the database.
As soon as you start putting some logic as in validations, custom changeset functions etc in those modules, you may be tying yourself up to individual applications that run on your umbrella app. I do not think that this is good practice, but I am more than expecting other people to disagree with me :D.
That is my preferred way of structuring apps as well. At least two apps in an umbrella - a domain one and an api one. First handles all the persistence and knows nothing of https, the second one calls plain function from the first one and knows nothing of ecto or any other persistence mechanism.
Very interesting. With the possibility of umbrella apps the current way of structuring an app felt somewhat wrong, but I didnât wanât to deviate too much from best practice
Very excited about this new direction. For now, are there any sample gh repoâs that are already following this approach?
@sergio Are you sure that you interpreted him correctly? I can see with doing away with the web folder, but Iâm not sure that implies moving away from umbrella apps. I would think it would seem more of doing away with the current (unnecessary?) complexity of the phoenix app folder structure. I just would find it hard to believe that he would be advocating for going against the umbrella app structure!
Ah, well I think it may have just been a little confusing, as the context of the thread is about separating ecto into a separate app in the umbrella, and as you stated:
But it sounds like what your saying is that any per-request compiled lib stuff that was in the web folder will be in the lib folder moving forward, i.e. controllers, phoenix-specific ecto, etc, and that the web folder will no longer have these?
Hi @michalmuskala It would be great to hear more about the way you separate your application this way.
I mean I looks very elegant solution for a very âpresentation - meetupâ problems.
I would like to know how/where you handle the validations, where you decide on the http error status. Because what I mean is that itâs nice to have your logic in separate app but
for me big part of API application itâs also the error handling, data validation, may be some request logging, sometimes taking in count the headers you receive in the request.
Thank you!
Iâm simplifying here, but it seems that (based on the example repo), that the files normally put in Phoenixâ models directory are extracted and put in separate umbrella app(s).
When looking at this part, personally I would order this more strictly. Maybe grouping the models and the functions in different folders. Mentally it seems hard to keep track what is doing what. Any thoughts?
Iâm experimenting with this now, and liking the following setup:
mix new project --umbrella
cd project/apps
mix new db --sup --module Project.DB
mix new app --sup --module Project.App or what have you for other otp apps
mix phoenix.new site --module Project.Site
Remove postgrex from site deps and applications, leave phoenix_ecto
Add {:db, in_umbrella: true} to site and other app deps, also add :db to the application:applications config in their mix.exss
Add ecto and postgrex to db deps and applications
mv site/lib/site/repo.ex db/lib/db/repo.ex
Rename all occurrences of Project.Site.Repo to Project.DB.Repo
mv site/priv/repo db/priv
mv site/web/models db/lib/db
Mirror phoenixâs config/<env>.exs files setup in db
Copy the /priv/static/ and /config/prod.secret.exs rules from site/.gitignore to db/.gitignore
Move ecto_repos configuration to db/config.exs
Move the repo adapter config in each site config/<env>.exs file to db
Make sure the "phoenix" and "phoenix_html" filepaths in your site/package.json's "dependencies" reference "file:../../deps/<dep>"
Stop using phoenix model generators, use mix ecto.gen.migration -r Project.DB.Repo from the db app
Freely access models, queries, etc from your other apps as Project.DB.<Model>
I think thatâs about everything necessary that I did.
Further optional conveniences:
Rename databases from "site_<env>" in these adapter configs to "project_<env>"
Delete def model... from site/web/web.ex
Make a db/lib/db/model.ex with a __using__ macro with the quote from your Project.Site.Web.model you just deleted
Replace all occurrences of use Project.Site.Web, :model to use Project.DB.Model in db/lib/db/models
Move site/mix.exs's ecto aliases to db/mix.exs (optionally under a new db namespace instead of ecto)
Move site/mix.exs's test alias to the umbrella appâs mix.exs
I removed all of the references to Project.DB.Repo in site/web/web.ex and other Ecto stuff and instead made it so that you can use Project.DB.<Model> to get aliases for both that model and the repo, then used that instead atop my controllers and channels. Ditto with my Project.DB.<Query>s.
I changed all my otp app names to be prefaced with âprojectâ, which requires changes in mix.exss, config/ files, db/lib/db/repo.ex, site/lib/site/endpoint.ex, and site/web/gettext.ex. You also have to rename your folders in the apps folder to follow the otp name, prefix and all, to avoid weird umbrella app and phoenix hot code reloading issues.
I experimented with moving everything from site/web into site/lib/site. You have to remove "web" from the siteâs "elixirc_paths" and rename references to the web dir with lib/site in brunch stuff, and change the root of your templates within your site/lib/web.ex directory. However, this destroys hot code reloading so I canât say I recommend it.
I personally like the idea to separate the ecto from the web app a lot. But I am going in another direction, where ecto will not be (in any way) used in the phoenix controllers layer.
What means that Iâm extracting all my app business logic to an OTP application, and this is the layer that knows everything about how to save to the DB. The phoenix layer have just my controllers and view logic, and they depend from the business layer, and thatâs it. They donât even really know that there is a database or whatever saves the entities on the business layer.
I really like that idea. Separating ecto and phoenix is where Iâve started (in my notes I mention how to remove all direct ecto references from the phoenix app) but Iâd love to evolve a business layer in between without indirectly invoking ecto through the models. Iâve already started on a separate Auth app to hold my guardian and comeonin dependencies and mediate between the two, but havenât dived in to how to cleanly do that for other things.
For a newbie in Elixir and Phoenix in general, like me, what the real benefits to gain starting out separating phoenix and business logic. Is not the Phoenixâs conventions (project structure and etc) good enough? And with the releasing of Phoenix 1.3 close, is there some concerns for this practice ? Thanks in advance.
I would love to read about your auth app as this is what Iâm aiming to do once I have finished building my own integrated auth as part of learning about Elixir/Phoenix - separate it out.
There appears to be a growing number of posts with people wanting to break their apps into âmicro apps/servicesâ that can be stopped/started/hosted independently to take advantage of the benefits of the OTP and scalability of Elixir/Phoenix, with each post touching on a part of this - from Ecto to auth.
Would be great to see a wiki entry/post on the âidealâ architecture for 1-3 scenarios eg an app that could be as popular as Twitter etc - how to build an umbrella app for this scalability while being easy to maintain/backup/etc.
Actually, the phoenix conventions are very different from the Rails conventions! Rails enforces itâs conventions a lot, there is even âthe rails wayâ. Phoenixâs conventions are very simple and they are more hands off than rails.
There is actually a new generator planned mix phoenix.new project --umbrella that will generate an umbrella project pretty much like the one we described here.
Thereâs a lot of people that are following this path, and the community itself is always concerned about umbrella projects, I guess that we could even consider this one of the phoenix conventions. I know itâs not a convention, hope you know what I mean!
