With Phoenix on a umbrella we lost the ability to join several functions together

Im porting a phoenix app to a umbrella, where I have a phoenix project and a domain project. Phoenix project dont “see” ecto, justo call domain modules.

I fully understand the advantages of decoupling these projects, but at the same time I ended up losing the ability to join several functions and modules to form a result. How have you deal with it?

example:

from phoenix without umbrella, I could do that on a controller:

assoc(user, :friendships)
|> Friendship.accepted
|> limit(5)
|> select([f], f.friend_id)
|> order_by([f], [desc: :id])
|> Repo.all

with phoenix into a umbrella:

Friendships.list_accepted(user, limit: 5, select: :friend_id, order: [desc: :id])

My concern is that in the end it seems that I’m doing “sql” in the function parameters, just to not use Ecto directly.

3 Likes

I’ve got a similar setup, and here’s my solution:

Umbrella app structure:

apps/
  db/
    lib/
      schemas/
        user.ex
        post.ex
    mix.exs
  api/
    web/
      controllers/
        user_controller.ex
        post_controller.ex
    mix.exs

In apps/api/mix.exs (your api app depends on your db app):

# ...
defp deps do 
  {:db, in_umbrella: true},
end
# ...

In apps/db/lib/schemas/post.ex for example:

defmodule DB.Post do
  use Ecto.Schema
  import Ecto.Query

  schema "posts" do
    belongs_to :author, DB.User
    field :slug, :string
    field :title, :string
    # ...
    timestamps
  end

  def changeset(post, params \\ %{}) do
    # ...
  end

  def where_author_id(query, user_id) do
    from p in query,
      where: p.author_id == ^user_id
  end

  def where_slug(query, slug) do
    from p in query,
      where: p.slug == ^slug
  end

  def sort(query) do
    from p in query,
      order_by: [desc: p.inserted_at]
  end
end

Then, in your controllers, you can just do this:

alias DB.{Post, Repo}
# ...
def index(conn, _params) do
  Post
  |> Post.sort()
  |> Repo.all()
end

def show(conn, %{"slug" => slug}) do
  Post
  |> Post.where_slug(slug)
  |> Repo.one()
end

def author_index(conn, %{"author_id" => author_id}) do
  Post
  |> Post.where_author_id(author_id)
  |> Post.sort()
  |> Repo.all()
end
# ...

As you can see, the queries compose together quite nicely. Tip: Keep any side effects out of your schemas (e.g. don’t use Repo in schema modules). All those pure functions make it really nice to compose things together.

Also, because your api app depends on your db app, and your db app depends on ecto, you actually do have access to Ecto.* modules in your api app. So, in your controllers, you can still do things like pattern match on %Ecto.Changeset{}.

5 Likes

Hi, thanks

But what I understood about this post: How to prepare for Phoenix 1.3? and some videos about “phoenix is not your app” is that you shoulnt use Ecto/Repo into your controller… Am i wrong?

2 Likes

In my experience, it is pretty easy to grow a service layer when your logic gets complex enough to have one.

So, for example, you might have a setup like this:

apps/
  api/
  db/
  post_service/

Where api is just Phoenix controllers and views, db is just Ecto.Schema definitions + pure functions, and post_service might be where your business logic is contained. So then api would call functions in post_service, and handle the results of those.

My very humble opinion: that’s a lot of overhead for something as simple as a blog (in my example), and something you could very easily refactor as complexity increases. All you would have to do is move your dependency on db to your service layer.

Now, something I do really like to do from the beginning is build a set of modules around Ecto.Multi. Again, these functions are pure and simply return an %Ecto.Multi{} struct that you can run through Repo.transaction/1 in your controllers or elsewhere. See this post for more info on that: http://blog.danielberkompas.com/2016/09/27/ecto-multi-services.html

6 Likes

If anyone else has any input, I’d love to hear if I’m totally off base. I will say that this has been working pretty well for me for a few months now. The key I’ve found is that Elixir is so easy to refactor (especially when you have a lot of pure functions) that you don’t need to over-organize and over-abstract too early. If you logic is simple, keep it simple. Once it gets complex, you can easily move it to it’s own app in your umbrella project, and abstract away.

1 Like

True, but that kind of defeats the purpose of having them separated in the first place, no? :slight_smile:

I think that one of the advantage of separating it is that you can design by domain-specific contract. For example your snippet here:

Friendships.list_accepted(user, limit: 5, select: :friend_id, order: [desc: :id])

Do all the parameters (like limit: 5, select: :friend_id) really need to be dynamic? Can you just limit the accepted parameters to be as minimal as necessary?

If you really want to do composition, @pdilyard’s solution is quite reasonable, although I’m not sure I like to still expose the Repo to the controller :slight_smile:

I myself have a habit of only having only one data call (such as the Friendships.list_accepted) in every controller, so the query composition, if any, happens inside that function in Friendships module. The controller doesn’t need to know any that. Of course I might be wrong, though.

This! I need to keep reminding myself of this often.

3 Likes

I think something like this is good, but I would not put it in the same modules as your schema. I would still recommend having some separate place for functions with side effects, (even if it is just a separate module, e.g. FriendshipService), that composes some pure functions together and then calls Repo.*.

So you might then have:

defmodule DB.Friendship do
  schema "friendships" do
    # ...
  end

  def limit(n) do
    from f in query,
      limit: n
  end

  # more query functions...
end

and

defmodule DB.FriendshipService do
  alias DB.Friendship
  import DB.Friendship

  def list(...) do
    Friendship
    |> limit(5)
    |> Repo.all()
  end
end

and call it in your controllers with Friendship.list(...)

2 Likes

In my case, it already is a large app. I ported from a monilitic monster on rails to a monolitic monster on elixir :wink: . Now I just trying to figure out how to separete it in microservices that is easy to maintain.

The ideia to keep controllers without Repo/Ecto is working very nice 90% of time, just a few controllers it is making things a little hard.

1 Like

Ah, yes, I forgot to mention this. I do keep them in separate modules, in my case for example it’s MyApp.Users and MyApp.Schema.Users.

3 Likes

Just my 2 cents on this subject.

I am porting a PHP monster to Elixir, and splitting it into different apps in an umbrella at the same time.

At first, we where following the “No Ecto in Phoenix” approach, but it was getting somewhat hard pretty fast. i.e. We have a lot of APIs which can be filtered by many different parameters and most of them are optional, so a lot of “Web related” issues (like pagination, receiving a map with string keys instead of atoms…) were flowing into the business domain app (the one with Ecto and stuff). So, while the controllers were getting pretty simple, the services were becoming complex with all the validation and checking.

IMHO, the purpose of the umbrella approach to building an app, is to make it easy to maintain it latter on and “keep the house in order”. So if the “No Ecto in Phoenix” is actually making it difficult to add functionality (when writing a “CRUD monster”, writing a simple create or update operation cannot take too long), it may not be the right way to do it.

What we did was establish that there will be no business logic whatsoever on the “web” app:

  • Queries are all built on the “domain” app;
  • Operations on the DB are created on the “domain” app via Multis (makes it easier to add more steps to an operation later);
  • Multis and queries are executed on the “web” side, or any other app that depends on the “domain” app;

This way we got some nice perks, like:

  • All the functions on the “domain” app are pure and easily tested;
  • In APIs where I need two (or more) views on the same data (i.e. a list and a sum) I don’t need to build two queries. I just build the base query and compose the select part differently;
  • Still get the separation of concerns, all the operations and queries are build in the “domain” app.

The main downside of this approach is that, if somehow in the future we choose to use some other kind of storage that does not uses Ecto, we are going to have problems. But given the nature of our application, I don’t think that’s is going to be a problem.

There may be other downsides to this approach, but it’s working pretty well for us so far ^^

5 Likes

It’s very interesting to read this, because my experience is exactly the opposite. Initially we had functions returning changesets (multis weren’t a thing back then) and exposed query bits as functions, so you could easily compose them in the controllers.

This proved to be not such a great idea in the long run for several reasons:

  • we have several endpoints where the JSON data doesn’t map 1-to-1 to the ecto schemas, so there are transformations on both data going into the database and out, and errors returned from validations (in case the differences are big enough we have some raw schemas acting as a sort of “form models”).
  • with multis you most often need to format the result in some way, having this done in the controller felt “dirty”.
  • we have two repos and two databases, so we want to keep the knowledge of what goes where outside of the controllers
  • it became really hard to do some one-off actions by hand.
  • there was a lot of logic in the controllers - what functions to call in what order, especially around the queries became problematic.

We migrated to the “one function per use case” style and it works much better. It’s immediately clear from looking at the controller what is happening (since those functions have meaningful names), and all controllers became very simple - matching on the result of the use case and dispatching appropriately.
I don’t agree that passing “raw” maps as options to those functions is http leaking into the system - anything that is not beam will give you “raw” arguments with strings instead of atoms, etc. If you provided a CLI interface you’d also get those.
I usually try to style all the domain “mutation” functions in a similar way:

foo(resource_or_id, params, actor)

Each of those functions may call other components to do the “dirty” work, but they all return {:ok, resource} or {:error, map}, where the errors map is already properly converted, processed and ready for rendering.

13 Likes

Thanks a lot for the reply, it’s awesome to see all the different positioning over the same issue, that’s gold! :relaxed:

I still didn’t had a chance to use this “form models” of Ecto (the endpoints are pretty much standardized, “Monster CRUD” app :sweat_smile:), but I’m looking forward for a chance of using them in our structure (maybe have Ecto as dependency of the Web app just as a tool to process the forms?).

To me it actually felt “dirty” doing so in the domain layer. It made more sense to leave the presentation logic to the web app, since, e.g., a background app that uses the domain layer would not need the same presentation logic.

Now that is some hard logic to go in the controllers. In our case there is a single DB/Repo, so we didn’t had this kind of problem to solve when deciding how to split the logic. Probably if we had this same issue the decision could be a lot different :smiley:

Actually we didn’t face this issue because the services encapsulated all the logic, the controllers just parse the params the way the service required (e.g. "YYYY-MM-DD" to %Date{}) and then execute the Multi, deciding what to return to the user in case of success or errors.

The querying part can became real messy real fast, I can only agree to that :smile:
What I did is, abstract the query composing logic via special “querier modules” (if someone have a better name for this, please help me :pray:). They basically define functions that take a query and a parameter and then extend the query accordingly.
Then I used some simple macro magic to define a function in each of these modules called build_query, that take a map with atom keys or keyword list where the key name is the function and the value is the parameter. Afterwards I build the query ignoring the nil parameter values (since Ecto does not suport nil parameters in queries).
This allows me to write this kind of code: SomeQuerier.build_query(filter1: params["param1"], filter2: params["param2"], ...) |> Repo.(all|one|aggregate|...) and have composable, dynamic, parameter based queries without too much fuzz on the controllers and still making being able to use it in a background like app without even noticing that there is some sort of communication with the outside world the uses the same code.

I agree that it’s not HTTP leaking into the system, but it’s like outside communication leaking into the system (a CLI, some external API), because the Elixir default to options or params is to use atom keys, and since we can’t trust outside of the BEAM information, we must use string keys so we don’t overflow the atom storage.
But a background app (some of them) does not have such concerns, it builds it’s own parameter maps without any interference from the outside world, so it doesn’t have a problem in using the Elixir default: atom keys.
So imagine a engineer that works only on one of those background apps. He would ask “Why do all these service calls use strings as param keys? Why not use atoms?”.
IMHO, if something is in an app, just because of some of the apps that depends on it, that’s enough leakage.

But then again, in our case we have a lot of these “background apps”, so in a lot of other cases this might not even apply :smile:


Now, I don’t think that how I did is the absolutely right choice, but I don’t think that it’s wrong either. Like a lot of people say, there’s no such thing as a silver bullet, that solves every possible problem we might find. We must find the right balance between pros and cons.

And I find it awesome that the Elixir land gives us so much options to better structure and scale our applications.

1 Like

I’ve started to refactor into a similar setup as @michalmuskala. I have the following setup in an umbrella app now:
db - Contains nothing but schemas and functions for composing queries (no side effects here)
services - All the real work is done here. Domain logic, database Repo.* functions, etc. Modules that build up Multis also live in this app.
api - The Phoenix app - with only controllers and views. I try to think of every controller action as doing the following: receive input, call a service, pattern match on the result, dispatch to view.

So, here’s an updated blog post implementation with this architecture:

# apps/db/post.ex
defmodule DB.Post do
  use Ecto.Schema
  import Ecto.{Changeset, Query}

  schema "posts" do
    belongs_to :user, DB.User
    field :title, :string
    field :body, :string
    timestamps
  end

  def changeset(post, params \\ %{}) do
    post
    |> cast(params, [:title, :body])
    |> validate_required([:title])
  end

  def where_title(query, title) do
    from p in query,
      where: p.title == ^title
  end
end
# apps/services/post_service.ex
defmodule Services.PostService do
  alias DB.{Post, Repo}

  @spec create(%DB.User{}, map) :: {:ok, %DB.Post{}} | {:error, %Ecto.Changeset{}}
  def create(user, params) do
    %Post{user_id: user.id}
    |> Post.changeset(params)
    |> Repo.insert()
  end
end
# apps/api/post_controller.ex
defmodule API.PostController do
  use API.Web, :controller
  alias Services.PostService

  def create(conn, params, user) do
    case Post.create(user, params) do
      {:ok, post} ->
        conn
        |> put_status(:created)
        |> render("post.json", post: post)

      {:error, changeset} ->
        conn
        |> put_status(:bad_request)
        |> render(ErrorView, "changeset.json", cs: changeset)
    end
  end
end

Want to create a post for a user from the REPL? It’s as easy as call Services.PostService.create/2, and you’re guaranteed to have all functionality from your controller wrapped up in one place.

Want to do something more complex? You’ve still used as many pure functions as possible to build up your service layer, so there’s no reason you can’t compose those functions differently to make more functionality.

We have one service that is substantially larger than the rest (and uses a lot of OTP features), so we pulled it out into it’s own app in the umbrella.

Note that refactoring to this service based organization was as easy as pulling out functionality from controllers, and replacing the use of conn with easy-to-pattern-match tuples, and throwing @specs in for easy future reference.

I’d love to hear if anyone has any feedback on this setup.

5 Likes

I end up with a pretty close solution, I have 2 projects into a umbrella, one for schema/squeries/services and other to phoenix.

activities/
-- activity.ex #schema/queries
-- activities.ex #on plural, service

Into activities I have (for now, I’m raising a exception if id not exist or current_user is not authorized):

defmodule App.Activities do 
  def update(id, params, current_user) do
    Activity
    |> Repo.get!(id) 
    |> authorize!(current_user)
    |> Activity.changeset(params)
    |> Repo.update
  end
end

Into controller:

def update(conn, %{"id" => id, "activity" => a_params}, current_user) do 
  with {:ok, activity} <- Activities.update(id, a_params, current_user) do 
    render conn, activity: activity
  end
end

I overrided phoenix action() to deal with {:error, :anything} in a global way.

My problem now is:
Some services generate “notifications” that need to be send to browser over websocket.
I think that I’ll create a new project that listen postgres pub/sub
or use GenStage in some way (services send data to a GenStage and on phoenix I have consumers)…

3 Likes

This structure look’s really awesome. Would you mind to clean this version and publish it on Github maybe? :slight_smile: I would love to reuse it.

2 Likes

How about this, just to decouple stuff a bit

defmodule App.Activities do 
  def update(id, params, current_user) do
    Activity
    |> Repo.get!(id) 
    |> Activity.changeset(params)
    |> Repo.update
  end
end
def update(conn, %{"id" => id, "activity" => a_params}, current_user) do 
  with :ok <- Authorizer.authorize_user(:update_activities, current_user),
       {:ok, activity} <- Activities.update(id, a_params) do 
    render conn, activity: activity
  end
end
defmodule Authorizer do
  def authorize_user(:insert_activities, nil), do: {:error, :unauthorized}
  def authorize_user(:insert_activities, user) do
    # check user
  end

  def authorize_user(:update_activities, nil), do: {:error, :unauthorized}
  def authorize_user(:update_activities, user) do
    # check user
  end
end
2 Likes

My umbrella:
|
|---- JSON API(Phoenix, no Ecto inside)
|---- Another Phoenix app for a couple of front end tasks. There is no web front end yet, so this app just handling different tasks like email confirmation via browser, password reset etc. No Ecto.
Both Phoenix apps communicate. Just a little. But they are friends not for a long time, I believe))

All other apps doesn’t have Phoenix inside
JSON API runs
|---- Auth app with ecto, separate db and API
|---- Main app with ecto, separate db and API
|---- Social app which runs tasks like Facebook graph queries. No Ecto just API
All that APIs are Plain Old GenServers(POGS)

Front end app runs
|---- Auth app with Ecto and API. This is the same app that is in use of JSON API app but started with different supervisor. And nothing else in this app because all required information is here in Auth.

All that zoo works fine for now. I can easily replace any of this microservices so I believe there is no any difference if you run Phoenix under umbrella, or Phoenix with Ecto, or without, or with friends. Just isolate all your queries to database(s) into different OTP apps and make them talk to each other.

4 Likes

I’ve been looking at doing something like this, and this post has given me some great ideas!

One thing that Im not sure on, is that they all seem to suggest keeping all the schemas in the DB app.
I was thinking more of having ‘isolated’ apps that have all related items grouped together.

So, have a ‘data’ app, that has the DB Repo in it (and any other data related things, redis etc), then multiple ‘feature’ apps - ‘customers’, ‘invoices’, ‘posts’ etc which have their own schemas, migrations tasks etc but all use the same Data.Repo.

That way I don’t need to edit the ‘data’ app as well as the ‘invoices’ app for instance, and things related to ‘invoices’ are all in the same place, rather than in different apps.

Would something like that work? Or am I missing something about having all the schemas living under the DB app?

2 Likes

That sounds like an interesting idea, I’d be curious to see what that would look like when your app starts to get complex, or when a feature depends on another.

1 Like

I’m not really sure to be honest, just playing with some ideas.

I’d thought about exposing a service layer in each app and then calling those from other apps, but the more I think about it, the more I think it would just be more complicated.

If I wanted to use absinthe with it - as another app - then I think I would lose the nice features of absinthe_ecto by having the resolvers calling services rather than the DB directly.

1 Like