Ecto is not your application: how to register your persistence application?

Hello,

Today, I have been reading a lot about “Phoenix is not your application”. With everything conceptually centered around the talk Architecture: The Lost Years (Robert C. Martin), I bumped into these Elixir specific resources on applying it with an umbrella app:

In the last article, he refers to an example app on Github in which I think he got most right, except the direction of the dependencies between the service and the DB apps. If I understood it correctly from Robert’s talk, the service - containing the business logic - should be presenter and persistence agnostic. I would expect that the service app should only have dependencies to the db_behaviour modules, but not to the db modules. Am I right on this assumption?

If so, the question then remains: how to fix this? How to set up this umbrella app in such a way that service and restaurants_db both depend on restaurants_db_behaviour, but the service not directly on the implementation while calls from the service to the gateway behaviour are correctly routed to the implementation?

Any example code available?

Ringo

5 Likes

Hi Ringo,
I also was inspired by Robert’s presentation and wrote those two posts that you mentioned. I’m glad you liked them.
I updated the code from my repo from that example application with a few fixes. Please check it out.

Going back to your question in Elixir if you want to access a module from another umbrella application you need to put it as a dependency inside mix. So we need to add as a dependency restaurants_db_behaviour and also the implementation restaurants_db.

We can change the dependency using config variables as below.

Inside config.exs:

config :service,
   restaurants_db: RestaurantsDb,
   users_db: UsersDb,
   shopping_db: ShoppingDb

In the test.exs:

config :service,
  restaurants_db: RestaurantsDbMock,
  users_db: UsersDbMock,
  shopping_db: ShoppingDbMock

As you see changing the implementation means just updating this files. For test env I use mox library to create mock expectations over behaviours. In test_helper.ex:

Mox.defmock(RestaurantsDbMock, for: RestaurantsDbBehaviour)
Mox.defmock(UsersDbMock, for: UsersDbBehaviour)
Mox.defmock(ShoppingDbMock, for: ShoppingDbBehaviour)
ExUnit.start()

Also another trick that I do is to not hardcode the service at compile time in a module attribute. I want to be able to switch the implementation at runtime (for example to do integration tests) so I am loading the implementation via a private method:

  defp restaurants_db, do: Application.get_env(:service, :restaurants_db)

As a personal note: one thing that we loose via dependency injection is autocomplete in editors. Since we do not call the module upfront it does not know what methods are inside. I would loved to be able to have some type inference here, to be able to specify the behaviour for example and to have autocomplete.

2 Likes

Hey Ringo,

I’ve just done something similar for our elixircards app - we have an app for authentication and I wanted to separate it from our accounts app - and in tests I didn’t want to hit the database - so here’s roughly what we came up with:

store

  • has StoreBehaviour.ex
  • has InMemoryStore.ex which is a GenServer implementation for testing purposes

auth

  • depends on “store”
  • needs to use a data store
  • in tests it uses the in_memory store provided by “store”
  • in dev/prod it uses the real store, eg. accounts

accounts

  • depends on “store”
  • needs to provide a data store implementation
  • the api conforms to “StoreBehaviour”

Here’s some code snippets:

defmodule Store.Behaviour do
  @moduledoc """
  This module defines the interface we expect all stores to support.
  By using this approach you can decouple your tests from your datastore

  defmodule Store do
    @behaviour Store.Behaviour
    ...
  end
  """

  @type behaviour_impl :: {module(), module() | pid()}

  @callback get(behaviour_impl, args :: Number.t) :: map()
  @callback get_by(behaviour_impl, args :: Keyword.t()) :: map()
  @callback get_all(behaviour_impl) :: list(map())
  @callback insert(behaviour_impl, args :: map()) :: map()

  @spec get(behaviour_impl, args :: Keyword.t) :: map()
  def get({impl, mod_or_pid}, args), do: apply(impl, :get, [mod_or_pid, args])

  @spec get_by(behaviour_impl, args :: Keyword.t()) :: map()
  def get_by({impl, mod_or_pid}, args), do: apply(impl, :get_by, [mod_or_pid, args])

  @spec get_all(behaviour_impl) :: list(map())
  def get_all({impl, mod_or_pid}), do: apply(impl, :get_all, [mod_or_pid])

  @spec insert(behaviour_impl, args :: Keyword.t()) :: {:ok, map()} | {:error, reason :: any()}
  def insert({impl, mod_or_pid}, args), do: apply(impl, :insert, [mod_or_pid, args])
end

The InMemory store is just an implementation of this behaviour using GenServer to persist the data:

defmodule Store.InMemory do
  @moduledoc """
  This is an in memory implementation of a datastore backed by a GenServer
  and a list of data.
  """

  @behaviour Store.Behaviour

  use GenServer

  @doc """
  Starts the process with an empty list by default
  """
  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, [], opts)
  end

  @doc """
  Insert an item into the store.
  """
  @impl Store.Behaviour
  def insert(pid, item) do
    GenServer.call(pid, {:insert, item})
  end

  # ...

end

And now in “auth” app we can make calls via StoreBehaviour. Note here we’re using a default @store unless a different store is passed to the function:

defmodule Auth.Guardian do
  use Guardian, otp_app: :auth

  alias Store.Behaviour, as: StoreBehaviour

  @store Application.fetch_env!(:auth, :user_repo)

  @doc """
  Decode a user token and try to get the user
  """
  @spec resource_from_claims(map) :: {:ok, user} | error
  def resource_from_claims(a, store \\ @store)
  def resource_from_claims(%{"sub" => "User:" <> id}, store) do
    result = StoreBehaviour.get(store, id)
    # ...
  end
  
  # ...
end

Which then means from “auth” tests we can call it like this:

defmodule Auth.GuardianTest do
  use ExUnit.Case, async: true

  alias Store.InMemory, as: Store
  alias Accounts.User

  @valid_email "hello@email.com"
  @valid_password_hash "asdasdsa"
  @valid_user %User{email: @valid_email, password_hash: @valid_password_hash}

  setup do
    {:ok, pid} = Store.start_link()
    {:ok, %{store: pid}}
  end

  describe ".resource_from_claims" do
    test "returns User when token passed in", ctx do
      user = Store.insert(ctx.store, @valid_user)

      result = Auth.Guardian.resource_from_claims(%{"sub" => "User:1"}, {Store, ctx.store})

      assert result == {:ok, user}
    end
  end
end

And finally our real data store, accounts, also implements StoreBehaviour so it looks something like this:

defmodule Accounts do
  @behaviour Store.Behaviour

  @impl Store.Behaviour
  def get(mod, id), do: Repo.get(mod, id)

  @impl Store.Behaviour
  def get_by(mod, opts), do: Repo.get_by(mod, opts)

  @impl Store.Behaviour
  def get_all(mod), do: Repo.all(mod)

  @impl Store.Behaviour
  def insert(mod, params), do: Repo.insert(mod, params)
  # ...
end

And in “auth” config we set up the default store like this:

use Mix.Config

config :auth, user_repo: {Accounts, Users.User}

So that’s pretty much it.

Note: I did look at replacing our InMemory store with Mox, but for now what we have works.

I hope that helps. It’s basically the same idea as @silviurosu showed.

3 Likes

There’s a lot of insight in this thread. Putting the persistence layer behind a behavior is a direction I’ve been heading towards. Decoupling the database from the core logic is especially valuable with legacy applications where database migration is a non-trivial task, or persistence is via REST API. It feels like there’s an un-answered middle-area between Phoenix Generator CRUD apps with it’s speed and conveniences and a full-on Event-Sourced CQRS solution like Commanded. I’d like to have stateful applications that persist to a database asynchronously when necessary. Injecting the database as a dependency and forgoing the core application’s knowledge of the persistence layer’s implementation is an important component to this middle-area I think.

What concerns me is around the interface of the in-memory/core_app entities and the database schema/entities. How much work and pre-requisite knowledge is required to implement a given change? If I have a %Person{} struct in the core_app, then for persistence, the db_app will need some sort of schema representation like %Person{} with changesets, mapping functions, etc. So our dependency situation goes from:

  • Business Logic <-> Database & Implementation (Phoenix+Ecto) to
  • core_app -> db_behavior & core_app schemas <- db_application

Is this much better? The core app doesn’t have to care about how a db_app is implemented, but I don’t see how the db_application avoids some kind of knowledge of the core_app's schemas. Map.from_struct and a keys to string conversion could let us avoid an explicit dependency, but it’s still there. Maybe I’m overthinking this. Probably. But for a given change to the application it’s possible we need to change the core_app, db_behavior, and the db_application. There’s at least 2 Person representations as well as mapping functions to maintain. We also add more required knowledge because a developer needs to know how to work with Typespecs, Behaviors, the core_app, the db_app, and a less conventional application structure. There’s definitely a few trade-offs to consider for the cost of decoupling the database when in many cases we still don’t get schema decoupling.

Event-sourcing decouples the database by moving the dependency from schemas to events. Persistence is an append-only eventstore, queries are implemented building projections from the events/event-store. Implementing applications with CQRS and event-sourcing has many other trade-offs such as a lot more required knowledge and typically eventual consistency, but we do achieve an Ecto/Database decoupling without schema coupling.

Anyway, I’d love to see what others have to say about the trade-offs of different approaches to dependency management in Elixir applications.

3 Likes

Hello @silviurosu,

Your revision of the example github repo is indeed more what I’m after. Regarding your note at the end: I was trying to add typespecs to your code but bumped into the fact that you can’t write a spec for a function returning a module which implements a Behaviour. After some searching, I bumped into this StackOverflow article:

How to use typespecs and Dialyzer with Behaviours?

@globalkeith, I guess with the Application.fetch_env!/2 call, you are also missing out on Dyalizer based code assist on the @store member?

Ringo

Hello @MrDoops,

You are adding a lot of additional insights to this thread too.

Reading up on how you see the separation of concerns with core_app, db_behavior and db_application, aren’t the 3 separate apps a bridge too far. To me, the behaviour to search, store and enumerate entities from the business domain (core_app) can easily be contained in a same core_app application. The db_application depends on core_app to access the entities (structs) and to register itself as the real persistence implementation.

The examples of @silviurosu and @globalkeith are simple and would work, but it is a pity you loose Dyalizer based code assist there. So if I would find an elegant, yet simple enough mechanism to implement the “service registration”, then my technical ego can feel at ease again. :slight_smile:

Ringo

This reminds me of this discussion and ideally it should be:

define core_app abstractions and core_app schemas (types) first

BUT changing the shape of persisted production data is an expensive effort which therefore runs into a lot of resistance, imposing a kind of technological inertia (Why Data Models Shouldn’t Drive Object Models (And Vice Versa) 2002).

Also the philosophy of:

is outmoded, which is why DDD puts domain types front and center, not database schemas. It’s the repository’s job to map the domain types to the DB schema and vice versa - serving as a special kind of anticorruption layer (ACL) to persistent storage. So ideally

  • defining a schema of domain types and
  • mapping those domain types to an existing storage schema

should be separate responsibilities. But for the sake of convenience of migrations those responsibilities are often conflated.

There’s at least 2 Person representations

Even when it just comes to bounded contexts there could be two representations per context (Fowler) - separate from the database schema:

  • one representation that the context (module) shares with the “outside” that is stripped of any implementation revealing details and
  • another representation (with implementation revealing details) used between all the functions inside the context to get actual work done.
4 Likes

Thanks for all the interesting links!! @ringods I’d certainly be interested if you come up with something elegant that works with dyalizer.

Talking of multiple representations…

We came up to an interesting intersection recently in our little project when adding a mailer application.

We have an umbrella project, with a shop application. In order to allow our apps to communicate we use an event pubsub app. The challenge came when we wanted to send an event order_completed which contained an Order. This Order struct was defined in shop however once we started sending events we needed the mailer to be able to understand an Order, but we didn’t want to have multiple definitions.

We debated the merits of defining multiple representations, with the required transcoding and the associated maintenance, vs adding shop as a dependency in the mailer but that just seemed wrong on all levels and defeated the point of communicating via our pubsub.

Eventually we felt the code was pushing us in a certain direction and settled on extracting the Order (and associated functionality) to a new app, which both shop and mailer could list as a dependency which then allows both apps to use Order struct. So far it’s seem to have worked out well, though time will tell.

3 Likes

Back to the original question, I have a strong feeling that Protocols would be a very good fit here to decouple one from the other (like the phoenix, ecto and phoenix_ecto libs)

Protocols revolve around a set of functions that can be applied to a data structure (i.e. a type) and as such are similar to interfaces in OO. That works if you want to hand an “instance” around that represents the particular database instance (pool or connection) and is absolutely necessary if you need to access a variety of distinct instances within the same runtime period.

But for the most part applications only actually deal with one single instance but need to retain the flexibility to use different, configurable implementations for different target environments (usually production, testing, development). In my opinion for that type of “swap-ability” a behaviour is a better fit.

Typically I’m not fond of Singletons (there rarely is only one Singleton in an application that uses Singletons) and in OO I will always favor Just Create One. But it seems that a Module/Behaviour is just about a perfect Singleton when you actually need one.

Mock Modules and Where to Find Them

1 Like