Sharing schemas across applications?

Here is my current structure:

  • scheduler app (Elixir app)
  • background worker app x 2 (Elixir app)
  • web app x 2 (Phoenix app)

The scheduler, background workers and web apps are completely separate applications (not part of an umbrella app); they could be on different nodes, even in different regions.

However, all of these applications need to speak to my database, and for that I have common ecto schemas.

My question is two-fold:

  1. would you use a library for the common domain model?
  2. what is the best way to handle that during development?

For example, in the past I would reference a local copy of the library in order to develop on, but unsure if this is straight-forward in Elixir, or if there are gotchas I need to be aware of (noting that only the web app is a Phoenix app).

2 Likes

What I have done in this situation is create a mix project that contains the migrations and schemas (and usually some of the common interaction functions, like ones to generate useful changesets) and then use that as a dependency in all the individual projects.

During development, I replace the usual git path to that library in the mix dependency with a local path, e.g. this:

 {:my_database, git: "git@git.somewhere.com:path/to/my_database"}

becomes

{:my_database, path: "../my_database"}

Then changes during devel to the schemas gets picked up in further calls to mix compile without having to commit, push, mix deps.update etc. Very handy! Then before committing into the feature branch of the application repo(s), I revert the mix.exs change (git stash is pretty handy here), so that the path version does not polute the app repo.

1 Like

I feel like you might be forcing separation of concerns without reason. If all the apps rely on the same schemas and database - why not just join them into a singular app? If you want some nodes to run certain services, but not others (ie. Your background workers to not run your endpoint) - you can just use env flags to control what is started in your supervision tree.

1 Like

This is how I’ve setup my current worker/web (as worker currently relies on schemas in web).

It has resulted in:

  def start(_type, _args) do
    import Supervisor.Spec

    children = if app_worker?() do
      [
        supervisor(Verk.Supervisor, []),
        supervisor(Pingerly.Repo, []),
        {Task.Supervisor, name: Pingerly.Task.Supervisor}
      ]
    else
      [
        supervisor(Pingerly.Repo, []),
        supervisor(PingerlyWeb.Endpoint, []),
        worker(Redix, [redis_url(), [name: Verk.Redis]], id: Verk.Redis)
      ]
    end

    opts = [strategy: :one_for_one, name: Pingerly.Supervisor]
    Supervisor.start_link(children, opts)
  end

Not entirely happy scaling that for every new application I introduce. Maybe if there were just two apps (worker/web), but not now I have a scheduler and [potentially] more apps down the line. The apps have separate responsibilities along with dependencies and, as you say, supervision trees, different configs… very messy to dump all that in a singular app.

But fair point.

Cheers. I think this is likely what I’ll end up doing.

It does get a bit ugly as more and more business logic related to a variety of modules and config ends up in the supervisor(s). One possible route is to Enum.reduce a children list by calling a function in each of the children which returns a spec or not, and is thereby included in the children list or not, e.g.:

Enum.reduce([Verk.Supervisor, Pingerly.Repo], [],
            fn module, children -> 
              case module.should_start?(args) do
                nil -> children
                spec -> [spec | children]
            end) |> Enum.reverse()

… or some such. At least then the decision making logic is in each child. For child implementations you don’t control (e.g. deps), creating a factory module that produces child specs based on env would do (the module in the children list does not need to return a child spec that resolves to itself!)

In fact, I wonder if this couldn’t be added as a general feature in the standard library. I know I would have used it in projects already if it were … something like this patch … @josevalim: what do you think? Obviously that patch only covers Supervisor and would need similar implementations in all the child_spec() implementations and supervisors…

Don’t forget that a mix.exs file is code, you can always conditionally pull a local path or git, or from an environment, or whatever. :slight_smile:

Folks tend to forget you can return :ignore from a start_link function. https://hexdocs.pm/elixir/GenServer.html#t:on_start/0. If you return :ignore from a start link it’ll just get ignored. This means you can have your worker list be a nice full worker list, and then the start_link function of each child can determine whether it should actually start or not.

This tactic works great for sort of “feature flag” style workers where they’re all pretty independent. Not so great for tightly coupled workers.

4 Likes