Shared data in a multi-language monorepo

Questions / Help
1

Hello, so we have a private repository https://example.com/event_schema_registry.git which contains event schemas (as files in JSON schema format) which are used for validating events published to a message broker. It also contains a Ruby gem schema_registry which uses (depends on) the before-mentioned schema files for validating the events in our Ruby code:

.
├── README.md
├── schemas             # Schema files directory
│  ├── # [... snip ...]
│  └── my_event.json
├── ruby                # Ruby gem schema_registry
│  ├── # [... snip ...]
│  └── schema_registry.gemspec
└── elixir              # The new schema_registry Elixir package
   ├── # [... snip ...]
   └── mix.exs

The ruby gem schema_registry, located in the /ruby directory, uses schema files in /schemas, and when I add this gem to another Ruby projects as a git dependency it works without issues (thanks to bundler magic).

Now I want to write another event validator, but this time in Elixir, so I created a new mix project schema_registry in the /elixir directory. If schema files belonged only to the Elixir project, I would put them inside the project’s priv directory, but alas, they are already used by the Ruby gem, and we don’t want to duplicate them.

The best idea I came up with was creating a symbolic link inside priv that points to the shared shared schemas directory /schemas. But it didn’t work when I tried to add schema_registry as a git dependency to another Elixir project. On my first try time I added schema_registry like this:

{:schema_registry, git: "https://example.com/event_schema_registry.git"}

but it failed to compile:

Could not compile :schema_registry, no "mix.exs", "rebar.config" or "Makefile" (pass :compile as an option to customize compilation, set it to "false" to do nothing)

On the next try I also added the :sparse option:

{:schema_registry, git: "https://example.com/event_schema_registry.git", sparse: "elixir"}

This time the project compiled successfully, but schema_registry doesn’t see any schema file (because now deps/schema_registry/elixir/priv contains a broken symlink to files that were not checked out).

So the question: is there any way I can bundle shared data with an Elixir package so this package would work as a git-based Mix dependency?

2

Since the schemas/ directory is on the same directory level as elixir/ you can try adding it as a dependency as well:

{:schemas, git: "https://example.com/event_schema_registry.git", sparse: "schemas", app: false}

If the symlinks are relative I think they should resolve correctly now.

2 Likes
3

IMO if you’re keeping schemas in a separate Git repo then your gems / Hex packages should be a derived artifact; updating the schema repo would then regenerate those, then the dependent applications could be updated.

The symlink in priv could work locally, but mix release is going to flatten that out and copy the files inside the release.

4

Thanks for the example, didn’t know about this option! I added schemas/ as a dependency as you suggested (I also had to add compile: false, otherwise it would result in compilation warning).

But I couldn’t figure out the correct relative symlink path so it would work both when compiled as a dependency and standalone. Maybe it’s because I read schema files during compilation to embed them into Elixir modules?

Then I read the Mix.Project docs again and decided to take another route. I initialized a new Elixir project at the repository root, created as a symlink schemas inside priv/ pointing to ../schemas, and moved all Elixir stuff (excluding priv/ and mix.exs) to elixir/:

.
├── mix.exs
├── elixir/
│  ├── _build/
│  ├── deps/
│  ├── lib/
│  ├── mix.lock
│  └── test/
├── priv/
│  └── schemas -> ../schemas
├── ruby/
│  ├── # [... snip ...]
│  └── schema_registry.gemspec
├── schemas/
│  ├── # [... snip ...]
│  └── my_event.json
└── README.md

Then I added some settings (namely :elixirc_paths, :deps_path, :lockfile, :build_path, :test_paths) to my mix.exs file:

defmodule SchemaRegistry.MixProject do
  use Mix.Project

  def project do
    [
      app: :schema_registry,
      version: "0.1.0",
      elixir: "~> 1.10",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      #
      # Move elixir stuff to `elixir/`
      elixirc_paths: elixirc_paths(Mix.env()),
      deps_path: "elixir/deps",
      lockfile: "elixir/mix.lock",
      build_path: "elixir/_build",
      test_paths: ["elixir/test"]
    ]
  end

  def application do
    [
      extra_applications: [:logger]
    ]
  end

  defp elixirc_paths(:test), do: ["elixir/lib", "elixir/test/support"]
  defp elixirc_paths(_), do: ["elixir/lib"]

  defp deps do
    [
      {:jason, "~> 1.0"},
      {:ex_json_schema, "~> 0.7"}
    ]
  end
end

And it finally works as intended!

2 Likes