Monolithic Repo and Multiple Packages - how to avoid having 100 git repos for 100 packages?

Let me introduce the context before of my questions.

In order to support some new kind of hardware in AtomVM, a driver should be written first.

It basically means writing an Elixir/Erlang module for managing that device.
E.g. In order to support lora radio chip “sx126x” I did a module called Sx126XDriver, same thing for “bq25896” power management chip and so on.

Most of the times I end having 1 driver = 1 module.

Then I would like to make some packages, it would be nice having a package for each single driver, so when I build a new project I don’t have to pull drivers for devices I don’t have.

How can I avoid having 100 git repos for 100 packages?

What I was thinking about is:

a) When dependency is on git:

  • mix.exs deps: use sparse option when package is on git, e.g.: sparse: “apps/sx126x_driver” (should I use apps as a container directory?)
  • rebar3 deps: use git_subdir

b) Otherwise on hex: just use regular hex packages, use CI for publishing in a automated way packages to hex.

  • Are you aware of any relevant issue with such a similar workflow?
  • Do you have any suggestion for doing this in the best way?
  • Do you have any suggestion for providing the best possible developer experience to package users?
  • Should I prefix/suffix each package with some common prefix/suffix to indicate they are comming from the same megarepo? Such as in my case: avm_drivers_sx126x, avm_drivers_bq25896, etc…

Otherwise do you have any opinion against a similar pattern?

@Nerves folks: did you have any similar experience?

I’m looking forward opinions about this approach.

1 Like

You can also use depth: 1 to avoid downloading the full repo history in case there are heavy binaries.

I’d say up to you, but once people start depending on the directory structure it’s better not to change it.

This may help with discoverability, but there are other ways to deal with that too, like cross linking, or new features in ExDoc supporting cross package search.

2 Likes

How about keeping all drivers in a single package and enabling/disabling them in config.exs? There’s always some maintenance burden with that many packages, even if you automate releases.

3 Likes

This isn’t an exact answer to your question, but I’ve been solving this in Jido with git subtree

Each package is still it’s own GIT repo, but I work on them together inside a monorepo where each sub-package is pulled in via a subtree.

I created this repo to demonstrate the concept in Elixir.

git subtree would keep the main complexity I would like to avoid: having an excessive number of git repos.

Also, personal opinion here: git subtree introduces additional friction during development.

1 Like

If you’re one person or small team owning and publishing all drivers, then a single repo would be the least maintenance burden. Trust your instincts!

1 Like

deps can have a targets: option, so maybe making them more explicit rather relying in config.exs might be the right choice. Also I’m looking for some sort of approach that works also with rebar3.

Using git subtree directly is a PITA - I write mix commands to wrap the basics and typically just have an LLM manage it all and it’s been a breeze

Again, just a thought