Livebook utility functions

We’re using Livebook for data analysis and are putting all of our notebooks in a common private git repository. We’ve built up some utility code to help with querying our data warehouse and other things. It’s nice to maintain that code in the same git repository since it’s tightly coupled to the notebooks. I was wondering how others did this, since what we’re doing right now seems “just ok”.

Our current solution is to put the utility code in a little library and Mix.install’ it as a path dependency from each Livebook. The directory structure looks like this:

.
├── notebooks
│   ├── xyz.livemd
│   └── ...
├── sandbox
│   └── ...
├── our_utils
│   ├── lib
│   ├── mix.exs
│   └── README.md
├── README.md
├── template.livemd
└── tutorial.livemd

The notebooks and sandbox directories contain Livebooks organized at various path depths. When creating a new notebook, we copy template.livemd which has a good Mix.install line and a couple generic sections. The Mix.install includes the our_utils library.

This mostly works, but could be improved.

For example, here’s the dependency snippet to find the our_utils directory since it could be at any depth:

base_dir = Stream.iterate(Path.expand(__DIR__), &Path.dirname/1) |> Enum.find(&File.dir?(Path.join(&1, "our_utils")))
Mix.install([
  {:kino_explorer, "~> 0.1.20"},
  {:our_utils, path: Path.join(base_dir, "our_utils")},
])

It’s a one-liner and spares us from errors based on different directory depths. It doesn’t work when the notebook hasn’t been saved in a subdirectory of our main project.

We’ve talked about switching the dependency to a github reference, but the branch would almost certainly effectively be main. It would also require switching back to a path dependency every time we work on it. Not terrible, but also seems like it could be better.

I suppose I’ll leave it at that and just see what has worked for others in this situation.