Livebook utility functions

Elixir Framework Forums Livebook Forum
,
1

We’re using Livebook for data analysis and are putting all of our notebooks in a common private git repository. We’ve built up some utility code to help with querying our data warehouse and other things. It’s nice to maintain that code in the same git repository since it’s tightly coupled to the notebooks. I was wondering how others did this, since what we’re doing right now seems “just ok”.

Our current solution is to put the utility code in a little library and Mix.install’ it as a path dependency from each Livebook. The directory structure looks like this:

.
├── notebooks
│   ├── xyz.livemd
│   └── ...
├── sandbox
│   └── ...
├── our_utils
│   ├── lib
│   ├── mix.exs
│   └── README.md
├── README.md
├── template.livemd
└── tutorial.livemd

The notebooks and sandbox directories contain Livebooks organized at various path depths. When creating a new notebook, we copy template.livemd which has a good Mix.install line and a couple generic sections. The Mix.install includes the our_utils library.

This mostly works, but could be improved.

For example, here’s the dependency snippet to find the our_utils directory since it could be at any depth:

base_dir = Stream.iterate(Path.expand(__DIR__), &Path.dirname/1) |> Enum.find(&File.dir?(Path.join(&1, "our_utils")))
Mix.install([
  {:kino_explorer, "~> 0.1.20"},
  {:our_utils, path: Path.join(base_dir, "our_utils")},
])

It’s a one-liner and spares us from errors based on different directory depths. It doesn’t work when the notebook hasn’t been saved in a subdirectory of our main project.

We’ve talked about switching the dependency to a github reference, but the branch would almost certainly effectively be main. It would also require switching back to a path dependency every time we work on it. Not terrible, but also seems like it could be better.

I suppose I’ll leave it at that and just see what has worked for others in this situation.

1 Like
2

How about environment variable?

base_dir = System.get_env("OUR_APP_GIT_DIR")

Simply add this env once to your shell script (like ~/.basrc) and you can easily use it even with git:

git clone url "$OUR_APP_GIT_DIR"

Yes, it’s “extra work” for each team member, but it’s better not only because it’s shorter, but also because it’s using a common way for environment-specific configuration. Also Livebook should support environment variables as well.

3

Hi @Eiji,

Thanks. An environment variable seems reasonable.

It’s hard for me to argue that any of the “extra work” is difficult in isolation, but somehow it comes together. Your point is well taken, and I’ll try to embrace the environment variable and look for friction reduction elsewhere.

1 Like