Injecting tracing the hot way

Let’s do Open Telemetry tracing.. by doing Erlang tracing.

And also hot code updates for special software delivery.

10 Likes

Excellent post, thank you. I am fully for influential voices in the community re-giving a chance to hot code loading.

I’m considering if this is a decent approach to a plugin system for another system I’m working with where plugins might make some sense. And for pure Elixir code this seems perfectly reasonable.

I am using hot code loading in prod to load tenant-specific features that conform to behaviours, so, kind of a plugin system with its idiosyncratic spec. This allows me to have the plugins part of the codebase and monorepo, so they are fully tested by mix test with the rest and in CI. Just at release building time, this source folder (and the matching test folder) is rm -rf’d so they are not included the release.

This, combined with a bespoke script I called an “abstraction boundary checker” allows me to have a nice development experience while guaranteeing that app code never calls plugin code, and plugin code does not call other plugin code. Since the plugins are part of the codebase, an incompatible API change breaks them in dev and test.

When I published an article about that last year, I insisted that what enabled that was my “1 client = 1 instance” rule but I am now doing it in a multi-node cluster with multiple tenants, so my stance about isolation wasn’t a real constraint.

2 Likes

Besides giving hot code loading another chance, I’d like to remove the stigma around spawning processes. Web developers are scared to spawn processes and instead settle on suboptimal designs which could’ve been correct with one or two more processes.

3 Likes

Charmer.

I do love experimenting with hot code updates. And of course since I develop a lot with Nerves the iteration cycle is a bit slower if I do a full firmware build plus upload so I am constantly pasting modules into iex.

I still want to do the art project of starting a project that is some online service where it is not allowed to be persisted. Ephemeral by nature and the sport is to keep updating and building it out without losing it.

2 Likes

Reminds me of Bryan Hunter’s Waterpark project. This clearly shows the advantage of erlang, OTP and hot updates.
Though I am still getting kinda nervous if I am thinking there is no backup to disk at all :slight_smile:

I believe they’ve over time added some persistence. But yeah, wild project :slight_smile: