Nice post! And thank you for the shout-out, and your many contributions to Peep

I think your post is currently missing the links to Ultravisor and Speedoscope.

This project now lives as [Ultravisor][]
[…]
That has disadvantages, but at least it was workable within [Speedoscope][]

Aside: Did you happen to explore using Linux’s perf(1) for less-obtrusive tracing? It’s a bit clunky to use (or maybe I haven’t found a better way of wielding it), but I find it super helpful, especially since the BEAM emits perf maps for jitted functions. When I was spending a lot of time trying to optimize stuff, which runs in a Docker container in prod, I repeatedly ran perf(1), and generated flame graphs (using GitHub - brendangregg/FlameGraph: Stack trace visualizer · GitHub ) like so:

# copy perf map out of Docker container, making it visible to perf running on the host.
sudo docker cp container:/tmp/perf-6.map /tmp/perf-2969015.map
sudo perf record --call-graph=fp --pid 2969015 -- sleep 10
sudo perf script > out.perf
sudo chown richard:richard out.perf
./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# strip [0-9]+_ and/or _[0-9]+ from all scheduler names
sed -e 's/^[0-9]\+_//' -e 's/^erts_\([^_]\+\)_[0-9]\+/erts_\1/' out.folded > out.folded_sched
./FlameGraph/flamegraph.pl out.folded_sched > out_sched.svg

These flame graphs weren’t totally complete (i.e. they missed stuff from NIFs compiled in the container without debug symbols), but it was nice to get a mix of BEAM and native code in a single flame graph.

This was an enjoyable read, thank you. I’ll try using Peep soon as a result.