:dbg and persistent_term or something else

Hi,

So I’m in the process of integrating :dbg , for tracing, into the product I’m working on. I’m using a GenServer to manage and analyze trace requests.

There are 2 types of processes to trace, long-running websocket processes and ephemeral processes handling REST requests. Each process is ‘owned’ by a user and the user_id is actually saved in the process dictionary of each process (looking forward to process labels in OTP 27). Typically a user has about 10 long-running process and maybe a new REST request every 5 seconds or so.

So what we want to do is say

trace bob@foobar.com for event X or for REST request {'GET', "/users/foo/baar"}"

Or we could create traces for the domain foobar.com

So the GenServer will authorize the request, create the match specs and identify the functions to apply the match specs to. A typical trace shall be a few seconds in duration, and tracing will be an infrequent occurrence. Only a single user / domain will be traced at a time.

How to identify the processes though? Basically we need add the pids to the tracer using :dbg.p(pid, [:c]) for all the processes owned by bob@foobar.com.

1. Use persistent_term and have a function such as below that will be called on every in-bound request.

def maybe_start_trace(user_id) do
    trace_user = :persistent_term.get(:trace_user, nil)
    if trace_user && String.contains?(user_id, trace_user) do
      :dbg.p(self(), [:call])
    end
end

I worry about the cost of setting and clearing the persistent term.

2. Dynamically compile a module such as:

defmodule TraceMatch do  
  # This line added when tracing is enabled
  def trace_me("bob@foobar.com"), do: :dbg.p(self(), [:call])
  # This line here all the time
  def trace_me(_), do: nil
end

3. Hybrid-mode. For the running long-lived processes we can identify them from their process dictionary entry and use one of the mechanisms above for the ephemeral processes. This means we are potentially adding hundreds of idle processes to be traced that don’t need to be.

4. Use the GenServer (or :ets) is not an option - see Amdahl`s law.

5. A community answer? Underpants gnomes.

Thanks in advance…