:dbg and persistent_term or something else

Questions / Help
, ,
1

Hi,

So I’m in the process of integrating :dbg , for tracing, into the product I’m working on. I’m using a GenServer to manage and analyze trace requests.

There are 2 types of processes to trace, long-running websocket processes and ephemeral processes handling REST requests. Each process is ‘owned’ by a user and the user_id is actually saved in the process dictionary of each process (looking forward to process labels in OTP 27). Typically a user has about 10 long-running process and maybe a new REST request every 5 seconds or so.

So what we want to do is say

trace bob@foobar.com for event X or for REST request {'GET', "/users/foo/baar"}"

Or we could create traces for the domain foobar.com

So the GenServer will authorize the request, create the match specs and identify the functions to apply the match specs to. A typical trace shall be a few seconds in duration, and tracing will be an infrequent occurrence. Only a single user / domain will be traced at a time.

How to identify the processes though? Basically we need add the pids to the tracer using :dbg.p(pid, [:c]) for all the processes owned by bob@foobar.com.

  1. Use persistent_term and have a function such as below that will be called on every in-bound request.
def maybe_start_trace(user_id) do
    trace_user = :persistent_term.get(:trace_user, nil)
    if trace_user && String.contains?(user_id, trace_user) do
      :dbg.p(self(), [:call])
    end
end

I worry about the cost of setting and clearing the persistent term.

  1. Dynamically compile a module such as:
defmodule TraceMatch do  
  # This line added when tracing is enabled
  def trace_me("bob@foobar.com"), do: :dbg.p(self(), [:call])
  # This line here all the time
  def trace_me(_), do: nil
end

  1. Hybrid-mode. For the running long-lived processes we can identify them from their process dictionary entry and use one of the mechanisms above for the ephemeral processes. This means we are potentially adding hundreds of idle processes to be traced that don’t need to be.

  2. Use the GenServer (or :ets) is not an option - see Amdahl`s law.

  3. A community answer? Underpants gnomes.

Thanks in advance…

2 
sos (set on spawn)
    Lets all processes created by the traced process 
    inherit the trace flags of the traced process.

sol (set on link)
    Lets another process, P2, inherit the trace flags of the traced process 
    whenever the traced process links to P2.

If you could have a supervisor per user.

Or have the processes register in :pg, Registry, etc.

Then again (depending on how many traces they generate) a few hundred idle processes doesn’t sound like that much.

3

The cost of setting and clearing a persistent can be enormous, the persistent documentation is quite explicit about this.

4

Why is it not an option? ETS has a read concurrency mode when you do much more reading than writing.

I’d go for Registry though. While I am fond of dynamically compiling modules to do what we need right on the spot, the BEAM is not always handling hundreds of pattern-matching clauses well in terms of performance. Registry should allow your code to be faster and to use what’s been purposefully built for scenarios like yours.

1 Like
5

I ended up using persistent term, but just saving an atom to indicate tracing is enabled or not . I then save details about what needs tracing in ets.

But maybe I should just use ets instead as @dimitarvp mentioned.

6

Do you have any idea about match specs? :stuck_out_tongue_winking_eye:

I left this on Erlang forum :

And on a personal note…

Thank you so much for inventing Erlang. Both Erlang and Elixir are a joy to program in. Now with Gleam joining the beam family I think beam has a great future. RIP Joe.

7

IIRC match specs are interpreted by a small VM and I am guessing that if you have excessive nesting then that VM can run out of stack. But I am guessing here.

Glad you like Erlang and then Elixir. I am not really a super static typing fan :smile: and I have done a lisp on top of Erlang/BEAM LFE (Lisp Flavoured Erlang)

3 Likes