Ignore Oban-related traces in OpenTelemetry for Ecto

Me too. It is super annoying seeing millions of metrics being collected when the system is barely used.

Yesterday I turned open telemetry down for ecto because of this

We just went through this, it’s a bit involved. I’m on my phone so I’ll have to show code later but here is a high level answer:

1. Attach a regular telemetry hook to ALL of the Oban internal events.
2. In that hook, set an open telemetry context value of like “oban_internal: true”
3. Configure your opentelemtry lib to use a custom sampler
4. in your sampler, drop all traces where the trace context is Oban internal

As I said, a bit involved. I’ll be able to provide our code later.

All requests are made with Oban.Repo, which wraps calls with custom options as seen here: oban/lib/oban/repo.ex at main · oban-bg/oban · GitHub. Notice there is an oban: true field that you can detect in functions like prepare_query/3, as well as oban_conf in telemetry options which could theoretically be used to ignore Oban emitted events.

Unfortunately, it looks like OpenTelemetryEcto doesn’t have a way to ignore certain events (yet).

You could add a small wrapper module to your application, like this:

defmodule MyApp.OpenTelemetryEcto do
  @moduledoc """
  A wrapper around OpenTelemetryEcto that filters out queries generated by Oban.
  """

  def setup(event_prefix, config \\ []) do
    event = event_prefix ++ [:query]

    :telemetry.attach({OpenTelemetryEcto, event}, event, &__MODULE__.handle_event/4, config)
  end

  @doc false
  def handle_event(_event, _measure, %{oban_config: _}, _config) do
    :ok
  end

  def handle_event(event, measure, meta, config) do
    OpenTelemetryEcto.handle_event(event, measure, meta, config)
  end
end

Interesting! We needed a sampler anyway but that does look like an easier way to do it! Will test out later today.

After playing with this locally this clause is slightly wrong, it needs to be:

  # filter out oban tagged values
  def handle_event(event, measure, %{options: options} = meta, config) do
    if Keyword.has_key?(options, :oban_conf) do
      :ok
    else
      OpentelemetryEcto.handle_event(event, measure, meta, config)
    end
  end

Good to know! Sorry, I wrote that without checking that it worked because I don’t currently use OpenTelemetry