Ignore Oban-related traces in OpenTelemetry for Ecto

Questions / Help
, ,
1

Hello,

I want to ignore Oban-related traces in Ecto OpenTelemetry which comes in two flavours:

Any advices how to approach this?

2

Me too. It is super annoying seeing millions of metrics being collected when the system is barely used.

Yesterday I turned open telemetry down for ecto because of this

3

We just went through this, it’s a bit involved. I’m on my phone so I’ll have to show code later but here is a high level answer:

  1. Attach a regular telemetry hook to ALL of the Oban internal events.
  2. In that hook, set an open telemetry context value of like “oban_internal: true”
  3. Configure your opentelemtry lib to use a custom sampler
  4. in your sampler, drop all traces where the trace context is Oban internal

As I said, a bit involved. I’ll be able to provide our code later.

2 Likes
4

All requests are made with Oban.Repo, which wraps calls with custom options as seen here: https://github.com/sorentwo/oban/blob/main/lib/oban/repo.ex#L97. Notice there is an oban: true field that you can detect in functions like prepare_query/3, as well as oban_conf in telemetry options which could theoretically be used to ignore Oban emitted events.

Unfortunately, it looks like OpenTelemetryEcto doesn’t have a way to ignore certain events (yet).

You could add a small wrapper module to your application, like this:

defmodule MyApp.OpenTelemetryEcto do
  @moduledoc """
  A wrapper around OpenTelemetryEcto that filters out queries generated by Oban.
  """

  def setup(event_prefix, config \\ []) do
    event = event_prefix ++ [:query]

    :telemetry.attach({OpenTelemetryEcto, event}, event, &__MODULE__.handle_event/4, config)
  end

  @doc false
  def handle_event(_event, _measure, %{oban_config: _}, _config) do
    :ok
  end

  def handle_event(event, measure, meta, config) do
    OpenTelemetryEcto.handle_event(event, measure, meta, config)
  end
end
2 Likes
5

Interesting! We needed a sampler anyway but that does look like an easier way to do it! Will test out later today.

6

After playing with this locally this clause is slightly wrong, it needs to be:

  # filter out oban tagged values
  def handle_event(event, measure, %{options: options} = meta, config) do
    if Keyword.has_key?(options, :oban_conf) do
      :ok
    else
      OpentelemetryEcto.handle_event(event, measure, meta, config)
    end
  end
1 Like
7

Good to know! Sorry, I wrote that without checking that it worked because I don’t currently use OpenTelemetry :slightly_smiling_face:

1 Like