Out of curiosity, how are you storing the slow queries and stacktraces?
gmile
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
For a long time we just logged it but lately we integrated OpenTelemetry with Datadog traces. This was more complicated to set up than I’d like but it’s now pretty cool since we can get traces from all our different services and even correlate traces between services.
EDIT: Although I should add we use sampling on our major data ingestion pipelines to avoid generating too many traces.
2
kenny-evitt
AppSignal has a standard “Slow queries” feature. It seems to work pretty well. (You do need to attach their Ecto handler in your app supervisor tree but I’m very sure their docs cover that.)
1
kenny-evitt
I just looked at the issue you opened about adding support for stacktraces – I get what you’re missing now! That has been mildly annoying.
1
davydog187
You should consider integrating OpenTelemetry with OpenTelemetryEcto that will capture spans for your Ecto queries, and then ship them to a good observability tool like https://www.honeycomb.io/ or https://www.servicenow.com/products/observability.html
I wrote about this around a year ago and there’s another good post that goes into more practical detail
7
barkerja
Thank you for these links! Any recommended resources for using DataDog with OpenTelemetry?
0
davydog187
Everything but the export will be exactly the same. The question is if you want to use the Otel Collector or not. Once you’re ready to configure exporting your data, you can check the Datadog Otel docs
2
gmile
After looking further, it appears that in it’s current state OpenTelemetry spec does not provide means to send a generic stacktrace in a span 
It does supports sending stacktraces for exceptions though.
I found this conversation Structured Stacktrace in Span Attributes/Logs · Issue #2839 · open-telemetry/opentelemetry-specification · GitHub by @tristan. It seems relevant, but appears to be far from being complete / accepted into the spec. On top of not being part of the standard, it would have to be adopted by opentelemetry-ecto, and I believe vendors too.
In opentelemetry-ecto, there seems to be an escape hatch though - ability to provide a list of additional_attributes: opentelemetry-erlang-contrib/instrumentation/opentelemetry_ecto/lib/opentelemetry_ecto.ex at 596fcbc2cbc53d6b31d1126e11a13c1a9ee36c0b · open-telemetry/opentelemetry-erlang-contrib · GitHub. But it’s not clear to me at the moment how to provide a stacktrace (produced by Ecto) as an item in additional_attributes.
We’re using Google Cloud, which offers “Cloud Span” product, which recommends using OpenTelemetry and supports a stackTrace field in a span (see this).
So, I think potentially with some patching of opentelemetry-ecto and reading more of Google Cloud docs, sending spans containing Ecto stacktraces might work Will need to look more into this.
0
davydog187
Why do you need a stack trace? If you’re trying to get a stack trace for every span that is going to be expensive, not to mention a bit unnecessary if you’re adding your own spans to your application.
Let’s say you do want to create your own spans for some service/context functions in your application, you can add additional attributes like so
defmodule MyContext do
require OpenTelemetry.Tracer
alias OpenTelemetry.Tracer
def list_products(%User{} = user) do
attributes = %{user_id: user.id, user_name: user.name}
Tracer.with_span "my-context:list_products", %{attributes: attributes} do
MyApp.Repo.all(from p in Products, where: p.user_id == ^user.id)
end
end
end
1
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
It really isn’t. As I said we make thousands of ecto queries per second and setting stacktrace: true has been perfectly fine.
People should definitely set top level traces, but that doesn’t help if you get a trace for an ecto query that happens outside of any top level trace. In a large code base it’s difficult to get 100% coverage of all entry points and so it can be tricky to sort out where a given ecto query is coming from. If you can at least get a stacktrace for where in the code the query came from you can work your way back up.
Top level traces also don’t help if you make many queries that have the same SQL but in different places have very different arguments.
Tl;DR: stacktraces rock, it’s useful to use them in combination with top level traces.
2