Sending tracing data to datadog from an ecs container

andym

hi everyone, i have an aws ecs fargate container running and i’m trying to send tracing information to datadog (i’m using phoenix + liveview). to do that i’m using the spandex library GitHub - spandex-project/spandex: A platform agnostic tracing library · GitHub along with the datadog adapter GitHub - spandex-project/spandex_datadog: A datadog adapter for the `spandex` library. · GitHub. i have also configured the sidecar container for datadog as per the documentation. the monitoring data from the datadog agent running on ecs reaches datadog just fine, however the custom tracing information from my application does not reach datadog. this is the configuration i have in my code

application.ex

    spandex_opts = [
      host: "localhost",
      port: 8126,
      batch_size: 10,
      sync_threshold: 100,
      http: HTTPoison
    ]

    children = [
      # Start the Telemetry supervisor
      {SpandexDatadog.ApiServer, spandex_opts},
      App.Telemetry,
      # Start the PubSub system
      {Phoenix.PubSub, name: App.PubSub},
      # Start the Endpoint (http/https)
      App.Endpoint
    ]

config.exs

config :app, app.Endpoint,
  url: [host: "localhost"],
  render_errors: [view: app.ErrorView, accepts: ~w(html json), layout: false],
  pubsub_server: app.PubSub,
  live_view: [signing_salt: "dyhFo42f"]

config :app, app.Tracer,
  service: :app,
  adapter: SpandexDatadog.Adapter,
  disabled?: false,
  env: "PROD"

Now in one of the controller i just have the following to make sure the data is going accross

MansionWeb.Tracer.start_trace("about_us")
MansionWeb.Tracer.update_span(service: :mansion, type: :web)

    app.Tracer.start_span("span1")
    app.Tracer.update_span(service: :app, type: :web)
    app.Tracer.finish_span()

    app.Tracer.finish_span()
    app.Tracer.finish_trace()

however this trace information never makes it to the actual datadog traces.

6 comments

#datadog #distributed-tracing

0 2135 6

2023-02-27 06:06:17 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

imsoulfly

Hello,

have you checked that the api_server.ex module was able to actually send the traces to the sidecar instance?
SpandexDatadog has an undocumented config setting called verbose? that allows some debugging of what the api_server is doing.
Here is the related code: github link

Post #1

andym

thanks for the reply @imsoulfly , i had actually enabled this and this is all i see in the logs

02:47:53.968 [info] Adding trace to stack with 2 spans

i also don’t see any errors on the logs as well of the agent.

outside of this i don’t see anything else in the logs. it does not say if the messages were successfully sent to the agent or failed.

i can see the metrics coming from the datadog agent itself like the cpu percentage and so on, i think the problem is the communication from my application container to the datadog container.

also if it helps this is the container definition for my datadog agent

 {
    "Name": "datadog-agent",
    "Image": "public.ecr.aws/datadog/agent:latest",
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-region": "ap-southeast-2",
        "awslogs-stream-prefix": "${service_name}-dd-log-stream",
        "awslogs-group": "/ecs/${service_name}"
      }
    },
    "portMappings": [
      {
        "hostPort": 8126,
        "protocol": "tcp",
        "containerPort": 8126
      }
    ],
    "Environment": [
      {
        "Name": "DD_API_KEY",
        "Value": "<REAL _ KEY _ HERE>"
      },
      {
        "Name": "DD_APM_ENABLED",
        "Value": "true"
      },
      {
        "Name": "DD_SITE",
        "Value": "us3.datadoghq.com"
      },
      {
        "Name": "ECS_FARGATE",
        "Value": "true"
      },
      {
        "name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
        "value": "true"
      }
    ]
  }

Post #2

imsoulfly

The Spandex.ApiServer actually should in any case produce logs of the following kind when it triggers a sendout of the traces.

"Sending 1 traces, 1 spans."
"Trace: [<<a lot of structs>>]"

Just to confirm one thing. do you have the batch_size config in mind. You would need to at least collect 10 traces before the ApiServer process would consider sending the traces.

Also if the ApiServer would receive a response from the Datadog client it would produce this log line.

"Trace response: <<some response details>>"

More info about that config can be found here

Post #3

andym

the only response i see is messages like this

27/02/2023, 04:38:24 23:08:24.374 [info] Adding trace to stack with 2 spans

nothing apart from that, i also sent a bit of a load as well to get it above the 10 batch size but still nothing.

Post #4

xlphs

Try using netcat (nc) or telnet to check connectivity. Most likely the elixir app cannot reach datadog agent.

btw you can set the batch size to 1 to make debugging easier

Post #5

andym

thanks everyone for your replies, i finally got it working, and i can see the traces now. if anyone else is facing a similar problem amongst other things it was primarily because of the phoenix spandex, it was causing some version conflicts and httppoison was missing.

Post #6

— All posts loaded —