neenjaw

neenjaw

Docker configuration for TelemetryMetricsStatsd

Looking for a bit of insight here to grow my skills around building observability into my elixir deployments.

I have a basic phoenix server and I am looking to publish statsd metrics to datadog via udp port 8125. I have a Telemetry module as produced by the phx bootstrapper without ecto that has TelemetryMetricsStatsd added to it:

Telemetry Module
defmodule StatsdDemoWeb.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def start_link(arg) do
    Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
  end

  @impl true
  def init(_arg) do
    children = [
      {TelemetryMetricsStatsd, metrics: metrics(), formatter: :datadog, port: 8125}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

  def metrics do
    [
      # Phoenix Metrics
      summary("phoenix.endpoint.stop.duration",
        unit: {:native, :millisecond}
      ),
      summary("phoenix.router_dispatch.stop.duration",
        tags: [:route],
        unit: {:native, :millisecond}
      ),
      counter("phoenix.socket_connected.count",
        tags: [:endpoint]
      ),

      # VM Metrics
      summary("vm.memory.total", unit: {:byte, :kilobyte}),
      summary("vm.total_run_queue_lengths.total"),
      summary("vm.total_run_queue_lengths.cpu"),
      summary("vm.total_run_queue_lengths.io")
    ]
  end
end

When I run this natively in dev/prod environments (via iex -S mix phx.server) I can verify that the vm and phoenix metrics are emitted using netcat (nc -ulkp 8125).

However, when I run this inside of a docker container, it appears that I can no longer get the metrics emitted to the udp port. The docker compose image is the default produced by the mix release docker command:

Dockerfile
# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian instead of
# Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
#
# This file is based on these images:
#
#   - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
#   - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20220801-slim - for the release image
#   - https://pkgs.org/ - resource for finding needed packages
#   - Ex: hexpm/elixir:1.14.0-erlang-25.1-debian-bullseye-20210902-slim
#
ARG ELIXIR_VERSION=1.14.0
ARG OTP_VERSION=25.1
ARG DEBIAN_VERSION=bullseye-20220801-slim

ARG NODE_BULDER_IMAGE="node:18.3.0-alpine3.16"
ARG ELIXIR_BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

########################################
# 1. Node build stage
########################################
FROM ${NODE_BULDER_IMAGE} as node-builder

# prepare build dir
WORKDIR /app
RUN mkdir assets

# set build ENV
ENV NODE_ENV=prod

# install npm dependencies
COPY ./services/statsd_demo/assets/package.json ./assets/
COPY ./services/statsd_demo/assets/package-lock.json ./assets/
RUN npm install --prefix assets

########################################
# 2. Elixir build stage
########################################
FROM ${ELIXIR_BUILDER_IMAGE} as elixir-builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
  mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"

# install mix dependencies
COPY ./services/statsd_demo/mix.exs ./
COPY ./services/statsd_demo/mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY ./services/statsd_demo/config/config.exs config/
COPY ./services/statsd_demo/config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY ./services/statsd_demo/priv priv

COPY ./services/statsd_demo/lib lib

COPY ./services/statsd_demo/assets assets
COPY --from=node-builder /app/assets/node_modules assets/node_modules

# compile assets
RUN mix assets.deploy

# Compile the release
RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY ./services/statsd_demo/config/runtime.exs config/

COPY ./services/statsd_demo/rel rel
RUN mix release

########################################
# 2. Build run stage
#
# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
########################################
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

WORKDIR "/app"
RUN chown nobody /app

# set runner ENV
ENV MIX_ENV="prod"

# Only copy the final release from the build stage
COPY --from=elixir-builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/socks ./

USER nobody

CMD ["/app/bin/server"]

I feel like I am missing something obvious, some declaration in the docker image or some configuration. Or perhaps it is working and I am testing it’s ability to emit from a docker container wrong.

Anyway, appreciate if someone could advise here or point out my lapse

Marked As Solved

neenjaw

neenjaw

I was modelling this off of another service (non-elixir) that I am intending to replace. Turns out there were obscure kubernetes configuration overlays injecting ENV_VARS to direct stats to a different host implicitly.

TL;DR, statsd needs to have a host outside of local host configured for the statsd packet to be sent properly.

Where Next?

Popular in Questions Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44265 214
New

We're in Beta

About us Mission Statement