How to convert metrics data into nested JSON format

Where did you get stuck?

I’d split the task into two problems: parsing the text and building the output. The first one seems a bit more challenging since you need to build a parser. You can do it by hand, but you can also use Nimble Parsec here which could help make the code more readable.

While You should use a parser… You can also use Regex.

iex> Regex.named_captures ~r/(?<key>[^{]*)(?<opt>{.*})?\ (?<value>.*$)/, "app_name_replication_validation_duration_bucket{le=\"0.01\"} 953"
%{
  "key" => "app_name_replication_validation_duration_bucket",
  "opt" => "{le=\"0.01\"}",
  "value" => "953"
}

defmodule Parser do

  def run(content) do
    content
    |> String.split("\n", trim: true)
    |> Enum.reduce(%{current: %{}, metrics: []}, fn line, acc ->

      %{ content: line, current: acc.current, metrics: acc.metrics}
      |> parse_help()
      |> parse_type()
      |> parse_metric()
    end)
    |> finalize()
  end

  defp finalize(%{current: current, metrics: metrics}) do
    [current | metrics]
  end

  defp parse_help(acc = %{content: content, current: current}) do
    case Regex.run(~r/HELP (.*)/, content) do
      [_, name] ->
        case Map.get(current, :name) do
          nil ->
            Map.put(acc, :current, %{name: String.trim(name) })
          _ ->
            acc
            |> Map.update!(:metrics, &[current | &1])
            |> Map.put(:current, %{name: name |> String.trim() })
        end
        
      _ ->
        acc
    end
  end

  defp parse_type(acc = %{content: content, current: current}) do
    case Regex.run(~r/TYPE (.*) (.*)/, content) do
      [_, _, type] ->
      %{ acc | current: Map.put(current, :type, type) }
    _ ->
      acc
    end
  end

  defp parse_metric(acc = %{content: content, current: current = %{name: metric_name}}) do    
    case Regex.run(~r/^#{metric_name}_(.*){(.*)} (.*)$/, content) do
      [_, "bucket", labels, value] ->
        %{ acc | current: add_bucket(labels, current, value) }
      [_, "sum", _labels, value] ->
        %{ acc | current: Map.put(current, :sum, value) }
      [_, "count", _labels, value] ->
        %{ acc | current: Map.put(current, :count, value) }
      nil ->
        case Regex.run(~r/^#{metric_name} (.*)$/, content) do
          [_, value] ->
            %{ acc | current: Map.put(current, :value, value) }
          _ ->
            acc
        end
      _ ->
        acc
    end
  end

  defp add_bucket(labels, current, value) do
    labels
    |> String.split(",", trim: true)
    |> Enum.reduce(current, fn label, acc ->
      [k, v] = String.split(label, "=")

      case k do
        "le" ->
          Map.update(acc, :quantiles, %{ v => value }, &Map.put(&1, v, value))
        _ ->
          Map.update(acc, :labels, %{ k => v}, &Map.put(&1, k, v))
      end
    end)
  end
end

with this code i’m in half way but the challenge is updating nested maps, if you see json format above the metrics has two layer’s of nesting, updating this laters i’m facing challenges and other challenge is some lines has multiple labels and bucket values i have to capture that also

I like your approach, can you please elaborate it more

The idea would be to split the parsing phase from the transformation phase, so you end up with two, but smaller and isolated problems to solve. On a high-level the code would be:

def run(content) do
  content
  |> parse()
  |> to_json()
end

The job of parse would be to turn the text into some data structure, probably one that is close to the input. The job of to_json would be to transform the intermediate data structure to the desired JSON.

Looking just at the example you gave, I’d go with something like this for the data structure:

defmodule Metric do
  defstruct [
    # e.g. app_name_self_repair_duration
    :name,
    # "gauge" or "histogram"
    :type,
    # only for type "gauge"
    :gauge_value,
    # only for type "histogram"
    :histogram_buckets
    ]
end

defmodule HistogramBucket do
  defstruct [
    # e.g. get_transaction
    :name,
    # tuples {"0.01", 443481} or a separate struct
    :quantiles,
    :sum,
    :count,
  ]
end

It should be pretty straightforward to build a list of %Metric{}s from the input you’re given and then transform it to the output JSON.

thank you @stefanchrobot i’m working on some idea given by my senior

i’m done with converting text into JSON structure,
now i wanted to do Aggregate histogram metrics,
to know what is the response time in given milliseconds
i have these below details:

in less then or equal to 10ms["0.01"] => number of events occurred 443481

in less then 25ms["0.025"] => events 447919

in less then 50ms["0.05"] => events 456449

in less then 100ms["0.1"] => events 472222

in less then 300ms["0.3"] => events 472571

in less then 500ms["0.5"] => events 472605

in less then 800ms["0.8"] => events 472617

in less then 1000ms["1"] => events 472621

in less then 1500ms["1.5"] => events 472624

prometheus has built-in tools for it
elastic search also has its own tools
but i could not get what logic they applied?
because i wanted to do my own functions to calculate and then use echarts to show results on graph

what logic to apply? to calculate response time