Fl4m3Ph03n1x

Does Logger have a hidden cap for message size?

Background

In one of our projects a client of ours complained that the logs he is getting are being capped at 2000 characters.

This client is gettign his logs via a tool called Splunk and perhaps some other systems I am not aware of.

Instead of capping the messages at 2K characters, I need to cap them at 8K.

Config

To me this is strange, because we specifically truncate the log to :infinity, as our config shows:

use Mix.Config

config :logger,
  level: :info,
  backends: [:console],
  utc_log: true,
  sync_threshold: 100,
  truncate: :infinity

if Mix.env() != :prod do
  config :logger,
    level: :debug
end

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [
    :module,
    :line,
    :function,
    :trace,
    :perf,
    :duration,
    :namespace
  ]

Furthermore, I didn’t find any specific Logger limits documented:

Question

Does the Logger have some internal limit that cuts messages down to 2000 characters? If so, how can I change it?

14 comments

#logging #logger

0 7334 14

2025-01-31 13:24:20 UTC

Marked As Solved

kip

ex_cldr Core Team

At least on the console backend I’m not seeing such a limit. You can test it easily with:

iex> require Logger
iex> Logger.debug String.duplicate("A", 8000)
:ok

23:04:58.307 [debug] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.......

Post #2

Also Liked

KristerV

just so my headache is logged somewhere.

there’s two different layers you can config in the logger. one for backends, one for logger itself. so in my case this helper a lot:

config :logger, truncate: :infinity
config :logger, :console, truncate: :infinity

thanks goes to @JonRowe and @LostKobrakai in Slack.

Post #7

c4710n

Updates that fit the current situation:

disable log truncation

As of Elixir 1.14:

config :logger, truncate: :infinity is enough for disabling log truncation.

config :logger, :console, truncate: doesn’t exist anymore.

log data

Another point to note if you are planing to print data with inspect/2:

Logger.info( inspect(data) )

Make sure that inspect will print all the information you want. The following line is a good start:

Logger.info( inspect(data, structs: false, limit: :infinity, printable_limit: :infinity) )

performance consideration

If the data is large, carefully consider before using the above two steps.

Post #9

kip

ex_cldr Core Team

Just for giggles I put it in a test case:

defmodule ThingTest do
  use ExUnit.Case
  import ExUnit.CaptureLog
  require Logger

  @message_size 8_000

  test "Logger backend end" do
    assert capture_log(fn -> Logger.error(String.duplicate("A", @message_size)) end) >= @message_size
  end
end

And ran the test:

kip@Kips-iMac-Pro thing % mix test
.

Finished in 0.04 seconds
1 test, 0 failures

Post #3

Last Post!

harrisi

Yeah, in that case I think just doing something like

def migrate do
  formatter = Logger.default_formatter(truncate: :infinity)
  :logger.update_handler_config(:default, :formatter, formatter)

  # do migrations

  :logger.update_handler_config(:default, :formatter, Logger.default_formatter())
end

would suffice. You’d maybe want to take care to ensure that if the migrations fail or something you still properly update the handler, but that may not matter in this case.

Just for posterity, here’s the other option I mentioned:

# config/config.exs

config :logs, :logger, [
  {
    :handler,
    :custom_logger,
    :logger_std_h,
    %{
      filters: [
        remote_gl: {&:logger_filters.remote_gl/2, :stop},
        custom_filter: {&Logs.custom_filter/2, []}
      ],
      filter_default: :log,
      formatter: Logger.default_formatter(truncate: :infinity),
      level: :all,
      module: :logger_std_h
    }
  }
]

# lib/logs.ex
defmodule Logs do
  def custom_filter(%{meta: %{no_truncate: true}} = event, _extra) do
    event
  end
  def custom_filter(_event, _extra), do: :stop
end

# lib/logs/application.ex

defmodule Logs.Application do
  use Application

  def start(_type, _args) do
    Logger.add_handlers(:logs)

    # ...
  end

# ...
end

Then when you need to log something without truncation, you would just do something like:

Logger.info("this won't be truncated", no_truncate: true)

One issue with that is that the default handler will still log it, truncated, but you’ll also get the non-truncated form. If you want to avoid that you would need to add an inverse filter to the default handler, like:

# config/config.exs
config :logger, :default_handler,
  filters: [
    inverse_custom_filter: {&Logs.inverse_custom_filter/2, []}
  ]

# lib/logs.ex
defmodule Logs do
  # ...

  def inverse_custom_filter(event, extra) do
    if custom_filter(event, extra) == :stop do
      event
    else
      :stop
    end
  end

  # ...
end

Maybe I should’ve written a blog post instead.