Does Logger have a hidden cap for message size?

Questions / Help
,
1

Background

In one of our projects a client of ours complained that the logs he is getting are being capped at 2000 characters.

This client is gettign his logs via a tool called Splunk and perhaps some other systems I am not aware of.

Instead of capping the messages at 2K characters, I need to cap them at 8K.

Config

To me this is strange, because we specifically truncate the log to :infinity, as our config shows:

use Mix.Config

config :logger,
  level: :info,
  backends: [:console],
  utc_log: true,
  sync_threshold: 100,
  truncate: :infinity

if Mix.env() != :prod do
  config :logger,
    level: :debug
end

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [
    :module,
    :line,
    :function,
    :trace,
    :perf,
    :duration,
    :namespace
  ]

Furthermore, I didn’t find any specific Logger limits documented:

https://hexdocs.pm/logger/Logger.html

Question

  • Does the Logger have some internal limit that cuts messages down to 2000 characters? If so, how can I change it?
2

At least on the console backend I’m not seeing such a limit. You can test it easily with:

iex> require Logger
iex> Logger.debug String.duplicate("A", 8000)
:ok

23:04:58.307 [debug] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.......
1 Like
3

Just for giggles I put it in a test case:

defmodule ThingTest do
  use ExUnit.Case
  import ExUnit.CaptureLog
  require Logger

  @message_size 8_000

  test "Logger backend end" do
    assert capture_log(fn -> Logger.error(String.duplicate("A", @message_size)) end) >= @message_size
  end
end

And ran the test:

kip@Kips-iMac-Pro thing % mix test
.

Finished in 0.04 seconds
1 test, 0 failures
2 Likes
4

Thanks for the super quick answer!
I have made the test you have mentioned in the first post and concluded that it is very unlikely for the issue to be in the Elixir code we have.

Once again thank you!

5

Is it perhaps the :truncate option? By default it’s 8kiB.

:truncate - the maximum message size to be logged (in bytes). Defaults to 8192 bytes. Note this configuration is approximate. Truncated messages will have " (truncated)" at the end. The atom :infinity can be passed to disable this behavior.

1 Like
6

FWIW there is a limiter (see @NobbZ’s post) but there can also be intermediate steps that trim things shorter - at work, lines long enough to hit Logger’s limit (and get (truncated) at the end) are still split into multiple lines someplace between Logger -> syslog -> CloudWatch. Not sure what specifically is causing that…

1 Like
7

just so my headache is logged somewhere.

there’s two different layers you can config in the logger. one for backends, one for logger itself. so in my case this helper a lot:

config :logger, truncate: :infinity
config :logger, :console, truncate: :infinity

thanks goes to @JonRowe and @LostKobrakai in Slack.

8 Likes
8

Apparently, the first line is enough. So happy to have found it :raised_hands:t2:

1 Like
9

Updates that fit the current situation:

disable log truncation

As of Elixir 1.14:

config :logger, truncate: :infinity is enough for disabling log truncation.

config :logger, :console, truncate: doesn’t exist anymore.

log data

Another point to note if you are planing to print data with inspect/2:

Logger.info( inspect(data) )

Make sure that inspect will print all the information you want. The following line is a good start:

Logger.info( inspect(data, structs: false, limit: :infinity, printable_limit: :infinity) )

performance consideration

If the data is large, carefully consider before using the above two steps.

4 Likes
10

To enable infinite truncation in the new Logger (since Elixir 1.15) do:

config :logger, :default_formatter,
  ...,
  truncate: :infinity
1 Like