Auto rotating logs by file size

Questions / Help
, ,
1

Installed:

logger_file_backend 0.0.10

logrotate for Ubuntu (config file)

Trying to get logs to automatically rotate when they hit 125MB with logrotate to no avail. How can logs be automatically rotated straight from Elixir?

2

Are you using journalctl?

You can set time or size limits for rotation, deletion

3

Yes you can, by the way of Erlang/OTP logging handler :logger_disk_log_h. The relevant documentation is in Logger and Erlang

You need to mentally combine the two doc in your head to have the complete picture.

Also you should keep a less chatty log to the console for your own good, use the OS provided facilities to rotate it. Keep the chatty log separate from the default and do whatever you want on this one.

4

Thanks for the suggestion. Here’s my latest attempt:

# config.exs
import Config

config :myapp, :logger,
  [
    {:handler, :default, :logger_disk_log_h, %{level: :info, file: 'test.log', max_no_bytes: 100_000_000}}
  ]

config :logger,
  utc_log: true,
  backends: [
    {LoggerFileBackend, :warnings_log},
    {LoggerFileBackend, :info_log}
  ]

config :logger, :warnings_log,
  path: "/home/myapp/Logs/warnings.log",
  level: :warn

config :logger, :info_log,
  path: "/home/myapp/Logs/info.log",
  level: :info

# application.ex
defmodule MyApp.Application do
  require Logger
  use Application

  def start(_type, _args) do
    :logger.add_handlers(:myapp)

    Logger.info("Initiating My App \n")
  end

Logs still don’t auto-rotate. What did I miss? :slight_smile:

5

Read carefully here, it is a map within a map.
Also I do not suggest you change the :default handler. I am not sure if it is possible.

6

What I meant is:

config :myapp, :logger,
  [
    {:handler, :my_disk_log, :logger_disk_log_h,
         %{config: %{file: 'test.log',
                     max_no_bytes: 100_000_000},
           level: :info}
    }
  ]

Untested, but you get the idea. It is a 2 level map, and each have it own set of keys. The outer level is the general configuration for any log_handler, the inner one is the configuration specifically for :logger_disk_log_h

1 Like
7

Success! :slight_smile:

Here’s the new config:

config :myapp, :logger, [
  {:handler, :myapp_disk_log, :logger_disk_log_h,
   %{
     config: %{file: 'Logs/info.log', max_no_bytes: 100_000_000},
     level: :info,
     formatter_config: %{time_offset: 0}
   }}
]

config :logger,
  utc_log: true,
  backends: [
    {LoggerFileBackend, :info_log}
  ]

Now, based on what the manual indicates here and here, formatter_config: %{time_offset: 0} was added to set the timestamp to UTC. However, this is not working. Where did I go wrong?

8

for formatter, it should be a map inside a tuple:

    formatter: {:logger_formatter, %{...}}

This thing is designed to be VERY flexible and powerful. Not the easiest API to use

9

Changed formatter_config: %{time_offset: to formatter: {:logger_formatter, %{time_offset: 0} and now logs are being recorded in UTC. (Also got rid of utc_log: true,)

Thanks for helping translate the Erlang into Elixir :slight_smile:

10

Do not listem to @derek-zhou and do not use :logger_disk_log_h. That one is left for compatibility reasons and :logger_std_h instead. It works better and in more reasonable way.

11

Care to elaborate more? I am still using Erlang 21 and :logger_std_h in 21 does not support circular log file rotation.

12

It supports it from the very beginning, you just need to configure max_no_bytes (by default set to infinity) and max_no_files (by default set to 0).

13

From the erlang 21 doc on logger_std_h:

Notice that the standard handler does not have support for circular logging. Use the disk_log handler, logger_disk_log_h, for this.

I checked the document I installed locally and it said the same. If you know more please share it. I can only refer to the official documentation.

14

Apparently logger_std_h gains circular rotation in OTP-15479 which is released in 2019-03-12 (roughly 2 years ago). And it was marked internal at that time.

In any case, it is dead simple to migrate to logger_std_h if anyone want to. all configurations are basically the same.

2 Likes
15

How do you mean, exactly?

16

I stand corrected.

I cannot find information about that in the change log, neither in the docs. It seems like it was public from OTP 21.3.

Assume that you have files:

  • log.0
  • log.1
  • log.2

If you use logger_disk_log_h you need to check out which file is currently used by logger via checking their last change timestamp (as these will be used in circular fashion, “wrapping” as it is defined by disk_log). If you use logger_std_h then log.0 is always the newest one (as it works more like logrotate). In general disk_log isn’t very good thing to work with, especially if you use it with external format.

17

I was referring to this sentence in the changelog. Probably “internal” does not mean non-public:

The standard logger handler, logger_std_h, now has a new internal feature for log rotation.

As for filename rotation, I prefer the scheme of svlogd from runit, which is somewhere between the two:

  • lastest file name is fixed
  • old files are not renamed

On one hand I want the latest file easily identifiable. On the other hand, I prefer old files just stay what they are. If something went wrong and the log files are rotating fast it is very frustrating to pin down the one file that has the log entries you want to analysis.

1 Like
18

It meant how it is implemented, not about “publicity” of an API.

It seems like more reasonable approach, indeed. However changing it now would be a breaking change. However I hope that logger_olp and logger_sup will became public API in near future, which mean that there will be a way to implement such feature directly in Erlang. However personally I still prefer to use syslog or journal for logging, and do not bother with rotation within the application itself. In that way, there is no problem with rotation or “where will logs land” as you have centralised place to manage everything.

1 Like