Logger crashes after upgrade to Erlang/OTP 22.1.1 & Elixir 1.9.1

So I’ve run into a mysterious error saying that :re.run/3 is undefined or private while upgrading my phoenix 14.10 based app to Erlang/OTP 22.1.1 and Elixir 1.9.1 when trying to restart the release in production.

Is there something obvious that I’ve missed, like compiling OTP with a certain flag?

I would deeply appreciate any help and can provide more info if needed.

Below is the interesting part of the crash report:

=CRASH REPORT==== 27-Sep-2019::23:05:22.988338 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.1896.0>
    registered_name: []
    exception exit: {{shutdown,
                      {failed_to_start_child,
                       'Elixir.Logger.BackendSupervisor',
                       {'EXIT',
                        {#{'__exception__' => true,
                           '__struct__' => 'Elixir.RuntimeError',
                           message =>
                            <<"EXIT when installing backend :console: an exception was raised:\n    ** (UndefinedFunctionError) function :re.run/3 is undefined or private\n        (stdlib) re.erl:780: :re.run(\"$time $metadata[$level] $message\\n\", \"(?<head>)\\\\$[a-z]+(?<tail>)\", [:global, {:capture, [:head, :tail]}])\n        (elixir) lib/regex.ex:537: Regex.split/3\n        (logger) lib/logger/formatter.ex:94: Logger.Formatter.compile/1\n        (logger) lib/logger/backends/console.ex:101: Logger.Backends.Console.init/2\n        (logger) lib/logger/backends/console.ex:22: Logger.Backends.Console.init/1\n        (stdlib) gen_event.erl:473: :gen_event.server_add_handler/4\n        (stdlib) gen_event.erl:318: :gen_event.handle_msg/6\n        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3">>},
                         [{'Elixir.Logger.BackendSupervisor',
                           '-start_link/1-fun-0-',2,
                           [{file,"lib/logger/backend_supervisor.ex"},
                            {line,19}]},
                          {'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,
                           [{file,"lib/enum.ex"},{line,1948}]},
                          {'Elixir.Logger.BackendSupervisor',start_link,1,
                           [{file,"lib/logger/backend_supervisor.ex"},
                            {line,13}]},
                          {supervisor,do_start_child_i,3,
                           [{file,"supervisor.erl"},{line,379}]},
                          {supervisor,do_start_child,2,
                           [{file,"supervisor.erl"},{line,365}]},
                          {supervisor,'-start_children/2-fun-0-',3,
                           [{file,"supervisor.erl"},{line,349}]},
                          {supervisor,children_map,4,
                           [{file,"supervisor.erl"},{line,1157}]},
                          {supervisor,init_children,2,
                           [{file,"supervisor.erl"},{line,315}]}]}}}},
                     {'Elixir.Logger.App',start,[normal,[]]}}
      in function  application_master:init/4 (application_master.erl, line 138)
    ancestors: [<0.1895.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.1897.0>,normal}]
    links: [<0.1895.0>,<0.1843.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 239
  neighbours:

=INFO REPORT==== 27-Sep-2019::23:05:22.991261 ===
    application: logger
    exited: {{shutdown,
                 {failed_to_start_child,'Elixir.Logger.BackendSupervisor',
                     {'EXIT',
                         {#{'__exception__' => true,
                            '__struct__' => 'Elixir.RuntimeError',
                            message =>
                                <<"EXIT when installing backend :console: an exception was raised:\n    ** (UndefinedFunctionError) function :re.run/3 is undefined or private\n        (stdlib) re.erl:780: :re.run(\"$time $metadata[$level] $message\\n\", \"(?<head>)\\\\$[a-z]+(?<tail>)\", [:global, {:capture, [:head, :tail]}])\n        (elixir) lib/regex.ex:537: Regex.split/3\n        (logger) lib/logger/formatter.ex:94: Logger.Formatter.compile/1\n        (logger) lib/logger/backends/console.ex:101: Logger.Backends.Console.init/2\n        (logger) lib/logger/backends/console.ex:22: Logger.Backends.Console.init/1\n        (stdlib) gen_event.erl:473: :gen_event.server_add_handler/4\n        (stdlib) gen_event.erl:318: :gen_event.handle_msg/6\n        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3">>},
                          [{'Elixir.Logger.BackendSupervisor',
                               '-start_link/1-fun-0-',2,
                               [{file,"lib/logger/backend_supervisor.ex"},
                                {line,19}]},
                           {'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,
                               [{file,"lib/enum.ex"},{line,1948}]},
                           {'Elixir.Logger.BackendSupervisor',start_link,1,
                               [{file,"lib/logger/backend_supervisor.ex"},
                                {line,13}]},
                           {supervisor,do_start_child_i,3,
                               [{file,"supervisor.erl"},{line,379}]},
                           {supervisor,do_start_child,2,
                               [{file,"supervisor.erl"},{line,365}]},
                           {supervisor,'-start_children/2-fun-0-',3,
                               [{file,"supervisor.erl"},{line,349}]},
                           {supervisor,children_map,4,
                               [{file,"supervisor.erl"},{line,1157}]},
                           {supervisor,init_children,2,
                               [{file,"supervisor.erl"},{line,315}]}]}}}},
             {'Elixir.Logger.App',start,[normal,[]]}}
    type: permanent
{"Kernel pid terminated",application_controller,"{application_start_failure,logger,{{shutdown,{failed_to_start_child,'Elixir.Logger.BackendSupervisor',{'EXIT',{#{'__exception__' => true,'__struct__' => 'Elixir.RuntimeError',message => <<\"EXIT when installing backend :console: an exception was raised:\n    ** (UndefinedFunctionError) function :re.run/3 is undefined or private\n        (stdlib) re.erl:780: :re.run(\\"$time $metadata[$level] $message\\n\\", \\"(?<head>)\\\\$[a-z]+(?<tail>)\\", [:global, {:capture, [:head, :tail]}])\n        (elixir) lib/regex.ex:537: Regex.split/3\n        (logger) lib/logger/formatter.ex:94: Logger.Formatter.compile/1\n        (logger) lib/logger/backends/console.ex:101: Logger.Backends.Console.init/2\n        (logger) lib/logger/backends/console.ex:22: Logger.Backends.Console.init/1\n        (stdlib) gen_event.erl:473: :gen_event.server_add_handler/4\n        (stdlib) gen_event.erl:318: :gen_event.handle_msg/6\n        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3\">>},[{'Elixir.Logger.BackendSupervisor','-start_link/1-fun-0-',2,[{file,\"lib/logger/backend_supervisor.ex\"},{line,19}]},{'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,[{file,\"lib/enum.ex\"},{line,1948}]},{'Elixir.Logger.BackendSupervisor',start_link,1,[{file,\"lib/logger/backend_supervisor.ex\"},{line,13}]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,379}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,365}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,349}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1157}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,315}]}]}}}},{'Elixir.Logger.App',start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,logger,{{shutdown,{failed_to_start_child,'Elixir.Logger.BackendSupervisor',{'EXIT',{#{'__exception__' => true,'__struct__' =>
1 Like

Just a quick test regarding versions.

I use Elixir 1.9.1-otp-22 and tried with both Erlang/OTP 22.1 and 22.1.1. Results are identical:

iex(1)> :re.run("$time $metadata[$level] $message\n", "(?<head>)\\$[a-z]+(?<tail>)", [:global, {:capture, [:head, :tail]}])
{:match,
 [[{0, 0}, {5, 0}], [{6, 0}, {15, 0}], [{16, 0}, {22, 0}], [{24, 0}, {32, 0}]]}

So it has to be something else.

Have you compiled it with PCRE support? How did you installed your Erlang VM?

I just ran into the same issue. Turns out there was a mismatch between the Erlang version the app was compiled with, and the ERTS that distillery shipped – there is some discussion on Github around that as well.

I manually fixed it by adjusting the ERTS version in the var/start_erl.data file. I hope that helps!

5 Likes