Ciboulette

Killing Registry Process, shutdown the entire application

Hello,

I am writing a fault tolerant application in Elixir, so I kill every process to check the way they handle restarting etc. I start a Registry from the Default Supervisor, when i try killing it, it shutdown the entire application “Application worker exited: shutdown”. Do you know how to handle a simple fresh restart?

defmodule Toto.Application do
    @moduledoc """
        Entry point of the toto application
    """
    use Application
    require Logger
    use Supervisor

    def start(_type, _args) do
        Logger.info "Starting #{__MODULE__}"
        Supervisor.start_link(__MODULE__, [], name: __MODULE__)
    end

    def init(_) do
        Supervisor.init([
            {Registry, keys: :unique, name: Worker.ProcessRegistry}
        ], strategy: :one_for_one)
    end
end

13 comments

#supervisor #registry

0 3543 13

2019-09-22 18:44:45 UTC

First 10 of 13 Posts

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Can you show how you’re killing it?

Post #1

Ciboulette

Process.exit(Process.whereis(Worker.ProcessRegistry), :kill)

Post #2

kelvinst

That’s very interesting. Did you discover what is it?

Post #3

isaacsanders

What does Process.exit(Process.whereis(Worker.ProcessRegistry), :normal) do?

Post #4

Ciboulette

No i didn’t find yet the exact answer..

Post #5

kelvinst

I’ve tested out, Process.exit(Process.whereis(Worker.ProcessRegistry), :normal) does not stop the process. Looks like Registry is a very special type of process, I was not expecting this behaviour in any way.

Post #6

isaacsanders

Another question.

What do you expect killing the Registry to do?

Registries are collections of names when used with :via and (I think) a complex process dictionary otherwise.

It is also a supervisor. It creates a named ETS table. The Registry module functions call into the ETS table. I imagine if you have a number of processes that interacting with this Registry, killing it might be a Really Bad™ thing. It also might not be super common since it shouldn’t be affected by your application code.

There might be a bug in Registry, but I am also thinking this might not be a place where faults happen.

Post #7

kelvinst

I guess @Ciboulette’s point is that he was testing the fault tolerance on every part of his system, and it’s very odd that killing a Registry shuts down the whole application. He’s not doing that on his system, only trying out possible failure points.

Faults can happen anywhere, and being fault tolerant is a basic property of BEAM, that’s why I’m very intrigued by this.

So I have managed to repeat the problem and traced it out:

15:48:56:912029 (<0.132.0>) getting_unlinked <0.133.0>
15:48:56:912032 (<0.132.0>) << {'EXIT',<0.133.0>,killed}
15:48:56:912207 (<0.132.0>) spawn <0.552.0> as proc_lib:init_p('Elixir.Toto.Supervisor',[<0.131.0>],gen,init_it,[gen_server,<0.132.0>,<0.132.0>,
 {local,'Elixir.Worker.ProcessRegistry'},
 supervisor,
 {{local,'Elixir.Worker.ProcessRegistry'},
  'Elixir.Registry.Supervisor',
  {unique,'Elixir.Worker.ProcessRegistry',1,[],[{-1,{unique,1,nil,nil,[]}},{-2,{unique,1,nil}}]}},
 []])
15:48:56:912226 (<0.132.0>) link <0.552.0>
15:48:56:912233 (<0.132.0>) out {proc_lib,sync_wait,2}
15:48:56:912724 (<0.132.0>) in {proc_lib,sync_wait,2}
15:48:56:912731 (<0.132.0>) << {ack,<0.552.0>,
                                   {error,
                                       {shutdown,
                                           {failed_to_start_child,
                                               'Elixir.Worker.ProcessRegistry.PIDPartition0',
                                               {already_started,<0.134.0>}}}}}
15:48:56:912737 (<0.132.0>) getting_unlinked <0.552.0>
15:48:56:912738 (<0.132.0>) << {'EXIT',<0.552.0>,
                                   {shutdown,
                                       {failed_to_start_child,
                                           'Elixir.Worker.ProcessRegistry.PIDPartition0',
                                           {already_started,<0.134.0>}}}}
15:48:56:912762 (<0.132.0>) <0.132.0> ! {'$gen_cast',
                                            {try_again_restart,'Elixir.Worker.ProcessRegistry'}}
15:48:56:912769 (<0.132.0>) << {'$gen_cast',{try_again_restart,'Elixir.Worker.ProcessRegistry'}}
15:48:56:912788 (<0.132.0>) spawn <0.553.0> as proc_lib:init_p('Elixir.Toto.Supervisor',[<0.131.0>],gen,init_it,[gen_server,<0.132.0>,<0.132.0>,
 {local,'Elixir.Worker.ProcessRegistry'},
 supervisor,
 {{local,'Elixir.Worker.ProcessRegistry'},
  'Elixir.Registry.Supervisor',
  {unique,'Elixir.Worker.ProcessRegistry',1,[],[{-1,{unique,1,nil,nil,[]}},{-2,{unique,1,nil}}]}},
 []])
15:48:56:912793 (<0.132.0>) link <0.553.0>
15:48:56:912797 (<0.132.0>) out {proc_lib,sync_wait,2}
15:48:56:913611 (<0.132.0>) in {proc_lib,sync_wait,2}
15:48:56:913622 (<0.132.0>) << {ack,<0.553.0>,
                                   {error,
                                       {shutdown,
                                           {failed_to_start_child,
                                               'Elixir.Worker.ProcessRegistry.PIDPartition0',
                                               {already_started,<0.134.0>}}}}}
15:48:56:913632 (<0.132.0>) getting_unlinked <0.553.0>
15:48:56:913635 (<0.132.0>) << {'EXIT',<0.553.0>,
                                   {shutdown,
                                       {failed_to_start_child,
                                           'Elixir.Worker.ProcessRegistry.PIDPartition0',
                                           {already_started,<0.134.0>}}}}
15:48:56:913692 (<0.132.0>) <0.132.0> ! {'$gen_cast',
                                            {try_again_restart,'Elixir.Worker.ProcessRegistry'}}
15:48:56:913702 (<0.132.0>) << {'$gen_cast',{try_again_restart,'Elixir.Worker.ProcessRegistry'}}
15:48:56:913733 (<0.132.0>) spawn <0.554.0> as proc_lib:init_p('Elixir.Toto.Supervisor',[<0.131.0>],gen,init_it,[gen_server,<0.132.0>,<0.132.0>,
 {local,'Elixir.Worker.ProcessRegistry'},
 supervisor,
 {{local,'Elixir.Worker.ProcessRegistry'},
  'Elixir.Registry.Supervisor',
  {unique,'Elixir.Worker.ProcessRegistry',1,[],[{-1,{unique,1,nil,nil,[]}},{-2,{unique,1,nil}}]}},
 []])
15:48:56:913741 (<0.132.0>) link <0.554.0>
15:48:56:913748 (<0.132.0>) out {proc_lib,sync_wait,2}
15:48:56:914126 (<0.132.0>) in {proc_lib,sync_wait,2}
15:48:56:914195 (<0.132.0>) << {ack,<0.554.0>,
                                   {error,
                                       {shutdown,
                                           {failed_to_start_child,
                                               'Elixir.Worker.ProcessRegistry.PIDPartition0',
                                               {already_started,<0.134.0>}}}}}
15:48:56:914216 (<0.132.0>) getting_unlinked <0.554.0>
15:48:56:914218 (<0.132.0>) << {'EXIT',<0.554.0>,
                                   {shutdown,
                                       {failed_to_start_child,
                                           'Elixir.Worker.ProcessRegistry.PIDPartition0',
                                           {already_started,<0.134.0>}}}}
15:48:56:914274 (<0.132.0>) <0.132.0> ! {'$gen_cast',
                                            {try_again_restart,'Elixir.Worker.ProcessRegistry'}}
15:48:56:914288 (<0.132.0>) << {'$gen_cast',{try_again_restart,'Elixir.Worker.ProcessRegistry'}}
15:48:56:914329 (<0.132.0>) exit shutdown
15:48:56:914334 (<0.132.0>) unregister 'Elixir.Toto.Supervisor'
15:48:56:914337 (<0.132.0>) out_exited 0

I’m not so experient tracing, so any help would be awesome!

Post #8

axelson

Scenic Core Team

I think that this blog post is relevant to this thread. You might want to consider the registry process as part of the “Error Kernel”

Post #9

isaacsanders

Your trace makes it appear that the registry’s children don’t die/aren’t in registered from their name registry when the registry supervisor is killed. The named processes are still alive, seen as already_started.

At least that is the appearance it has to me.

So this sounds like a bug in Registry.

Post #10

Load more posts