Difference between Supervisors and DynamicSupervisors what am I missing?

[thiagomajesk]

Hi! I’ve been diving deeper into Elixir recently and reading through some of the guides and docs about Processes and Supervisors. After reading Dynamic supervisors - The Elixir programming language and comparing the docs from Supervisor — Elixir v1.13.4 and DynamicSupervisor — Elixir v1.13.4 I’m not sure I fully grasped the advantages of DynamicSupervisors.

I think the confusion is mainly because I’ve seen that both modules have the same function to start a child “dynamically”: Supervisor.start_child/2 and DynamicSupervisor.start_child/2. However, every guide I read, states that DynamicSupervisors’s main purpose is to add children to the supervision tree dynamically. What am I missing here?

There’s an example in this code base that shows a Supervisor adding children dynamically as well, so I’m not sure I understand the advantages of one over the other.

PS.: Not directly related to the topic, but I also notice that Supervisor.child_spec/2 and DynamicSupervisor.child_spec/1 have some different semantics that seems not documented. The latter doesn’t allow us to pass the module and uses the name option to define the spec id.

[derek-zhou]

I think the DynamicSupervisor is only a special case of Supervisor that handle only dynamic children. In Erlang there is only supervisor. It is a good habit to separate the static children and the dynamic children in different branches of the supervision tree.

[msimonborg]

The way that codebase is using Supervisor.start_child/2

  def init(_args) do
    children = [
      World.Location
    ]

    Supervisor.init(children, strategy: :simple_one_for_one)
  end

  def start_child(opts) do
    Supervisor.start_child(__MODULE__, [opts])
  end

is deprecated

  def start_child(supervisor, args) when is_list(args) do
    IO.warn_once(
      {__MODULE__, :start_child},
      "Supervisor.start_child/2 with a list of args is deprecated, please use DynamicSupervisor instead",
      _stacktrace_drop_levels = 2
    )

    call(supervisor, {:start_child, args})
  end

  # in def init/2
        :simple_one_for_one ->
          IO.warn(
            ":simple_one_for_one strategy is deprecated, please use DynamicSupervisor instead"
          )

Migrating from Supervisor’s :simple_one_for_one

I think if you have a supervisor with “regular” strategy, e.g. :one_for_one, and start processes with Supervisor.start_child/2 it’s assumed they’ll be permanent. DynamicSupervisor should be used for dynamic start/termination of arbitrary children.

@thiagomajesk

PS.: Not directly related to the topic, but I also notice that Supervisor.child_spec/2 and DynamicSupervisor.child_spec/1 have some different semantics that seems not documented.

Supervisor.child_spec/2 is a unique function which receives any child spec (Module, {Module, arg}, or map) and a list of overrides, and returns that child spec with the overrides applied. DynamicSupervisor.child_spec/1 returns a child spec to start a DynamicSupervisor under a supervisor, the same way Registry.child_spec/1 returns a child spec to start a Registry under a supervisor.

iex> DynamicSupervisor.child_spec(name: MyApp.DynamicSupervisor)
%{
  id: MyApp.DynamicSupervisor,
  start: {DynamicSupervisor, :start_link, [[name: MyApp.DynamicSupervisor]]},
  type: :supervisor
}
iex> |> Supervisor.child_spec(id: OverriddenID, shutdown: 10_000)
%{
  id: OverriddenID,
  shutdown: 10000,
  start: {DynamicSupervisor, :start_link, [[name: MyApp.DynamicSupervisor]]},
  type: :supervisor
}

[josevalim]

I improved the docs to say:

The Supervisor module was designed to handle mostly static children
that are started in the given order when the supervisor starts. A
DynamicSupervisor starts with no children. Instead, children are
started on demand via start_child/2 and there is no ordering between
children. This allows the DynamicSupervisor to hold millions of
children by using efficient data structures and to execute certain
options, such as shutting down, concurrently.

[thiagomajesk]

@derek-zhou

I think the DynamicSupervisor is only a special case of Supervisor that handle only dynamic children. In Erlang there is only supervisor. It is a good habit to separate the static children and the dynamic children in different branches of the supervision tree.

This is an important distinction that was not clear to me in the docs or guides, this makes things a lot clearer, thanks!

@msimonborg

The way that codebase is using Supervisor.start_child/2 is deprecated

I’m aware that this strategy is deprecated from reading lots and lots of old posts about it . However, I was still confused that we could add children dynamically to Supervisor because of the wording on the guides.

@msimonborg

Supervisor.child_spec/2 is a unique function which receives any child spec (Module, {Module, arg}, or map) and a list of overrides, and returns that child spec with the overrides applied. DynamicSupervisor.child_spec/1 returns a child spec to start a DynamicSupervisor under a supervisor, the same way Registry.child_spec/1 returns a child spec to start a Registry under a supervisor

I saw they were different by looking at the implementation, but have you found this documented somewhere? The documentation of child_spec/1 from DynamicSupervisor only points to the Supervisor docs.

@msimonborg

I think if you have a supervisor with “regular” strategy, e.g. :one_for_one, and start processes with Supervisor.start_child/2 it’s assumed they’ll be permanent. DynamicSupervisor should be used for dynamic start/termination of arbitrary children.

What do you mean by “permanent” in this case? Besides the point on efficiency that @josevalim made, how would that work? I just want to get the wording right here, perhaps I’m not understanding what “permanent” and “dynamic” truly means in this scenario.

@josevalim

I improved the docs to say:

The Supervisor module was designed to handle mostly static children
that are started in the given order when the supervisor starts. A
DynamicSupervisor starts with no children. Instead, children are
started on demand via start_child/2 and there is no ordering between
children. This allows the DynamicSupervisor to hold millions of
children by using efficient data structures and to execute certain
options, such as shutting down, concurrently.

Hey @josevalim, I think what has been tripping me up in the docs and guides is that there’s some nuance in the language that doesn’t make the distinction immediately obvious. Improving on your suggestion, I think this would make it a little bit clear, what do you think?

Differently from the Supervisor module, which requires its children to be given when the supervisor starts, a DynamicSupervisor starts with no children. Instead, children are only started on demand via start_child/2 and there is no ordering between them. This allows the DynamicSupervisor to hold millions of children by using efficient data structures and to execute certain options, such as shutting down, concurrently.

By the way, is there some way we can improve the reference on the other side as well? When I read this section, that states: To dynamically supervise children, see DynamicSupervisor, it seems that this is the only way it can be done and not just the most efficient way as you mentioned (it’s my general perception while reading the guides as well).

[msimonborg]

@thiagomajesk

I saw they were different by looking at the implementation, but have you found this documented somewhere? The documentation of child_spec/1 from DynamicSupervisor only points to the Supervisor docs.

IMO the documentation seems clear about it, but I could see how it would be confusing that they’re both supervisors and DyanmicSupervisor says “see Supervisor”, but FWIW it does not say “see Supervisor.child_spec/2”. I think it’s pointing to the Supervisor docs in general for more info about child specs. Maybe it could link directly to the child_spec/1 section. Returns a specification to start a dynamic supervisor under a supervisor is very different from what’s written in the Supervisor.child_spec/2 doc. The doc for DynamicSupervisor.child_spec/1 is much more similar to Registry.child_spec/1 or Agent.child_spec/1.

@thiagomajesk

What do you mean by “permanent” in this case?

I’ll admit this is maybe a bad or incomplete assumption but gleaned from what I’ve seen written elsewhere on the forum. I think José’s new documentation makes it more clear that DynamicSupervisor is preferred for efficiency of starting and stopping thousands or millions of processes, because there is no specific ordering of them. My takeaway which may be incorrect is that when you add a child spec to a Supervisor it will always be added to the end of the child list and never removed, and shutdown order is always strictly preserved.

[dimitarvp]

@thiagomajesk

What do you mean by “permanent” in this case?

Permanent == started on app startup and staying there until the app shuts down.

Thanks to this distinction, DynamicSupervisor allows you to start and stop children processes at any time. You also are not strictly mandated to stop them; they will be stopped when the supervisor itself is stopped (although it’s definitely a good idea to stop children proactively after they’ve done their job IMO).

To me, using DynamicSupervisor is appropriate in two scenarios:

1. When you would like to spawn quite a lot of (say, at least 200,000+) processes because I’d wager then a performance (or memory) impact would start becoming noticeable.
2. When you need to create processes in response to – or because of the specifics of – external systems whose resources you have to monitor e.g. a web crawling service to which you give a new domain to crawl. It would need to spawn processes to enumerate URLs and then maintain a small pool of workers to do the crawling (and apply rate limits if applicable).

I’d say especially the latter option is pretty valid; there are many real-world examples where you cannot neatly plan a supervision from the start because things do change at runtime. DynamicSupervisor is ideal for this.