Representing the world with processes

Hello, everyone! I have been having a little bit of fun with processes and supervisors recently and I was thinking about how to better represent the/a world with processes…

I was imagining the following structure:

  • One supervisor that watches over continents
  • One supervisor per continent, that watches over its countries
  • One supervisor per county, that watches over its states
  • One supervisor per state, that watches over its cities

After my recently acquired knowledge about DynamicSupervisors, I think it’s probably the right way to go for the performance benefits.

Visually, I think it goes something like this:

[WorldSupervisor]          ⟶ (NorthAmericaGenServer)
↳ [NorthAmericaSupervisor] ⟶ (UnitedStatesGenServer)
↳ [UnitedStatesSupervisor] ⟶ (VirginiaGenServer)
↳ [VirginiaSupervisor]     ⟶ (RichmondGenServer)
[WorldSupervisor]          ⟶ (SouthAmericaGenServer)
↳ [SouthAmericaSupervisor] ⟶ (BrazilGenServer)
↳ [BrazilSupervisor]       ⟶ (SaoPauloGenServer)
↳ [SaoPauloSupervisor]     ⟶ (SantosGenServer)

So, let’s imagine we want to start all of this dynamically. The first thing I tried was creating a WorldSupervisor module that implements DynamicSupervisor, to start all of it:

# ids can and will be more descriptive terms and names use a via tuple with a custom Registry
def start_continent(%Continent{id: id} = continent) do
  name = name_for_process(id)
  # starts a supervisor for the continent and starts the server under that supervisor  (?) 
  {:ok, continent_sup} = DynamicSupervisor.start_child(__MODULE__, {__MODULE__, name: name})
  {:ok, _server} = DynamicSupervisor.start_child(continent_sup, {ContinentServer, continet})
end

def start_country(%Country{id: id, continent_id: continent_id} = country) do
  name = name_for_process(id)
  # finds the continent that should supervise this country
  continent_sup = find_continent_supervisor!(continent_id)
  # starts a supervisor for the country and starts the server under that supervisor (?)
  {:ok,  country_sup} = DynamicSupervisor.start_child(continent_sup, name: name)
  {:ok, _server} = DynamicSupervisor.start_child(country_sup, {CountrySever, country})
end

def start_state(%State{id: id, country_id: country_id} = state) do
  name = name_for_process(id)
  # finds the country that should supervise this state
  country_sup = find_country_supervisor!(country_id)
  # starts a supervisor for the state and starts the server under that supervisor (?)
  {:ok,  state_sup} = DynamicSupervisor.start_child(country_sup, name: name)
  {:ok, _server} = DynamicSupervisor.start_child(state_sup, {StateSever, state})
end

def start_city(%City{id: id, state_id: state_id} = city) do
  name = name_for_process(id)
  # finds the state that should supervise this city
  state_sup = find_state_supervisor!(state_id)
  # starts a supervisor for the city and starts the server under that supervisor (?)
  {:ok,  city_sup} = DynamicSupervisor.start_child(state_sup, name: name)
  {:ok, _server} = DynamicSupervisor.start_child(city_sup, {CitySever, city})
end

This is somewhat of a pseudo code. I wrote a version of this before, which didn’t work properly. I believe it was complaining about something related to starting the same process again (I think the supervisors were starting correctly but not the servers).

So, there are some problems I see with this implementation (besides the obvious ones):

  • I’m starting to think that this is getting too complex for my taste. Would it be better to just have a “flat” organization? I mean, having ContinentSupevevisor, StateSupervisor, and CitySupervisor side by side. This way all continents, states, and cities are under just one supervisor. The obvious loss here is on the organization side since everything will be mixed under its own category of supervisor.
  • Given that starting all of this is dynamic, I’m using a custom Registry and I still haven’t found a way to show the proper names instead of its PIDs in :observer, which I think should work!?

So, has anyone tried something similar before? What do you think would be the better approach for a structure like this? Any comments are very welcome. Thanks in advance.

1 Like

Since this is a pretty static and finite list of processes with a predetermined structure, why are you thinking DynamicSupervisor is appropriate? What is to be gained by that decision? My first choice would be a regular Supervisor if I wasn’t actually expecting these trees to change after everything starts up.

For purposes of organization and error isolation, I think it should not have a flat structure. For one thing, you can group a country into its own module World.SouthAmerica.BrazilSupervisor and then see in that module an explicit list of all of its children in one place. Each child has a module and file, and you can see all of their children in one place. If you ever need to add or remove something, you know exactly where to go.

Also in this case if SaoPaolo and RioDeJaneiro and MinasGerais all crash within the max_seconds time, then only Brazil restarts. If you had a flat structure, then the whole world would restart unless you were exceedingly clever with your restart settings. Which design models the world better? I know about the butterfly effect, but does chaos in Brazil cause an instant general strike in Hoboken?

I think using Supervisor and a hierarchical structure also makes the startup process simpler to get right and easier to understand. If you had DynamicSupervisors all the way down, you’d need a script to get everything started up. Using Supervisor you just need to structure your code to model the supervision tree, and it all starts up automatically when you boot the system. Granted, with DynamicSupervisor you could dynamically read static files of place names which would result in less code to write, but it’s a tradeoff of brevity for clarity. And really, you still could read the static files to generate your static children lists if you wanted to.

This doesn’t work, if you name processes with a registry and :via naming then you will only see pids in observer. But again, you have a finite, known list of processes, so why not give them registered atom names? Even for all the cities in the world, you won’t run out of atoms

1 Like

Hi @msimonborg, thanks for your reply… You already gave me a lot of things to think about :slightly_smiling_face:.
So, just to respond to some of your questions and make the intent a little bit more clear…

Precisely what you mentioned here :blush::

Granted, with DynamicSupervisor you could dynamically read static files of place names which would result in less code to write, but it’s a tradeoff of brevity for clarity. And really, you still could read the static files to generate your static children lists if you wanted to.

This was just a very small example, but let’s say we want to represent the whole world: continents, countries, states, cities, and perhaps neighborhoods and more. Although this is a finite list, I think that filling this by hand would be a cumbersome task. For all intents and purposes, let’s assume we will query this from a database and start everything dynamically.

This was my first impression as well, but I had some problems naming the processes. This of course would not happen if all supervisors and servers were declared upfront since we could just use atoms.

I read something about this in the forums but I still wasn’t sure. I read someone talking about using :global, but I tested it and the result was the same. I wonder if there’s a way to solve this.

1 Like

Thanks for the clarifications! I agree that writing it out by hand is the more cumbersome task, especially as you drill down to even finer divisions.

You can still dynamically define a static children list for a regular Supervisor. You could have generic modules such as CountrySupervisor, CitySupervisor, CityServer etc. and compose those child specs and supervision trees dynamically at runtime from your static data. Worth asking though if everything in the world is generic enough that Sao Paolo and Hoboken, NJ can have the same server implementation :slightly_smiling_face:

You can dynamically generate the atom names as long as you completely trust that the data is the finite set you expect it to be.

I can’t recall if global names appear in observer, but either way I think they still require atom names {:global, atom}, and the module is intended for use in distributed systems, so might as well use regular name registration IMO.

All in all, if your static data source (files, database, whichever) has a well defined structure and naming conventions, then I’m of the opinion that there’s no reason you can’t dynamically (at runtime) start up a static supervision tree with the Supervisor module and registered atom names.

1 Like

Yes, you are right, I was even considering this; however - and I just recently discovered this - you can also start children dynamically with a standard Supervisor. So the thing with DynamicSupervisor (as explained to me) is that it’s better suited to deal with dynamic children (there’s some explanation about it in the linked topic).

I think this is along the lines of what I was thinking because servers are supposed to be just a bag of data with a specific behavior corresponding to its type (continents, countries, states, and cities).

BTW, do you have a general idea on how to implement it? Do you think the examples I provided are in the right direction? Any considerations are very welcome :blush:.

Yes, agree, but this would still be kind of limiting for a real use case. I mean, realistically, how many neighborhoods do we have in the world!? It would be cool if there were a way to retain process names for practical purposes (or perhaps visualization is not that important, idk :sweat_smile:).

It depends - IMO the key to understanding where to insert additional layers of supervisors is “what should happen when one of these processes crashes?”

3 Likes

I think perhaps I haven’t explained my idea well :sweat_smile: I don’t mean using Supervisor.start_child/2 in the way you would with DynamicSupervisor, but instead to dynamically generate a static list of children at runtime when the app starts, and pass that list to Supervisor.start_link. The use case of DynamicSupervisor is for dynamically starting children that can’t be defined in advance in a static list, but are truly dynamic at runtime and may start in response to unknown events like user input or other events in the system, like a game server in a multiplayer game, or a chat room, or a LiveView process.

Imagine a naive example where you have data in a directory structure of static files like this

world 
  |-- united_states
  |  |-- new_jersey
  |  `-- texas
  `-- brazil
     |-- sao_paolo
     `-- bahia

You could dynamically read this static data at runtime and start pre-determined tree structures with Supervisor

defmodule World.Application do
  use Application

  def start(_type, _args) do
    world_path = Path.relative("world")
    countries = File.ls!(world_path)

    children =
      Enum.flat_map(countries, fn country ->
        country_path = Path.join([world_path, country])
        [{World.CountryServer, country_path}, {World.CountrySupervisor, country_path}]
      end)

    opts = [strategy: :one_for_one, name: World.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

defmodule World.CountrySupervisor do
  use Supervisor

  def child_spec(path) do
    basename =
      path
      |> Path.basename()
      |> String.split("_")
      |> Enum.map(&String.capitalize/1)
      |> Enum.join()

    name = Module.concat([World, basename, Supervisor])

    [name: name, path: path]
    |> super()
    |> Supervisor.child_spec(id: name)
  end

  def start_link(opts) do
    Supervisor.start_link(__MODULE__, opts[:path], name: opts[:name])
  end

  def init(path) do
    states = File.ls!(path)

    children =
      Enum.flat_map(states, fn state ->
        state_path = Path.join([path, state])
        [{World.StateServer, state_path}, {World.StateSupervisor, state_path}]
      end)

    Supervisor.init(children, strategy: :one_for_one)
  end
end

If you really want to drill down to neighborhoods, that would be tough to answer and tougher to gather all the data! But your default atom limit is about 1 million, and you can configure it to be higher than that. pids in a registry will still take up memory like the atoms in the atom table. Also worth stating that you only need to register names at all if you need processes to discover and communicate with each other. Even then it’s not always necessary if the scope of communication networks is small and isolated.

2 Likes

Processes are supposed to be failure domains. I’m not sure why these regions need to be separate failure domains in the first place, but assuming you know what you are talking about: The correct structure is to put everything in a single supervisor (not dynamic unless you have a reason for a region to be offline). There is no reason for say the rio failure domain to be preferentially taken down with the sao Paolo failure domain vs say a Berlin failure domain.

Maybe instead of a process tree use a map?

3 Likes

anyone from germany can relate. Supervisor please restart! :wink:

3 Likes

Hey, @msimonborg, I’m sorry I was not clear in my previous reply and I gave the impression that I overlooked your suggestion. What I meant by that comment is that I discovered that DynamicSupervisor has (apparently) some optimizations other than just starting children dynamically (look at the new docs here and here to see why this was on my head) and thus I thought it would be better suited for this use-case. However, after looking at your last reply, I see that I could rely on the supervision ordering for some guarantees. For instance, instead of starting always a supervisor + server under it, I could start all supervisors and servers from one kind (eg: continents) and then all supervisors and servers from another kind (eg: countries), this way, if I need to lookup the continent server from a country server, the continent server process is guaranteed to already be started. Thanks for again for the suggestion and the useful examples :fist_right::fist_left:.

That’s a very good point! My main concern however was to have a better visualization of what is what in the supervision tree. I’m thinking that perhaps this could be solved by simply implementing a LiveView that receives a heartbeat from the processes or something like that.

In this case, what do you mean by “put everything in a single supervisor”? From what I understand, the idea of having multiple supervisors would be to have different “branches” organized under the same tree and, as @msimonborg suggested, when one “location” crashes only the one would restart.

I’m not sure if you are suggesting something too different from what we already talked about, but if so, could you elaborate a bit more?

PS.:

not dynamic unless you have a reason for a region to be offline

Thinking about it in terms of a “simulation game” like Risk, Monopoly, or Catan (but at a global scale :sweat_smile:), it crossed my mind to make this possible. This way we could shut down a “location” if there’s nothing important happening there and bring it back up when there is.

Their point is that supervision trees are not for code organisation and it seems you’re approaching this from the code organisation side of things. You haven’t talked much about what the processes do or how they relate to the others.

https://www.theerlangelist.com/article/spawn_or_not

Hi, @cmo thanks for your reply! I don’t mean to be pedantic here, but I don’t think we are using the word “organization” in the same way. When I think “organization” in this case, I’m thinking loosely about structure, hierarchy, the dispositions of the actors and how they may depend on each other.

Perhaps something closer to this quote about supervisors in the Elixir in Action book:

This mechanism plays an important role in so-called supervision trees, where supervisors and workers are organized in a deeper hierarchy that allows you to control how the system recovers from errors.


I actually mentioned before that processes could be represented as a bag of data about the location (aka: manage some trivial state). What’s important is that the premise is: “what if we wanted to represent the world as processes”. In the end, it isn’t really relevant what they are going to be used for, as long as we can model the hierarchy (continents → countries → states → cities → ???).

:point_up: I dunno if this is helpful or not, but for the sake of creating a shared mental image… Imagine modeling an advanced simulation game - a mix of Risk and Catan if you will. Like in real life (of course), each actor produces a steady amount of resources over time, that can be shared, traded, conquered, and retrieved. So, the aforementioned actors should have some sort of communication between them and be hierarchically dependent in some way; like having to pay taxes to its parent and seize to exist if they are not in-game anymore.

Anyway, now I’m curious about what’s exactly @ityonemo suggestion is so I can understand it better since it’s a different take on the general approach I was thinking about so far.

You don’t need to use a specially crafted supervisor tree to do that; in fact, the tree shape would be a handicap instead of a help, because you want the finest granularity you can get: If there are only one or two countries in Africa that have actions, you don’t want to keep all the other countries in Africa in the idle loop, right?

Hey @derek-zhou! I just updated my previous reply to have a better illustration while you were typing. See if it changes anything about your suggestion. If not, could you illustrate what you think would be a better “organization” for this supervision structure? Cheers! :blush::call_me_hand:

I would use a flat supervision tree and encode the geographical relationship among all the entities in my own data structure.

3 Likes

this is the way.

Anyways, you don’t need to even delete your genservers, just put them into hibernation and they won’t be woken up until they receive a message.

You are imposing a hierarchical relationship based off of geography, which probably has nothing to do with actual dynamics how compute is going to be utilized in of your “business case”, if I’m understanding what you are doing.

2 Likes

Well in that case, if there is no requirement about what the processes can do or how they should be restarted then I guess the answer would be spawn/1.

As other pointed out, this is probably suboptimal exercise. It’s similar to the exercises from Object-Oriented world where you try to map a car into an object Car that inherits from Vehicle that embeds four Wheels and has one Engine that could be DieselEngine or ElectricEngine etc. etc.

In reality you do not build objects like that and you do not build supervision trees like that.

Generally speaking, I think the process should be thought of as a wrapper around a runtime thread, an execution path, rather than an entity, real life or otherwise.

1 Like

So, correct me if I’m wrong, is this is what you are talking about?

[ContinentSupervisor] ⟶ (NorthAmericaServer), (SouthAmericaServer)
[CountrySupervisor]   ⟶ (UnitedStatesServer), (BrazilServer)
[CitySupervisor]      ⟶ (VirginiaSupervisor), (SaoPauloSupervisor)

And when you talk about “encoding the relationship in the data structure”, do you think of something in particular or would you use something like I did in my examples (eg: referencing ids)?

If that’s the case I’d pose the same question as before: aren’t you worried that a flat structure would be too chaotic to monitor? And if not, would you use something else other than :observer to monitor it?

Yes, that’s a general idea.

Correctly, this is the tricky part of the question.

This is fine, my end goal is not to construct an “optimal exercise”. I’m already having such a great time picking your guy’s brains because all these different opinions are very enriching. BTW are you familiar with Cunningham’s Law (something similar :sweat_smile:)?

No, I would just use one Supervisor to supervise everything. I don’t even want to impose a rigid 3 level structure. There could be 2 level leaf entities such as Singapore, or another layer at the province level.

The data structure can be as simple as a map of name → pid in each non-leaf entity.

1 Like