Proposing Registry

Yeah, I thought about comparing with Syn as well, though I agree with @josevalim that a direct comparison is not really appropriate. But just to satisfy our curiosity:

Bottom line: on a standalone node, Syn is always a bit slower than Gproc, but at high concurrency it is faster than the standard Erlang process registry (as well as Registry without partitions).

Registry has been updated to v0.3.0 with:

  1. Improved performance for 1 partition (most common case)
  2. Allow a process to update its own value
  3. Support for named listeners
  4. Allow metadata to be stored along-side the registry

The last two features should make the registry more extensible. Listeners allows you to give a named process when the registry is started that will receive events whenever a process registers or unregisters a key. You could use those features to add Await support to the registry or even use the registry as a pool. We have added an example of using the registry as a sojourn pool to the examples directory. The pool is an example, do not use it in production:

https://github.com/elixir-lang/registry/blob/master/examples/sojourn.exs

The pool above measures the sojourn time, which is how long messages stay in each pooled process queue. That provides an idea of how fast processes are responding. Whenever there is a dispatch, we pick two random pooled processes and choose the one with the most recent reply and smaller sojourn time.

The pool itself works by starting a worker and a registry. The worker is a listener of the registry and a supervisor guarantees that both are restarted in case any of crashes. Whenever a new process is added to the registry, the worker is notified and starts sending sampling messages to that process which updates its own entry in case of crashes.

I hope it serves as inspiration for playing with the registry for other use cases.

3 Likes

Definitely like the changes. :slight_smile:

Btw, @voltone has re-ran the benchmarks, including more partitions and parameters. For those interested, please check out the updated spreadsheet:

Thank you @voltone!

2 Likes

Looks like the optimal performance is achieved when having a partition per core. Is this correct?

Yes but it doesn’t mean having a partition per core is necessarily the way to go. If you don’t expect an intensive workload, using 4 or 8 partitions for 40 cores would be fine.

Being the most pessimistic, I would do a factor of 0.25 on the number of cores. So for 40 cores, I would do 10 partitions. Like this: System.schedulers_online |> div(4) |> max(1).

1 Like

The Registry has been merged into Elixir master.

10 Likes

Hey José,

I was taking a look at the code and was puzzled over two things:

  • First one is, why use a positional argument for the registry name, rather than using the convention of passing name in the keyword list options.
  • The second one is the reasoning for defining multiple modules inside the same file? I was trying to locate Registry.Supervisor and it took me a while to understand it was in the file rather then in a Registry folder.

Is there a convention in place for cases like this?

Because the name is not optional on this case.

No conventions as far as I know.

Does anyone know where can I find more info on the OTP Team’s plans on a distributed registry?

Perhaps in the documentation of the global module :wink:

Doesn’t global predate the Elixir Registry and this whole thread by years? I think that @josevalim was referring to something else. Still, I couldn’t see reference to any new work in progress in the global docs.

The project was briefly mentioned at the conference in Stockholm last year:

It’s the distributed hash table mentioned as an alternative to Global on the last slide of “News from the OTP team.

There was also this tidbit:

http://www.erlang-factory.com/sfbay2017/christopher-meiklejohn.html

Recently, attempts have been made to resolve the problems of Distributed Erlang: SD-Erlang and Ericsson’s planned port of global to the Kademlia DHT are focused on reducing the amount of information that each node in the global registry needs to replicate, but that only temporarily mitigates the problem.

That’s about all I’ve heard personally.

3 Likes

There is a talk about the DHT plans here:

2 Likes

Thank you both, that was very helpful!