ProcessHub - process distribution library

Just FYI, ProcessHub is used in NervesHub which means it is used for some reasonably large deployments. Several 100K devices connected to double digit nodes in some installs.

Don’t think we’ve had any issues. I haven’t worked on that part so much but seems like a clean solution.

We use it for a fairly simple thing. Important but not intense. Make sure we run one orchestrator process for distribution of firmware updates per deployment group im the cluster. The orchestrator keeps tabs on the number of concurrent updates and stuff. Tells other processes what to do.

Though that might be of interest

I started exploring this library a few hours ago and it is extremely good. I have used horde in the past and pogo more recently. Both of those libs are great but process_hub is truly a joy to use.

There is a lot to love and I have not even started using the hot-swapping and hooks stuff. Reading the list of hook events blew my mind. My imagination was piqued.

I love the simplicity of the hub_id atom. This plays well with Fly because the fly region can be used to build the hub_id, for simple per-region clustering.

I also love the way you use local ETS tables for fast lookup. Perfect use-case for ets.

I am still doing a refactor on a feature branch to see if my own hacky :pogo-based attempts at distributed process management can be replaced with :process_hub. So far I refactored a cache for oauth “state” and a per-kitchen “global singleton” order number generator process. I have way less LOC already and it is far simpler to read and understand.

I am in awe. Congratulations.

New version release 0.5.0-beta

This one includes many performance optimisations, bugfixes, new features, some breaking changes and large scale code refactoring.
You can look at the changelog here: Release v0.5.0-beta · alfetahe/process-hub · GitHub

I have a lot of ideas and features I’d like to add to this library, but I also want to prioritize stability. In this release, I’ve marked several new features as experimental. This means they may undergo breaking changes or be dropped entirely in future updates.

Here are some performance benchmarks comparing this version and the previous one starting and stopping 10k processes on 10 nodes (running on the same machine).
I used the same laptop to run the tests.
The new version is up to ~10x faster then the previous one and uses ~2x less memory.
This is ~250ms to start and stop 10k processes on 10 nodes.

Host:
Operating System: Linux
CPU Information: 13th Gen Intel(R) Core™ i7-13700H
Number of Available Cores: 20
Available memory: 62.43 GB
Elixir 1.19.3
Erlang 28.1.1
JIT enabled: true

v0.5.0-beta

Name                             ips        average  deviation         median         99th %
start_&_stop_processes          4.28      233.64 ms   ±129.04%       57.73 ms      827.01 ms

Memory usage statistics:

Name                           average  deviation         median         99th %
start_&_stop_processes       449.77 KB    ±77.20%      380.79 KB     1030.14 KB

v0.4.1-beta

Name                             ips        average  deviation         median         99th %
start_&_stop_processes          0.48         2.07 s     ±0.63%         2.08 s         2.08 s

Memory usage statistics:

Name                      Memory usage
start_&_stop_processes       879.66 KB