ProcessHub - process distribution library

New version released 0.7.0

The license has changed from GPL-3.0 to Apache-2.0 to make adoption easier.

Alost includes a bug fix where a race that lost the result of awaitable: true operations, reported and fixed by @slouchpie thank you!

Recovery (still experimental) got a big rework. Children started with durable: true go into a cluster wide list that every node keeps the cluster in line with, so a full outage recovers on its own while a single node rejoining won’t restart what’s already running. A custom storage options added.

Link to the changelog: Release v0.7.0 · alfetahe/process-hub · GitHub

New version released 0.6.0 yes, the beta tag is officially gone!

The library has been in use for some time now and has been stable. While I still have many ideas for new features, the core functionality and public API will likely stay as they are unless there is a clear need to change them.

In the future, I will keep releasing new features, and some will be marked as experimental just as I’ve done so far. If these features stand the test of time and prove useful, they will become part of the stable core.

Here’s the changelog:

## v0.6.0 - 2026-07-11
This release is aimed at making the system more resilient: a pluggable registry backend (ETS stays the default, with on-disk and hybrid options today and room for further storage integrations), safe migration — processes can defer their own migration and a node can be drained before shutdown — and experimental recovery mechanisms. It also includes a few bug fixes.

### Added
- Pluggable registry storage backend: new `:registry_backend` field on `%ProcessHub{}` (default `:ets`). Accepts `:ets`, `{:dets, opts}`, `{:durable_ets, opts}`, or `{Module, opts}` for a custom backend implementing `ProcessHub.Service.Storage.Behaviour`.
  - `:ets` — default in-memory backend (unchanged behaviour).
  - `{:dets, opts}` — on-disk persistence: sync-after-write durability, `repair: true` on open, and corruption rotation.
  - `{:durable_ets, opts}` — hybrid: in-memory reads with synchronous DETS mirroring for restart survival. Shares the DETS on-disk format, so a hub can switch between `:dets` and `:durable_ets` against the same `:path` and keep its rows.
- Opt-in migration consent: `:consent_settings` on `HotSwap`/`ColdSwap` (default `nil` — disabled, no overhead). A process that does `use ProcessHub.Strategy.Migration.MigrationConsent` is asked before it is migrated; `:defer` (or no reply within `:consent_timeout`) parks it in a deferred list, retried every `:retry_interval` and force-migrated after `:max_defer_time`. `ProcessHub.Service.Migration.migration_ready/2` signals readiness early. Primary instances only; processes started through wrapper modules are not detected.
- `ProcessHub.Service.Migration.drain/2` — blocking graceful node drain before shutdown. Removes the node from the distribution cluster-wide, migrates all local children away through the consent gate, and force-migrates whatever is still deferred at the `:timeout` deadline (default `60_000` ms). Returns `{:ok, %{migrated: n, forced: m}}`, or an error without touching children when no other node exists or the hub is partitioned/locked. Not reversible — restart the hub to rejoin.
- New hooks: `migration_deferred` (child ids parked) and `drain_completed` (drain summary).
- New guide `guides/Persistence.md` covering the persistence backends and the recovery lifecycle / operator runbook.
- **Experimental:** opt-in, marker-gated boot recovery via the single `:auto_recovery` field on `%ProcessHub{}` (default `false`). Accepts `false`, `true`, or a keyword list of `:marker_path` (operator override for the marker file location) and `:recovery_timeout_ms` (safety ceiling on the `:recovering` state, default `30_000`). A successful boot writes the marker file; on next boot the node skips replay if the marker is present, or rebuilds from disk if it is absent (state is `:recovering` → `:normal`). Disk replay restores only child specs — stale pids and metadata are dropped (fixes a restarted node pushing stale registry data into a healthy cluster).
  - API on `ProcessHub.Service.Recovery`: `recovery_state/1`, `await_normal/2`, `prepare_recovery/1`, and `prepare_recovery_cluster/1` (all safe to call on non-opted-in hubs).
  - New hooks: `recovery_state_changed`, `pre_recovery_replay` (**synchronous** — the coordinator awaits each handler), and `post_recovery_replay`.
- **Experimental:** `:nodeup_reconcile_interval` field on `%ProcessHub{}` (default `3000` ms; `0` disables) — fail-safe that merges a peer if the `pg` cluster-join notification is missed on a reconnection, closing a split-brain gap.

### Fixed
- Fast-restart stale bindings are now reaped on every backend (including default `:ets`). A node that restarts within `:net_ticktime` (no `:nodedown`) left peers pointing at its dead pids, since the rejoin sync only appends. Each node now broadcasts a per-boot token; peers purge a node's bindings only when its token changes (distinguishing a restart from a network flap). This replaces the previous purge signal that was gated behind `:auto_recovery`.
- Registry sync (`Synchronizer.append_data/2`) no longer overwrites a known child's local child-spec and metadata with a peer's copy on a pid change — it only updates the pid map. A node replaying stale durable state could otherwise clobber a peer's current spec, leaving a degraded process after redistribution.
- `mix docs` now builds with zero warnings: dropped the auto-generated `:supervisor` delegate docs, corrected doc references to the `init_opts`/`stop_opts` types and the `ProcessHub.Service.Storage.Behaviour` module, and removed a reference to a non-existent DETS compaction function.
- Janitor TTL cleanup no longer wipes a re-registered process. `purge_pending_registry/1` scanned the registry with a dirty `:ets.match` read and then deleted unconditionally; an incoming registration that re-populated a TTL tombstone between the scan and the delete could be erased. Cleanup now goes through `ProcessRegistry.delete_if_expired/2`, which re-validates expiry inside the registry process and skips the delete if the entry was re-populated or re-leased.
- Registry no longer keeps dead pids after quorum recovery. On quorum failure the distributed supervisor is terminated and its children are killed, but their registry entries are retained as the restart manifest. On recovery (`State.toggle_quorum_success/1`) the supervisor restarts the children with fresh pids that were only tracked internally, leaving the registry pointing at the dead pids — which were then propagated to the rest of the cluster on the next sync. Recovery now refreshes the local node's pids in the registry to the actually-running ones.

New version release 0.5.0-beta

This one includes many performance optimisations, bugfixes, new features, some breaking changes and large scale code refactoring.
You can look at the changelog here: Release v0.5.0-beta · alfetahe/process-hub · GitHub

I have a lot of ideas and features I’d like to add to this library, but I also want to prioritize stability. In this release, I’ve marked several new features as experimental. This means they may undergo breaking changes or be dropped entirely in future updates.

Here are some performance benchmarks comparing this version and the previous one starting and stopping 10k processes on 10 nodes (running on the same machine).
I used the same laptop to run the tests.
The new version is up to ~10x faster then the previous one and uses ~2x less memory.
This is ~250ms to start and stop 10k processes on 10 nodes.

Host:
Operating System: Linux
CPU Information: 13th Gen Intel(R) Core™ i7-13700H
Number of Available Cores: 20
Available memory: 62.43 GB
Elixir 1.19.3
Erlang 28.1.1
JIT enabled: true

v0.5.0-beta

Name                             ips        average  deviation         median         99th %
start_&_stop_processes          4.28      233.64 ms   ±129.04%       57.73 ms      827.01 ms

Memory usage statistics:

Name                           average  deviation         median         99th %
start_&_stop_processes       449.77 KB    ±77.20%      380.79 KB     1030.14 KB

v0.4.1-beta

Name                             ips        average  deviation         median         99th %
start_&_stop_processes          0.48         2.07 s     ±0.63%         2.08 s         2.08 s

Memory usage statistics:

Name                      Memory usage
start_&_stop_processes       879.66 KB

Sorry for the late reply.

I don’t know your exact requirements but in general you should be able to do it.
ProcessHub takes care of restarting the children on the new node automatically.

In case you need to handover the process states before terminating them on the old node you can use either `ProcessHub.Strategy.Migration.HotSwap` or `ProcessHub.Strategy.Migration.ColdSwap` migration strategies with `handover` option. In this case you need to do graceful shutdown on the nodes so they have enough time to pass the states to the new node. You can use `System.stop()`.

Just FYI, ProcessHub is used in NervesHub which means it is used for some reasonably large deployments. Several 100K devices connected to double digit nodes in some installs.

Don’t think we’ve had any issues. I haven’t worked on that part so much but seems like a clean solution.

We use it for a fairly simple thing. Important but not intense. Make sure we run one orchestrator process for distribution of firmware updates per deployment group im the cluster. The orchestrator keeps tabs on the number of concurrent updates and stuff. Tells other processes what to do.

Though that might be of interest

This is a nice library. I am looking to see if it will work well with rolling deploys in AWS ECS.

New version release 0.4.0-beta

This new release delivers bug fixes, improvements, and new features. The highlight is the introduction of enhanced formatting options with Future.await, providing more powerful result handling capabilities:
Starting and stopping processes — ProcessHub v0.5.0-beta .

Additionally, this release includes some deprecations and introduces an innovative distribution strategy that intelligently distributes processes based on real-time node load:
ProcessHub.Strategy.Distribution.CentralizedLoadBalancer — ProcessHub v0.5.0-beta . Please note that this load balancing feature is currently experimental, with further enhancements planned for the next release.

## v0.4.0-beta - 2025-09-29
This release introduces improvements to ProcessHub's API design and internal architecture. The major focus is on enhancing the developer experience with new promise-based return types and improving system performance with experimental load balancing capabilities.

This release maintains backward compatibility with the main ProcessHub module while introducing new patterns that will become the standard in future versions.

### Added
- `ProcessHub.StartResult` module which can be used to format and manipulate the results of process startups. This struct is returned by the `ProcessHub.Future.await/1` function.
- `ProcessHub.StopResult` module which can be used to format and manipulate the results of process shutdowns. This struct is returned by the `ProcessHub.Future.await/1` function.

### Fixed
- Nested child processes were returned using `ProcessHub.Strategy.Redundancy.Replication` strategy. The result is now properly formatted.
- Using the `async_wait` option on process startup or shutdown now returns a promise that can be awaited instead of a function. This avoids polluting the caller's mailbox if the caller does not await the result. It also prevents the caller from receiving messages from their mailbox before executing the await function.
The new `:await_timeout` option specifies a timeout for the spawned collector process to wait before automatically terminating itself.
- Gossip synchronization was sending empty messages during cluster updates.
- Race condition when multiple nodes were started simultaneously and the cluster information was not seen the same by all nodes.
- `ProcessHub.Strategy.PartitionTolerance.StaticQuorum` with `startup_confirm: true` shuts down the distributed supervisor when in partition mode.
When new nodes joining the cluster this leads to errors when trying to call the dead distributed supervisor process.

### Changed
- **Internal messaging system redesigned**: ProcessHub no longer converts `hub_id` atoms to other process-specific atoms internally. Instead, it uses Registry with tuples for message dispatching. This change:
  - Eliminates the overhead of atom conversion
  - Prevents dangerous dynamic atom generation
  - Reduces the number of dynamic atoms created per `hub_id`
  - **Breaking change**: Users directly accessing ProcessHub service modules (instead of the main `ProcessHub` module) may experience issues. Users who only use the main `ProcessHub` module are unaffected.
- All public API functions defined in the `ProcessHub` module now call the coordinator process instead of calling the services directly.
This avoids the potential issue of generating new atoms when calling those functions with a `hub_id` that is not known to the system.
- Calling `ProcessHub.start_child/3`, `ProcessHub.start_children/3`, `ProcessHub.stop_child/3`, `ProcessHub.stop_child/3` with the `awaitable: true` option (or deprecated `async_wait: true`) now returns a `ProcessHub.Future.t()` struct that can be awaited instead of regular anonymous function. Calling `ProcessHub.Future.await/1` will either return `ProcessHub.StartResult.t()` or `ProcessHub.StopResult.t()` instead of tuples.

### Deprecated
- The option `:async_wait` is deprecated in favor of `:awaitable`. This option will be removed in the 0.5.x version.
- `ProcessHub.await/1` is deprecated in favor of `ProcessHub.Future.await/1`. The `ProcessHub.await/1` function will be removed in the 0.5.x version or replaced with the new implementation.

### Soft deprecations
- `ProcessHub.which_children/2` is soft deprecated. Use `ProcessHub.process_list/2` instead.

### Breaking changes
- If you are defining your own custom distribution strategy, you must update the `belongs_to/4` to accepts a **list** of child ids instead of a single child id. This allows the strategy to optimize the node selection for multiple children at once.

I’m considering adding a more dynamic approach to process distribution.

Currently, the main strategy is based on consistent hashing, meaning all nodes agree on the distribution without requiring a master or coordinator node. This works well, but it’s currently not possible (without using workarounds) to alter the distribution dynamically. We do have the guided distribution strategy, but it lacks features like automatic migration and other flexibility.

The idea is to create a new distribution strategy—ideally built on top of guided distribution—that allows users to define their own callbacks for selecting preferred nodes, instead of relying solely on automatic assignment via a hash ring. This approach would likely require one central or master node within the cluster.

My personal use case for this would be to distribute processes based on BEAM or system load, allowing processes to be placed on nodes with lower current load, for example.

If anyone has ideas or relevant experience, I’d love to hear your thoughts!

New version release 0.3.3-alpha

## v0.3.3-alpha - 2025-06-21
This release focuses on improving process migrations and the state handover mechanism.
Users who previously implemented custom callbacks for the `HotSwap` migration strategy must update their code to use the new macro-based approach.

This version also includes bug fixes, soft deprecations, documentation improvements, and new features.
Notably, it introduces the ability to start child processes with attached metadata that is synchronized across nodes.
Users can now query the process registry by tag and dump the entire process registry.

### Fixed
- Using the `HotSwap` migration strategy with graceful shutdown caused migration messages to be sent to a `nil` node when no other nodes were available. This has been fixed; no migration messages are sent if no other nodes are present.
- The `HotSwap` shutdown migration could cause a timeout error on the shutting-down node if the target node for takeover became unavailable. This has been resolved by sending an asynchronous message (`GenServer.cast/2`) instead of a synchronous one, avoiding process spawning on the receiving node and ensuring it receives data before starting the new children.
- The hooks cheat sheet guide was pointing to incorrect hook keys in some cases.

### Breaking changes
- If you are using `ProcessHub.Strategy.Migration.HotSwap` with state handover and have implemented your own custom callbacks instead of using the provided macro, you must add `use ProcessHub.Strategy.Migration.HotSwap` to your module and override the `prepare_handover_state/1` and `alter_handover_state/2` functions to modify the state before it is set on the new process. Remove the existing implementations as they are no longer needed and are now provided by the macro.

### Added
- Ability to start child processes with attached metadata.
- Child processes can now be started with additional metadata that is stored in the process registry and synchronized across nodes.
- `ProcessHub.child_lookup/3` now accepts an `opts` parameter to allow returning metadata.
- `ProcessHub.registry_dump/1` dumps the entire process registry. This will eventually supersede `ProcessHub.process_registry/1`.
- `ProcessHub.tag_query/2` allows querying the process registry by tag.
- `ProcessRegistry.update/3` enables advanced users to manually update the process registry.
- Alter hooks to modify data before processing. Currently, only one alter hook is available: `child_data_alter_hook`, which is invoked right before the supervisor starts the child process. This allows altering the child spec, metadata, or node list on the node where the process will be started.
- New option `:confirm_handover` for `ProcessHub.Strategy.Migration.HotSwap`. This option allows confirming state handovers by waiting for synchronization messages from the target node. It is particularly useful for ensuring that the state has been properly handed over before proceeding with further operations, especially during tests.

### Changed
- Calling `ProcessHub.Service.Synchronizer.exec_interval_sync/4` has been made synchronous to avoid possible race conditions.
- Hooks registered by `ProcessHub` now have a default priority of `100`.

### Soft Deprecations
- `ProcessHub.process_registry/1` will be deprecated in favor of `ProcessHub.registry_dump/1` as it does not return associated metadata with the processes.

New version released 0.3.2-alpha

## v0.3.2-alpha - 2025-04-16
Includes new feature and a breaking change. Minor improvements on documentations and other fixes.

### Breaking changes
- The process state handover has been unified for different types of migrations. Users who previously used HotSwap migration with process state handover and implemented their own custom callbacks instead of using the provided macro, will need to update their callback functions. See `ProcessHub.Strategy.Migration.HotSwap` for examples. This also removes the need to have a separate callback to handle `{:process_hub, :get_state, cid, from}` graceful shutdown migrations.

### Fixed
- Improved error handling. Single child start/stop operation error result no longer returns list.
- Fix: Prevent mix tasks from being included in production build. [https://github.com/alfetahe/process-hub/issues/10]
- Documentation typespecs and some comments.
- Minor code refactors.

### Added
- Adjusted `HookManager` documentation to allow `String.t()` as hook handler keys.
- Added new function `ProcessHub.promote_to_node/2` to handle updating the process registry after turning node node into distributed node. Useful when the node has been started as none distributed and later made into one by calling `Node.start/3` function.