ProcessHub - process distribution library

Your Libraries & OS Mentoring Libraries
, ,
21

Good catch! Thanks for the pr!
First pull request for the project :tada:

2 Likes
22

Released v0.2.5-alpha Release v0.2.5-alpha Ā· alfetahe/process-hub Ā· GitHub

This release includes bug fixes and documentation improvements.

2 Likes
23

this is a bit naive question, but how does this compare to using the built-in pg module for keeping track of processes in a cluster and doing service discovery that way?

3 Likes
24

This is a very good question! My initial plan was to use the pg module to track the registered processes.

Using pg would solve some of the issues, the biggest probably being keeping track of registered processes across the cluster. However, I would still need to figure out the underlying mechanisms to distribute processes, react to cluster changes, migrate processes to keep the load balanced, handle process state handovers, manage process replications, deal with network splits, and more.

My initial requirements kept growing, and I realized I needed all of that. At this stage, using pg to track processes was not an option anymore because I needed a way to store much more information regarding the registered processes. I could have used pg along with my custom ETS tables, but I decided to store all the data in the ETS tables and synchronize the table myself.

In fact, ProcessHub relies on another library I built which is based on pg. This same mechanism is used to synchronize the ETS tables between the nodes. Basically, I use the pg module to emit process registration/unregistration events to keep the ETS tables in sync. I use the same library for other pub/sub related jobs as well.

I would recommend others to use the pg module when starting out if the distribution requirements are not that strict because it is a really simple and reliable option. Once the distribution requirements become more stringent, there are other mature libraries that can help, and ProcessHub aims to be one in the future.

A small fact: I wasnā€™t initially planning to build such a library. I am actually writing software that needs this functionality, and at some point, I decided to extract the code into separate libraries. Thatā€™s how ProcessHub was born. :slight_smile:

1 Like
25

Version 0.2.6-alpha released

This version includes bug fixes, new API functions, and minor improvements to the documentation.

Changed

  • Replaced Cachex with our custom implementation to enhance performance.
  • Updated the default values for max_restarts and max_seconds to 100 and 4, respectively.
  • Storage module now accepts the table identifier as the first parameter, allowing it to be used with multiple tables.

Added

  • Introduced new API functions get_pids/2 and get_pid/2 to get the pid/s by child_id.
  • New guide page for interacting with the process registry.

Fixed

  • Corrected an issue where local supervisor restarts were not properly updating the global registry.
  • Fixed various typos and errors in the documentation.
1 Like
26

Thanks for another update!

In my current project I use :syn (GitHub - ostinelli/syn: A scalable global Process Registry and Process Group manager for Erlang and Elixir.) for process registry for a pub/sub mechanism and :libcluster (GitHub - bitwalker/libcluster: Automatic cluster formation/healing for Elixir applications) as as an automated clustering mechanism as well as ProcessHub for process distribution and control. It does seem like there is overlap (there is!) but each does a good job for what i use it for i.e. i am using selected bits of each one as they all have there strengths and weaknesses.

I was thinking is it worth making the process tracking method selectable so you can use/make different implementations if you wish? i.e. yours, pg, syn, custom etc as each has trade offs.

My project is just a hobby project but its intended to make a changing cluster of machines look like a single machine with a single memory space and have the system behave as if its a single machine with automated handling of nodes joining and leaving the cluster and auto migration of processes to keep the system complete and running.

Its a play thing and speed and memory constraints are not of any concern and i intend to use a lot of processes like objects (similar to GitHub - wojtekmach/oop: OOP in Elixir!) but with all the genserver stuff hidden behind a macro DSL. You could then submit a program to the cluster without knowing or caring about where it is running.

Well enough rambling (sorry!), the joys of retirement is you can play with all these interesting techs!!

Best wishes

Shifters

4 Likes
27

I was thinking is it worth making the process tracking method selectable so you can use/make different implementations if you wish? i.e. yours, pg, syn, custom etc as each has trade offs.

While I wouldnā€™t rule out the possibility of making it configurable in the future, given that the project is still evolving, the process tracking and process registry are currently very coupled with other sections of the code, making it difficult to make configurable at this moment.

Well enough rambling (sorry!), the joys of retirement is you can play with all these interesting techs!!

I wish I can get there myself too :slight_smile:

1 Like
28

Good evening, version 0.2.9-alpha out! :slight_smile:

Includes some bugfixes related with binary child_ids.

3 Likes
29

Hello,

Any idea when your library will be ready for production use ? Any other library you would suggest in the meantime ?

30

Can you open-source this? Iā€™d love to take a look!

31

Hi, I plan to put the library through a testing phase in a real system within 6 months, but there are no guarantees.

The Horde library is very popular, and you might find it useful. Another option I recommend is building your own solution using Erlangā€™s :pg module.

32

Well Iā€™m kind of a beginner in elixir, I come from nodeJs + I donā€™t know erlang + I have 6 months ahead of me to release a mvp.
I donā€™t think Iā€™ll have enough time in that timeframe to learn Erlang and develop my own distributed solutionā€¦

Thatā€™s why Iā€™m looking for a better solution. I could use Horde but read that some people had a lot of problems with it. May be the easy way out for now is just to save data to a DB or use something like redis ?

33

I can when (if!) its finished/usable - i have restarted 3 times over 5 years, changing the tooling each time.

I have been taking a break from it recently (its more a winter project) so not much to actually share at the moment.

The DSL is very much a ā€˜in my headā€™ design rather than something i have coded up yet! Still reading the Metaprogramming elixir book and looking at other coding examples.

Itā€™s very much a play thing so has no schedule for development etc!

Shifters

34

I donā€™t know your specific requirements, which makes it hard to suggest anything.

Horde has been around for a while and is widely used, so I believe it performs well for its intended purpose.

You can also try this library, though it hasnā€™t been battle-tested yet. Itā€™s still in development, but reported bugs will be addressed.