Hello Elixir Community,

I am working on an Elixir application where I need to introduce a process pooling mechanism (thinking of using :poolboy). The goal is to create a pool of worker processes that can dynamically handle different types of payloads( such as company IDs or other data). Here’s a brief overview of my current setup and the changes I plan to make:

Current Setup:

1. Application Module: The application has several GenServers and one supervisor with all the genservers as direct childrens with a :one_for_one strategy.
2. GenServer Process: Each GenServer has a self-calling mechanism to process a list of companies periodically with the help of delay and what we call :tick as a message. These genservers do thing like fetching data for these companies, tracking internal progress of what work has been done, state of work being done, submitting work and so on.

The way it currently works is that there is a process CompanyDataManager who has the responsibility of maintaining a list of companies which it periodically refreshes from an API. On arrival of a new company or change in it’s metadata, it does chores like creating a new schema, creating new required entities in existing table and so on, post which it updates it’s local ets and the company (along with it’s metadata) is then ready to be processed.

Other gen server processes right now call a function in CompanyDataManager::process_enabled_companies which basically takes a function and args as parameters and runs the function for all the companies.

New setup:

The new setup plans to introduce a pooling in between so that any genserver process right now can be retrofitted to run in a pooled fashion based on the defined config.

The payload submission logic should be overridable wherein one process might have a different way of generating/submitting payload and handling it.

The default payload submission logic would be same as now which would be an infinite stream or a process variable based circular list kind of setup to iterate over list of companies and submit it to the worker.

Concern/Questions

For most of the part I understand that i can individually convert each process into a supervisor with two childrens: 1) ProcessManager- which will manage the logic of submission to worker, 2) DynamicWorkerSupervisor: which will manage the workers and it’s lifecycles.

• What i want to do is have a generic enough design so that i can later on move any other process from the :tick based mechanism to the pool based mech?
• Float the errors up and have custom error handling logic, similar to how genservers errors are treated by Supervisors (fail if X no. of errors are observed in Y time).
• Recommendation on a better design or alternative.