Hey everyone,
Can you help a guy out with a case of analysis paralysis?
Context
I developed a small library for a VM-wide pool of persistent workers – i.e. they don’t get started on demand; they get started together with your application and stay there until it shuts down. The idea is to have a global limiting of a resource in the app a la an Ecto.Repo
with a connection pool.
It’s a relatively thin wrapper around Task.Supervisor.async_stream_nolink
and its main value proposition is offering a function named each
that accepts a list and a function. It then multiplexes execution of the function on each item on all available workers e.g. sending a list of 7 items to a pool of 3 workers will result in only 3 parallel executions of the functions inside the worker processes, at a time. This each
function blocks until all items are processed.
I made it work and it works pretty well and I like it. The part I got analysis paralysis about is which usage pattern to utilize. I made two and I just couldn’t decide between both of them (although I do have a preference).
Option 1: dedicated module + use MyLibrary, params: ...
With this option I can do the following:
defmodule YourApp.YourWorkerPool do
# The following injects code in the current module via `defmacro __using__(opts)`
use MyLibrary, workers: 3, call_timeout: :infinity, shutdown_timeout: :infinity
end
then you put it in your app supervision tree:
def start(_type, _args) do
children = [
YourApp.YourWorkerPool,
# ...any other supervised processes...
]
opts = [strategy: :one_for_one, name: BusyBee.Supervisor]
Supervisor.start_link(children, opts)
end
You then use it like so:
YourApp.YourWorkerPool.each(items, function)
This works fine. The part I dislike is the need to have a dedicated module for it.
Option 2: a tuple inside the app’s supervision tree without the use
construct
E.g.:
def start(_type, _args) do
children = [
{MyLibrary, name: YourApp.YourWorkerPool, workers: 3, shutdown_timeout: 30_000},
# ...any other supervised processes...
]
opts = [strategy: :one_for_one, name: BusyBee.Supervisor]
Supervisor.start_link(children, opts)
end
Which is then used like so:
MyLibrary.each(YourApp.YourWorkerPool, items, function)
This works fine as well.
Option 3: have both
I… really don’t want to. To me that seems like a classic case of bloat.
Questions
If you were writing a library, which usage pattern would you go for? I admit I am leaning towards Option 1 for the following reasons:
- Making a minimal placeholder module to serve as an injection target of code + have it be a neatly separated place responsible for this functionality feels right. And having one more mini module in the app doesn’t feel like a big sacrifice.
- It’s more intuitive – I think, I am not sure – to do
MyModule.function
thanExternalLibrary.function(MyModule)
. - And pollutes the application’s supervised children with less visual noise.
Still, if you have any arguments in either direction, I am very curious to hear them. I personally knew former colleagues who would cringe hard at having to do use MyLibrary, ...
inject code that is basically copy-paste and they would insist they only want it in one place (yes, even if they only did the use
pattern only 2 times in their app). Are they focusing on the wrong thing and am I trying too hard to cater to such people?
Thank you for your time.