dimitarvp
Making a library with Supervisor + GenServers: can't decide what's more convenient to use
Hey everyone,
Can you help a guy out with a case of analysis paralysis?
Context
I developed a small library for a VM-wide pool of persistent workers – i.e. they don’t get started on demand; they get started together with your application and stay there until it shuts down. The idea is to have a global limiting of a resource in the app a la an Ecto.Repo with a connection pool.
It’s a relatively thin wrapper around Task.Supervisor.async_stream_nolink and its main value proposition is offering a function named each that accepts a list and a function. It then multiplexes execution of the function on each item on all available workers e.g. sending a list of 7 items to a pool of 3 workers will result in only 3 parallel executions of the functions inside the worker processes, at a time. This each function blocks until all items are processed.
I made it work and it works pretty well and I like it. The part I got analysis paralysis about is which usage pattern to utilize. I made two and I just couldn’t decide between both of them (although I do have a preference).
Option 1: dedicated module + use MyLibrary, params: ...
With this option I can do the following:
defmodule YourApp.YourWorkerPool do
# The following injects code in the current module via `defmacro __using__(opts)`
use MyLibrary, workers: 3, call_timeout: :infinity, shutdown_timeout: :infinity
end
then you put it in your app supervision tree:
def start(_type, _args) do
children = [
YourApp.YourWorkerPool,
# ...any other supervised processes...
]
opts = [strategy: :one_for_one, name: BusyBee.Supervisor]
Supervisor.start_link(children, opts)
end
You then use it like so:
YourApp.YourWorkerPool.each(items, function)
This works fine. The part I dislike is the need to have a dedicated module for it.
Option 2: a tuple inside the app’s supervision tree without the use construct
E.g.:
def start(_type, _args) do
children = [
{MyLibrary, name: YourApp.YourWorkerPool, workers: 3, shutdown_timeout: 30_000},
# ...any other supervised processes...
]
opts = [strategy: :one_for_one, name: BusyBee.Supervisor]
Supervisor.start_link(children, opts)
end
Which is then used like so:
MyLibrary.each(YourApp.YourWorkerPool, items, function)
This works fine as well.
Option 3: have both
I… really don’t want to. To me that seems like a classic case of bloat.
Questions
If you were writing a library, which usage pattern would you go for? I admit I am leaning towards Option 1 for the following reasons:
- Making a minimal placeholder module to serve as an injection target of code + have it be a neatly separated place responsible for this functionality feels right. And having one more mini module in the app doesn’t feel like a big sacrifice.
- It’s more intuitive – I think, I am not sure – to do
MyModule.functionthanExternalLibrary.function(MyModule). - And pollutes the application’s supervised children with less visual noise.
Still, if you have any arguments in either direction, I am very curious to hear them. I personally knew former colleagues who would cringe hard at having to do use MyLibrary, ... inject code that is basically copy-paste and they would insist they only want it in one place (yes, even if they only did the use pattern only 2 times in their app). Are they focusing on the wrong thing and am I trying too hard to cater to such people?
Thank you for your time.
Most Liked
Nicd
Option 2, 100%. The less macros there are in the Elixir ecosystem, the better. use makes it difficult to know what is going on with the module and makes compilation slower due to more compile dependencies. New users of Elixir already have it difficult enough with all the various libraries that have their macro tricks.
The latter is the “standard” way how things should work and it’s more readily understandable what is going on: clearly we are using lib MyLibrary to run something, instead of this magic module. If the user so wants, they can easily write
defmodule MyPool do
def each(items, function) do
MyLibrary.each(MyWorkerPool, items, function)
end
end
which again stays clear as to what is going on vs. a use statement (who knows what code is generated? how can I ensure I don’t accidentally make a function that conflicts with a generated one?).
If you want clarity in the supervision tree, you can in a similar vein write a function and use it in the list: children = [..., MyPool.child_spec(), ...].
The point is, option 2 leaves these tools available for the dev but doesn’t force them. As a dev I appreciate when I can make my own choices. The documentation can then provide useful copy-paste examples for novices.
One additional thing, which may not appear so useful here but might be more useful in other libraries: option 2 allows for runtime selection of the pool to use, whereas option 1 doesn’t. If this was needed the dev would need to use the option 2 form anyway, which would be non-obvious to discover and would mean they have two different calling conventions used.
mindok
Hi @dimitarvp - you would be in good company with either option - I’ve just looked through the app supervision trees of 3 or 4 apps and it appears reasonably evenly split - e.g. Ecto.Repo uses your option 1, whereas Phoenix.PubSub uses your option 2.
I prefer the clarity in the supervision tree of Option 1, but…
One advantage of Option 2 is less indirection - it’s easy to take a look into the entry points of your library and see what it is up to. Macro-based code can take a bit more effort to grok if there are issues, depending on what macro-fu moves you pull.
Also, Cachex is probably a good comparison given what it does. It uses Option 2. If your :name option is any atom (rather than a module), that could make your examples a bit cleaner, e.g.:
MyLibrary.each(:pool1, items, function)
instead of
MyLibrary.each(YourApp.YourWorkerPool, items, function)
Option 2 makes it a little easier to play with in iex:
{:ok, pid} = MyLibrary.start_link(name: :test_pool, workers: 15)
MyLibrary.each(:test_pool, 1..200, &{IO.inspect({self(), &1})})
GenServer.stop(pid)
Another point… libraries that use use do typically add behaviour (e.g. Broadway) or complex configuration that has to “stay alive” (e.g. Ecto.Repo) in MyModule, whereas in your case, the vast majority of the application-specific complex behaviour is externalised into items and function and only introduced into your scope for the purpose of multiplexing the calls, so IMO there is little value in having MyModule
It’s a good question, and no easy answer!
mattbaker
I came here to say something similar! I like the idea that option 2 allows me to create an abstraction if I want to de-clutter or hide details but I don’t have to, whereas option 1 doesn’t really give me the choice.
I think I often favor libraries that present collections of composable things because I can choose the level of abstraction I want.
What a fun question to ponder, I liked that you asked this @dimitarvp, I recently found myself in a similar bit of analysis paralysis myself
Popular in Discussions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









