dimitarvp

dimitarvp

Making a library with Supervisor + GenServers: can't decide what's more convenient to use

Hey everyone,
Can you help a guy out with a case of analysis paralysis?

Context

I developed a small library for a VM-wide pool of persistent workers – i.e. they don’t get started on demand; they get started together with your application and stay there until it shuts down. The idea is to have a global limiting of a resource in the app a la an Ecto.Repo with a connection pool.

It’s a relatively thin wrapper around Task.Supervisor.async_stream_nolink and its main value proposition is offering a function named each that accepts a list and a function. It then multiplexes execution of the function on each item on all available workers e.g. sending a list of 7 items to a pool of 3 workers will result in only 3 parallel executions of the functions inside the worker processes, at a time. This each function blocks until all items are processed.

I made it work and it works pretty well and I like it. The part I got analysis paralysis about is which usage pattern to utilize. I made two and I just couldn’t decide between both of them (although I do have a preference).


Option 1: dedicated module + use MyLibrary, params: ...

With this option I can do the following:

defmodule YourApp.YourWorkerPool do
  # The following injects code in the current module via `defmacro __using__(opts)`
  use MyLibrary, workers: 3, call_timeout: :infinity, shutdown_timeout: :infinity
end

then you put it in your app supervision tree:

  def start(_type, _args) do
    children = [
      YourApp.YourWorkerPool,
      # ...any other supervised processes...
    ]

    opts = [strategy: :one_for_one, name: BusyBee.Supervisor]
    Supervisor.start_link(children, opts)
  end

You then use it like so:

YourApp.YourWorkerPool.each(items, function)

This works fine. The part I dislike is the need to have a dedicated module for it.


Option 2: a tuple inside the app’s supervision tree without the use construct

E.g.:

  def start(_type, _args) do
    children = [
      {MyLibrary, name: YourApp.YourWorkerPool, workers: 3, shutdown_timeout: 30_000},
      # ...any other supervised processes...
    ]

    opts = [strategy: :one_for_one, name: BusyBee.Supervisor]
    Supervisor.start_link(children, opts)
  end

Which is then used like so:

MyLibrary.each(YourApp.YourWorkerPool, items, function)

This works fine as well.


Option 3: have both

I… really don’t want to. To me that seems like a classic case of bloat.


Questions

If you were writing a library, which usage pattern would you go for? I admit I am leaning towards Option 1 for the following reasons:

  • Making a minimal placeholder module to serve as an injection target of code + have it be a neatly separated place responsible for this functionality feels right. And having one more mini module in the app doesn’t feel like a big sacrifice.
  • It’s more intuitive – I think, I am not sure – to do MyModule.function than ExternalLibrary.function(MyModule).
  • And pollutes the application’s supervised children with less visual noise.

Still, if you have any arguments in either direction, I am very curious to hear them. I personally knew former colleagues who would cringe hard at having to do use MyLibrary, ... inject code that is basically copy-paste and they would insist they only want it in one place (yes, even if they only did the use pattern only 2 times in their app). Are they focusing on the wrong thing and am I trying too hard to cater to such people?

Thank you for your time.

Most Liked

Nicd

Nicd

Option 2, 100%. The less macros there are in the Elixir ecosystem, the better. use makes it difficult to know what is going on with the module and makes compilation slower due to more compile dependencies. New users of Elixir already have it difficult enough with all the various libraries that have their macro tricks.

The latter is the “standard” way how things should work and it’s more readily understandable what is going on: clearly we are using lib MyLibrary to run something, instead of this magic module. If the user so wants, they can easily write

defmodule MyPool do
  def each(items, function) do
    MyLibrary.each(MyWorkerPool, items, function)
  end
end

which again stays clear as to what is going on vs. a use statement (who knows what code is generated? how can I ensure I don’t accidentally make a function that conflicts with a generated one?).

If you want clarity in the supervision tree, you can in a similar vein write a function and use it in the list: children = [..., MyPool.child_spec(), ...].

The point is, option 2 leaves these tools available for the dev but doesn’t force them. As a dev I appreciate when I can make my own choices. The documentation can then provide useful copy-paste examples for novices.

One additional thing, which may not appear so useful here but might be more useful in other libraries: option 2 allows for runtime selection of the pool to use, whereas option 1 doesn’t. If this was needed the dev would need to use the option 2 form anyway, which would be non-obvious to discover and would mean they have two different calling conventions used.

13
Post #3
mindok

mindok

Hi @dimitarvp - you would be in good company with either option - I’ve just looked through the app supervision trees of 3 or 4 apps and it appears reasonably evenly split - e.g. Ecto.Repo uses your option 1, whereas Phoenix.PubSub uses your option 2.

I prefer the clarity in the supervision tree of Option 1, but…

One advantage of Option 2 is less indirection - it’s easy to take a look into the entry points of your library and see what it is up to. Macro-based code can take a bit more effort to grok if there are issues, depending on what macro-fu moves you pull.

Also, Cachex is probably a good comparison given what it does. It uses Option 2. If your :name option is any atom (rather than a module), that could make your examples a bit cleaner, e.g.:

MyLibrary.each(:pool1, items, function)

instead of

MyLibrary.each(YourApp.YourWorkerPool, items, function)

Option 2 makes it a little easier to play with in iex:

  {:ok, pid} = MyLibrary.start_link(name: :test_pool, workers: 15)
  MyLibrary.each(:test_pool, 1..200, &{IO.inspect({self(), &1})})
  GenServer.stop(pid)  

Another point… libraries that use use do typically add behaviour (e.g. Broadway) or complex configuration that has to “stay alive” (e.g. Ecto.Repo) in MyModule, whereas in your case, the vast majority of the application-specific complex behaviour is externalised into items and function and only introduced into your scope for the purpose of multiplexing the calls, so IMO there is little value in having MyModule

It’s a good question, and no easy answer!

mattbaker

mattbaker

I came here to say something similar! I like the idea that option 2 allows me to create an abstraction if I want to de-clutter or hide details but I don’t have to, whereas option 1 doesn’t really give me the choice.

I think I often favor libraries that present collections of composable things because I can choose the level of abstraction I want.

What a fun question to ponder, I liked that you asked this @dimitarvp, I recently found myself in a similar bit of analysis paralysis myself

Where Next?

Popular in Discussions Top

vans163
So useless benchmarks aside, Its possible to write a webserver that can serve 300k requests per second (perhaps more with optimizations)....
New
Donovan
Hello everyone, I’m so glad to have discovered this awesome community. Thanks for creating it! This is my second post, and apologies for...
New
Ankhers
Just a little information upfront. Generally speaking, if I feel like I need to either break a pipe chain or use an anonymous function in...
New
AlexMcConnell
The reason that Rails is as popular as it is is because it’s very easy for relatively inexperienced developers to get a lot of work done....
588 19652 166
New
AstonJ
Are there any Elixir or Erlang libraries that help with this? I’ve been thinking how streaming services like twitch have exploded recentl...
New
Crowdhailer
I’ve been hearing much about the new formatter and it’s something I have been keen to try. I find examples buy far the most illuminating...
248 19327 150
New
CharlesO
Erlang :list.nth simple, but 1 - based nth(1, [H|_]) -> H; nth(N, [_|T]) when N > 1 -> nth(N - 1, T). Elixir Enum.at … coo...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
griffinbyatt
Sobelow Sobelow is a security-focused static analysis tool for the Phoenix framework. For security researchers, it is a useful tool for g...
New
sashaafm
Piggy backing a bit on @dvcrn topic BEAM optimization for functions with static return type?, I’ve been trying to understand in a deeper ...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement