Understanding nimble_pool internals

I don’t know the internals of nimble_pool but I remember it saying that you should accept a single-process access to your resource. Meaning that if you have 5 processes each needing access to the resource they’ll have to wait for each other to finish. Is this what you require?

As for handling crashes, you can simply spawn your workers under a Supervisor or DynamicSupervisor that you configure and they’ll get automatically restarted – though I am not sure that is your intended behavior either.

Can you talk a little more about your exact ideal scenario? There are good options for many requirements but we need to know a little more about what you need.

yeah, I think in case of worker crash, just restarting them would work. But there is also the possibility that a process checks out a worker and crashes, which means it wouldn’t return the worker to the pool which means although the worker is free, other processes wouldn’t be able to check it out.
The solution to this, would to be to monitor any process that checks out the worker and return it to the pool if that process crashes but that would require listening to the DOWN message from that process which would block the manager and that would mean as soon as a process checks out a resource, no one else can check out a resource again until the first process returns the resource or crashes. That means no parallel resource usage, which I think is the main point of having a resource pool.

Are you saying using nimble_pool you can’t have parallel resource usage? But then what’s the point?! I could just not use a pool and have one resource and use it sequentially! Although I don’t remember their docs saying such a thing, could you quote the relevant part?

As for my use case, I’ve written several packages in gleam, two of which are mongodb and redis clients. I want to use puddle(the package being discussed here) to add connection pooling to them but I want it to be general enough so people can use it for pooling any resource(potentially connection pooling for any database client). But as I said, if the only way to do this is using connections sequentially, then I don’t understand what even the point is of having connection pooling!

From https://hexdocs.pm/nimble_pool/NimblePool.html:

The downside of NimblePool is that, because all resources are under a single process, any resource management operation will happen on this single process, which is more likely to become a bottleneck. This can be addressed, however, by starting one NimblePool per scheduler and by doing scheduler-based dispatches.

I believe the idea is to give you tools and for you to build your own LEGO, which is a strong reason why I don’t like nimble_pool.

Not quite, ecto for example controls parallel usage of a DB quite fine e.g. maximum 10 connections per DB. A single-process access is not assumed when talking about a resource pool – hence my question to you to clarify your requirements.

If you want a single-process bottlenecked access to a resource then, well, a simple GenServer will do the trick because you can send messages to it (to give you access or copy data back to you etc.) because the messages arrive one by one.

If you want parallel access then using a job pool library (like Erlang’s :jobs or Elixir’s opq) can also have their arms twisted to help you.

Another option is just using good old :poolboy.

I don’t read that as parallel resource usage being impossible, it just means because one process is managing everything, that one process would be a bottleneck which would be true regardless of parallel or sequential usage!

As far I as know, ecto itself does not communicate with any database, it uses database clients. Although I get your point, e.g. postgrex uses db_connection, so maybe I should look at that.

That’s actually a good point, I think I wasn’t looking at it the right way!
Although it would take some time for the connection to both send the command to the database and receive the result, it would be good to be able to still handle requests in that time. But maybe that’s just not industry standard, I don’t know!

Yeah, I should look at those too, seems like just looking at nimble_pool is not enough.
Thanks

Point still stands, incl. for ecto + postgrex + db_connection because you still can do 10 parallel connections to a DB, it’s never just one.

There are plenty of those but still, what’s most important are your own requirements. Again, can you explain what you require exactly? I feel we’re going into circles. Elixir and its ecosystem have everything you need for any scenario in job pooling or resource pooling. But you need to give a clear problem statement.

My 2 cents is that you either choose simplicity advantage(debugging and reasoning about how this works) or performance, the 2 concepts discussed here cannot be mixed in way where you don’t sacrifice one for other.

Fair, but I don’t have a specific use case in mind. puddle is a package for resource pooling and the primary reason I wrote it is to use it to add connection pooling to mungo(my mongodb client) and radish(my redis client). So what I want is to have something on par with tools currently used in elixir. So basically if something is good enough for a top package in elixir, it’s good enough for me and I’m trying to find out what those requirements are.

true, but the fact is, I’m trying to understand the two sides of that tradeoff so I can make an informed decision.

Then definitely start with :poolboy because nimble_pool is much more manual and can confuse you.

Pooler appears to have a leg up on poolboy due to starting workers asynchronously and tracking worker process state:

https://github.com/epgsql/pooler