Hi
I’ve been working on a resource pool package for gleam for some time and this is what I have right now.
Currently it accepts a pool size and a function that creates a resource(can be anything) and creates a pool size number of processes that each has a resource as the state.
Then people can use an apply function that accepts a function that acts on the resource and returns a result. apply will checkout a process, send a message to it with the function so that the process can use that function on the resource, get the result and return it to the manager process, which will checkin the process and return the result to the user.
But right now this approach does not handle the situation in which either the worker process or the user process crashes and it seems to me that I won’t be able to handle those situations and still be able to provide parallel resource usage with this approach.
So I’ve been trying to understand how nimble_pool works so that I can implement a similar approach. But I’m having a hard time following what’s going on in the code(I’m a nodejs developer by day and do BEAM stuff as side projects, so not really proficient in OTP).
The only thing I’m sure about right now, is that nimble_pool will directly give the resource to the user. It also seems like there is a timeout after which it just checks the resource in, regardless of what the user has done with it but I’m not really sure about it or anything else.
I don’t know the internals of nimble_pool but I remember it saying that you should accept a single-process access to your resource. Meaning that if you have 5 processes each needing access to the resource they’ll have to wait for each other to finish. Is this what you require?
As for handling crashes, you can simply spawn your workers under a Supervisor or DynamicSupervisor that you configure and they’ll get automatically restarted – though I am not sure that is your intended behavior either.
Can you talk a little more about your exact ideal scenario? There are good options for many requirements but we need to know a little more about what you need.
yeah, I think in case of worker crash, just restarting them would work. But there is also the possibility that a process checks out a worker and crashes, which means it wouldn’t return the worker to the pool which means although the worker is free, other processes wouldn’t be able to check it out.
The solution to this, would to be to monitor any process that checks out the worker and return it to the pool if that process crashes but that would require listening to the DOWN message from that process which would block the manager and that would mean as soon as a process checks out a resource, no one else can check out a resource again until the first process returns the resource or crashes. That means no parallel resource usage, which I think is the main point of having a resource pool.
Are you saying using nimble_pool you can’t have parallel resource usage? But then what’s the point?! I could just not use a pool and have one resource and use it sequentially! Although I don’t remember their docs saying such a thing, could you quote the relevant part?
As for my use case, I’ve written several packages in gleam, two of which are mongodb and redis clients. I want to use puddle(the package being discussed here) to add connection pooling to them but I want it to be general enough so people can use it for pooling any resource(potentially connection pooling for any database client). But as I said, if the only way to do this is using connections sequentially, then I don’t understand what even the point is of having connection pooling!
The downside of NimblePool is that, because all resources are under a single process, any resource management operation will happen on this single process, which is more likely to become a bottleneck. This can be addressed, however, by starting one NimblePool per scheduler and by doing scheduler-based dispatches.
I believe the idea is to give you tools and for you to build your own LEGO, which is a strong reason why I don’t like nimble_pool.
Not quite, ecto for example controls parallel usage of a DB quite fine e.g. maximum 10 connections per DB. A single-process access is not assumed when talking about a resource pool – hence my question to you to clarify your requirements.
If you want a single-process bottlenecked access to a resource then, well, a simple GenServer will do the trick because you can send messages to it (to give you access or copy data back to you etc.) because the messages arrive one by one.
If you want parallel access then using a job pool library (like Erlang’s :jobs or Elixir’s opq) can also have their arms twisted to help you.
I don’t read that as parallel resource usage being impossible, it just means because one process is managing everything, that one process would be a bottleneck which would be true regardless of parallel or sequential usage!
As far I as know, ecto itself does not communicate with any database, it uses database clients. Although I get your point, e.g. postgrex uses db_connection, so maybe I should look at that.
That’s actually a good point, I think I wasn’t looking at it the right way!
Although it would take some time for the connection to both send the command to the database and receive the result, it would be good to be able to still handle requests in that time. But maybe that’s just not industry standard, I don’t know!
Yeah, I should look at those too, seems like just looking at nimble_pool is not enough.
Thanks
Point still stands, incl. for ecto + postgrex + db_connection because you still can do 10 parallel connections to a DB, it’s never just one.
There are plenty of those but still, what’s most important are your own requirements. Again, can you explain what you require exactly? I feel we’re going into circles. Elixir and its ecosystem have everything you need for any scenario in job pooling or resource pooling. But you need to give a clear problem statement.
My 2 cents is that you either choose simplicity advantage(debugging and reasoning about how this works) or performance, the 2 concepts discussed here cannot be mixed in way where you don’t sacrifice one for other.
Fair, but I don’t have a specific use case in mind. puddle is a package for resource pooling and the primary reason I wrote it is to use it to add connection pooling to mungo(my mongodb client) and radish(my redis client). So what I want is to have something on par with tools currently used in elixir. So basically if something is good enough for a top package in elixir, it’s good enough for me and I’m trying to find out what those requirements are.
