Fl4m3Ph03n1x

Why should I uses Tasks instead of Pools?

Background

A book I am reading makes the following statement:

Tasks will allow us to do one-off jobs and still rely on the rich OTP library. Connection pooling libraries let us share long-running connections across processes.

So, my understanding of this is as follows:

if I have a one off task, I should use tasks
if I need persistent connections to something, I should use pools

Why not always use pools?

And this is my questions. Even if I have one-off tasks, I still think pools are superior because they don’t have to create a new process to do the work, everything is already created.

I could understand the argument here if using pools was significantly harder than using one-off Tasks, but today with libraries like poolboy and books like Elixir in Action I do feel it is easy to understand, create and use pools these days.

Questions

What are the major benefits of using one-off Tasks instead of a Pool?
In what scenario would I be better of using a one-off Task instead of a Pool?

10 comments

#worker-pools #task

15 1859 10

2019-11-28 14:56:46 UTC

Marked As Solved

peerreynders

For me the whole point of the BEAM is that processes are lightweight and creating processes is inexpensive.

This property makes it very attractive to view processes a disposable - much in the same way as memory is “disposable” in a garbage collected language.

This is also helpful when it comes to resilience. If a process is disposed of after a single use, it is much less likely that you’ll end up with a situation where some weird edge case is corrupting your process state over time.

If the process lifetime is short enough, memory may never have to be garbage collected and is simply returned wholesale to the system after process termination.

So, personally I would favour short lived processes over long lived ones - until there is a good reason for making it long lived.

if I need persistent connections to something, I should use pools

More generally, if your process works with resources that are expensive to create and reusable then it should be a worker in a pool. Connection pools are built around that concept because database connections are expensive to create and there is an effective, finite limit to the quantity of simultaneous database connections.

The typical tradeoff is that long-lived processes should be as simple as possible to minimize the danger of state corruption over the long term. So it wouldn’t be that unusual for a long-lived process to delegate a lot of its work to short-lived processes while the long-lived process only manages simple state transitions.

Furthermore, from the resilience perspective, it may be worthwhile making worker processes “perishable” - terminate them once they have been used too often or aged too much and replace them with a fresh worker (and resources).

Post #10

Also Liked

NobbZ

The only 2 reason I could spontaniously come up are those:

Pools also always introduce some kind of rate limit or upper bound for concurrent processing, while you can spawn as many tasks as you want. Of course there are pools which can handle this by dynamically spawn processes when exhausted, but if this occurs often, then you are basically back to even slower spawning tasks.
Task is part of the standard library, pools aren’t.

Post #2

lud

Well for some reasons people could use the process dictionary, for random numbers related stuff for instance. Or some people could register the process to a registry, use a library that would link/monitor the process from another process, create ETS tables that would not be destroyed.

Post #6

hauleth

Because often we pool other resources as well. Sometimes these resources are limited (RAM when you do image processing, you want to limit amount of the images processed at once to keep RAM usage in reasonable boundaries), sometimes setup can be long and it is better to do it once and then keep it alive for longer time (HTTP keep-alive connections), and sometimes there is mix of both of those (for example DB connections).

Post #7