Utilizing 100% CPU through spawning multiple processes to execute parts of the loop in parallel is obviously better and turns out to be much faster than a simple single process loop for huge enumerations. A code snippet for people to use if needed. Cheers!
Looks neat! One nit, though, there are unnecessary bindings on line 5 (
followers =), 14 (
task =), 15 (
followers_chunk =) and 21 (
followers =) that may trigger compiler warnings on unused variable. They can be removed, since last expressions are always returned.
Yeah I forgot removing those parts, I did face those compiler warnings. I have updated the gist, cheers!
Try [element | acc] rather than acc ++ [element]. Appending (++) in a reduce loop means you’re copying the whole output list on every iteration. If order matters, you can reverse the list at the end anyway.
Awesome tip, thanks!