Thank you for your contribution.
As you mentioned, the threshold parameter is very sensitive to performance, so I guess it requires some kind of parameter optimization technology for practical usage. I have a plan to develop such a technology.
zacky1972
Creator of Pelemay
1
zacky1972
Creator of Pelemay
dom
Why is there such a big performance difference with the pmap here:
https://github.com/zeam-vm/pelemay_fp_benchmark/blob/main/lib/pelemay_fp_benchmark/parallel.ex
Is it because the particular benchmark you’re doing benefits from batching? If so, could you get the same benefit by calling Enum.chunk_every before the map, and Enum.concat after? (modifying the async task to call Enum.map as well)
0
zacky1972
Creator of Pelemay
This code is from Programming Elixir by Dave Thomas.
It is simple, but has such a performance bottleneck, which you mentioned.
I tried to use Enum.chunk_every before implementing PelemayFp, but it was slower than I expected. After the new year’s holidays, I’ll show it to you.
0
dom
Oh you’re right, I played with it a bit and Enum.chunk_every is quite slow because the implementation is generic for all enumerables. If I write a specialized version just for lists instead, then I get similar timings for PelemayFp, Task.async_stream, and pmap (with batches of 12,000 items).
I noticed a small detail, the benchmark code is using a module constant for the list:
@list Enum.to_list(1..100000)
This may affect the timing a bit since constants aren’t subject to garbage collection and don’t get copied when they’re sent from one process to another, unlike variable data.
0
zacky1972
Creator of Pelemay
@list Enum.to_list(1..100000)This may affect the timing a bit since constants aren’t subject to garbage collection and don’t get copied when they’re sent from one process to another, unlike variable data.
To solve it, I guess it should be implemented using before_each_bench and after_each_bench.
0
zacky1972
Creator of Pelemay
I’ve implemented PelemayFp.ParallelSplitter.split/7 as a faster integrated function like Enum.chunck_every/2 with Process.spawn/4. See it hexdoc of it: PelemayFp.ParallelSplitter — PelemayFp v0.1.2
0