Are you sure that such a size is an actual issue here? I would tentatively expect the binary format of Erlang terms to be more compact that JSON (if keys are atoms), but even it they were not, copying one megabyte from process to process should take very little compared to its download and deserialization - that you have to do anyway.
l3nz
ollien
No, not necessarily. It’s just what I’ve narrowed down after a number of investigations
To give some details
- The “batch” operation that each thread waits on (inclusive of any I/O and JSON deserialization) takes very little time compared to how long the worker processes wait.
- The scheduler utilization is low, <= 10% on any scheduler. I don’t believe this means I’m waiting on a context switch.
- The work that the workers are doing is taking much more than what would seem to be the “sum of their parts”, as it were.
- This latency only happens when I apply a large amount of load to the service (i.e. many batched operations at once)
Ergo, by elimination, the only thing that makes any sense to me is that this a problem with the message passing taking too long.
0
TheMaikXX
Just wild thought. Instead of tossing tons of data around, there is another option, another point of view. If Mohammed cannot get to the mountain, mountain has to go to the Mohammed: Don’t send data to work, send work to the data..
You could have a process holding the requested data and have it receive a function which would process the data..
(I have no idea if it would actually be beneficial.. but just wanted to point it out, that it could be a way as well)
0
cevado
totally unrelated, but…
do you know if jobs is still maintained? there are several PRs from the maintainer that are not merged. i’d expect at least dependencies update and stuff like that.
0
dimitarvp
My apologies, I have no idea. I wouldn’t think such a library needs regular updates once it reaches feature completion though.
I tried the library once, about a year ago, and it got the job done. Liked it, would use again.
1
mpope
Passing refc binaries is a very reasonable solution that should be tested before being discounted. You’ll have a single reference instead of many small terms polluting in a process’s heap as they’re collected before sending.
0
l3nz
You could likely do two direct tests to check if that’s the case:
- “stuff” the map you are sending with a large, known map, that you know it’s approx a megabyte, maybe adding a large known response under
:uselesskey, so that every message you send is 2x as big as it was before. - do the correct fetching etc, but then send over the smallest possible map, and see if it makes a difference
My gut feeling is that on a single-box BEAM it should make very little difference, if the sending is 1:N - and even if you hit the network. Slow, maybe, but latency is where the BEAM supposedly shines, because most things end up being “context switches”.
Honest question - you are 100% sure that you are NOT sending all the batch to a single worker, where latency would be caused by it consuming messages in sequence?
0