Almost everywhere in my code snippet it is possible to use Stream instead of Enum, which means you end up with a stream of to-be-created jobs, meaning that the next one is only constructed once there is a worker that wants to consume it.
If you’d like to do this, you probably want to change the Enum.map(&create_jobs/1) to a Stream.flat_map, as well as changing the Enum.map(&create_job(pivot, &1)) to a Stream.flat_map.
It is only difficult to reverse a Stream since we don’t yet know how many elements there will be, but this is something that the code you showed in the first post also does not address.
(And for that note, it is definitely possible to create a version that works with streams that starts from the beginning, but memory usage will ever increase since the first elements of old batches will all have to be retained to compare to newer oners; this is why I presumed that your set of input elements have some well-defined maximum size after which old ones do not have to be compared to newer ones anymore).
This is a fun exercise 
