I was wondering if in the context of Flow a Flow.chunk would be useful. I could wrap partition and reduce in a function for sure, but maybe the dev experience would be nicer if Flow.chunk would exist.
What’s your opinion?
Ah so the issue is, the default demand values for GenStage (and thus flow) is 1000, which means it tries to pull 1000 chunks, each with 2000 items in them before passing it off to a worker. This is obviously problematic. You should do
Ben, thank you that solves the problem in a more elegant way!
Yes, I would use Flow.map(fn x -> Indexer.index(chunk) end) to have a chunk of 2000 lines
It was not crystal clear to me that the default value for max_demand was 1000, I guess I should RTFM better