Flow Process Time Estimation

[pmangalakader]

Hi all,

I have been recently working a lot with Flow and the process usually takes hours to complete with basic resources, but I need a way to estimate the process completion time before or after flow has started and running. i.e., similar to progress bar

Thank you,

[aseigo]

Do you know how many items (e.g. jobs) there are to process? Do you know, or have a way to estimate, how long each step for each item in the flow will take? Those are prerequisites. With those in hand, you can start to do more or less hand-wavy progress estimations (# of jobs/steps to process; or amount of expected time) and then send messages from within the functions in the flow to an updater process as to how the flow is going.

[pmangalakader]

Let’s say a process takes 0.5 secs and I have 1 million items to process, stages and demands are default and I don’t want the results, I am using flow to execute process simultaneously and I want to estimate the time for above scenario and need a solid way to find out how much time would it take for flow to complete all 1 million or in the future say few millions.

[aseigo]

Great, so take your volume*per_item_time_slice and divide that by the number of CPU cores you throw at it and, as a rule of thumb, multiple by 0.7 (allowing for 30% “overhead for messaging, context switching, etc.”) to get a basic idea of what you can expect in the best of conditions: no bottlenecks in the GenStage pipeline steps, and one Flow running on that hardware.

I usually start with that and then actually measure against a few run “dry runs” to see how close it is to reality. Assuming it is close, then comes the actual hard parts:

• Analyzing the flow for processing stalls; this can be due to greater serialization in a given step (such as in a reduce phase) or one phase dominating in computation over others (despite the level of parallelism applied)
• Measuring the effect(s) of running multiple of the same Flows simultaneously

Then you can move to a slightly better idea of expected runtimes across the Flow pipeline. From there if your Flow reports metrics / progress back you can reasonably expect to map that with some level of fudge to your expected runtimes for display. In the end, raw progress (N out of X, where X is a combination of processing steps and things to process) is going to be more “accurate” to show, but you can also get some rough time-based guidelines as well if you work through it.

[pmangalakader]

@aseigo

Great, so take your volume*per_item_time_slice and divide that by the number of CPU cores you throw at it and, as a rule of thumb, multiple by 0.7 (allowing for 30% “overhead for messaging, context switching, etc.”) to get a basic idea of what you can expect in the best of conditions: no bottlenecks in the GenStage pipeline steps, and one Flow running on that hardware.

I usually start with that and then actually measure against a few run “dry runs” to see how close it is to reality. Assuming it is close, then comes the actual hard parts:

• Analyzing the flow for processing stalls; this can be due to greater serialization in a given step (such as in a reduce phase) or one phase dominating in computation over others (despite the level of parallelism applied)
• Measuring the effect(s) of running multiple of the same Flows simultaneously

Then you can move to a slightly better idea of expected runtimes across the Flow pipeline. From there if your Flow reports metrics / progress back you can reasonably expect to map that with some level of fudge to your expected runtimes for display. In the end, raw progress (N out of X, where X is a combination of processing steps and things to process) is going to be more “accurate” to show, but you can also get some rough time-based guidelines as well if you work through it.

Thanks @aseigo. That is what I was looking for.

[zubairshokh]

@aseigo Thanks. we have been struggling with this since few days. Thanks a lot