I’m thinking about a way I’d like to visualise genstage/flow pipelines possibly with liveview. It’s a tricky thing but quite quickly I realised I need
- flow stats - messages going through each stage per second, maybe some demand stats
- A view to the code executing in each stage.
I guess the idea is to get something like a rudimentary spark ui type view. Is anyone already working on something like this?
Flow stats are easy with telemetry. But I have no idea if you can get code that is running? Any ideas?
For example, in a flatmap which is splitting a string, i’d like to get back "&String.split(&1, " “)”
Any ideas would be appreciated.