Looking for a way to get source code at runtime

Questions / Help
,
1

Hey,

I’m thinking about a way I’d like to visualise genstage/flow pipelines possibly with liveview. It’s a tricky thing but quite quickly I realised I need

  1. flow stats - messages going through each stage per second, maybe some demand stats
  2. A view to the code executing in each stage.

I guess the idea is to get something like a rudimentary spark ui type view. Is anyone already working on something like this?

Flow stats are easy with telemetry. But I have no idea if you can get code that is running? Any ideas?

For example, in a flatmap which is splitting a string, i’d like to get back "&String.split(&1, " “)”

Any ideas would be appreciated.

cheers,

2

You can try to get it via beam_lib, however whether it work or not highly depends if you strip your releases.

3

It might be better to have users instrument their stages with a macro. Then you can ensure appropriate information is set. Otherwise you’d need releases that aren’t stripped as mentioned above. It might also be possible using a compiler tracing plugin?

2 Likes
4

thanks for the ideas, i’ll look into them and get back if i have a solution :slight_smile: