Thanks for bringing Domo to my attention. It looks like a very powerful (and nicely documented!) suite of tools. However, I suspect that runtime performance concerns may scare off some possible users. There’s also a bit of a conflict between the “check everything” and “let it crash” mantras.
So, I’ve been speculating about ways to employ Domo on a dynamic basis (e.g., controlling its activity via supervision trees). Although the following notes are pure science fiction, it might be possible to make something similar work.
Let’s assume that Domo has been compiled into a set of modules, but its runtime checking has not been activated. At some point, a process crashes and the appropriate supervision tree is called in to repair the mess.
At this point, we can “activate the immune system” of each of the restarted processes. We give each one a numeric value which, if non-zero, causes it to turn on Domo’s checking. Each time the process calls itself, the value gets decremented (stopping at zero). So, if this was just a random data glitch, the checking will go away.
However, if some other process is sending bad data on a repetitive basis, Domo will be primed to catch, analyze, and report the problem. And, as part of this activity, the initial counter value(s) can be restored or even increased.
It might even be possible to use some sort of graph-based tracking and analysis to get at the root cause(s) of the problem. For example, Domo could alert the process that sent the bad data, saying “you seem to have a problem; please turn on your Domo checks.”.
OK, that’s enough late-night speculation for this tired camper; let me know if anything here seems worth considering…