This feels like two separate layers of testing to me but I just don’t have any experience with telemetry so I can’t answer intelligently. But it would seem to me you would want to test that a chat message triggers a telemetry event and then a general test that telemetry events are being consumed and saving to the db (and depending on how detailed you wanna get, you could test all your event names are valid).
EDIT: sorry @hkrutzer I realize now you essentially said this already.
This can and should be reformed into sending messages to dedicated OTP process(es) and just assert they are received. Persistence of telemetry is an implementation detail that has zero relevance to your app’s tests.
If process A does a cast to process B, then process A can afterwards call :sys.get_state on B and after that returns you are guaranteed that the cast has also been handled. Messages between two processes A and B are always received in the order they are sent, and so the GenServer.call done inside :sys.get_state will be processed after the cast.
Telemetry hooks are always fired from the the process that emits the telemetry event, eg the process calling Repo.insert. So as far as I can tell if your test pid calls a function that calls Repo.insert, and then that has telemetry cast to some other pid2, your test code should be able to :sys.get_state(pid2) and at that point you’re guaranteed to be after it has processed the cast.
That is kind of true, but the telemetry is transformed first, so there is an interdependency between the data across the modules and processes.
For example, you fire from process A some telemetry that contains a boolean value. You receive the telemetry in process B, and for some reason you need to transform it into a 0 or 1 and save it into your database. You test these things separately. Now if someone changes some code so that A no longer produces a boolean, but instead produces a string (or a 0 or 1, doesn’t matter). They also change A’s tests. All looks to be fine. But after deployment, B tries to consume the telemetry and now it breaks, because its format has changed.
You can imagine a similar case where data isn’t transformed in this way in B, but instead filtered, merged etc.
Thanks! I didn’t think of the :sys.get_state trick. That will probably solve at least this specific case!
Not entirely, I have one process (A) that fires telemetry, which is cast to another process (B) and B then performs the insert (and in some cases waits for multiple events, to merge them together). But your suggestion should still work if I get_state(B).
I feel you on this view as I felt it at first, but the way the way I see it is that we create mocks for wrappers around services (if we’re following “don’t mock what you don’t own”). Even if we only end up having one production implementation, it’s still probably a good idea to define a behaviour. For example, perhaps there are some public functions added to an implementation that aren’t actually part of the contract. That’s the way I see it, at least, but I’ve been wrong a bunch over the past 24 hours 
I haven’t followed the complete discussion, but to me this sounds like the sole reason you have a hard time testing this. Instead of waiting for process A to generate telemetry can you make it do that on command? If so your test can command it to do what it does and you can expect all the downstream effects to happen and assert on the resulting db records.