I am implementing the Raft protocol as part of my master thesis using Elixir.
I am using a gen server for each major component of the Raft protocol including, the communication layer between the nodes, the MessageProcessing for each node and more.
The algorithm works fine most of the time but sometimes a function that typically takes few milliseconds (~4ms) to run, runs for more than 100ms. I am measuring the run time of the function using the Time.diff/3 method and timestamps at the entry and exit of the function.
I am collecting the logs using the Logger module and saving it to a file.
I am searching for a way to save detailed system logs for each genserver to try and understand why the processes hang on some lines for more than 100ms at random times for no reason. The timestamp on logs is very important in order to correlate the events.