Understanding CPU under-utilisation with observer

I am a newbie to Elixir and have a question about a small application that processes messages from two websocket servers. It holds state in a key-value bucket and sends messages out to the servers periodically.

There are five modules:

  1. A key-value bucket Agent
  2. One websockex client with some processing of messages, storing results in (1)
  3. Another websockex client with very little processing, also storing results in (1)
  4. A diagnostic module that prints to the console occasionally, pulling state from (1). It mostly sleeps.
  5. A continuously looping module/function that pulls state from (1), writes to (1) and sends some websocket messages out.

As expected (1) and (5) are very busy. The application runs on a four logical core system but consistently uses only 25 percent of total CPU available spread across two cores/schedulers. Also, the CPU usage is strictly out of phase with each other.

As the calls to the key-value Agent are synchronous I am not surprised if that it is a bottleneck, but I don’t understand why the application maxes out at precisely 25 percent of total CPU (i.e. one core’s worth) that is distributed across two cores in an exactly out of phase manner. Shouldn’t (5) run much hotter on average than (1)?

Or am I not reading the relevant info from observer correctly? Are there other things I should be looking at? Any help most appreciated.

It’s a bit hard to say without looking at the code. Your use of Process.sleep within a process isn’t particularly idiomatic, perhaps there is another bottleneck as well.

Based on what you’ve said though, let’s imagine that the agent is on scheduler 1, and the looper is on scheduler 2. If the agent is busier, that is to say, it’s under contention, then that directly impacts the ability of the looper to perform work. If it has to wait for 10ms to get data, then that’s 10 ms when scheduler 2 is idle. The more work the Agent scheduler has to do, the more other schedulers are effectively blocked and doing no work waiting on it to be done. This creates an inverse relationship between the activity of scheduler 1, and the activities of other schedulers.

Thank you for the guidance. I am no doubt that my code is highly inefficient and unidiomatic. The key value bucket process is exactly like the “KV.Bucket” example in the standard docs.

The looper simple reads a bunch of values from the key value bucket, performs some simple calculations and fires off some websocket messages occasionally. May I ask what I can read on how to identify bottlenecks in the code using observer or any other tools?