Need some advice for Mozart performance testing

Are your GenServers CPU-bound?

Good question, but sorry. How do I determine this? I don’t see this info available in the observer app.

Can you show your “run n times” function?

Sure, this file:

https://github.com/CharlesIrvineKC/mozart/blob/main/lib/mozart/performance/demo.ex

the last function in the file.

If anyone cares to run these tests themselves, it wouldn’t require too much work:

• Clone the repo: https://github.com/CharlesIrvineKC/mozart
• get deps and compile
• iex -S mix

Paste to iex:

:observer.start()
import Mozart.Performance.Demo
clear_and_load()
spawn(fn -> :timer.tc(fn -> run_process_n_times(%{}, :process_with_single_service_task, 10000) end) end)

Thanks!

One thing that doesn’t look right in the observer System panel is the “Run Queue”. It stays consistently at 2. I would think it would go higher.

I discovered that each GenServer was getting shutdown after about one half of a millisecond. I increased the alive time of the GenServer to several seconds. When I did this, all 12 of the schedulers became utilized. I don’t understand this completely, but it kind of makes sense. So, that was one issue.

The second issue is that processing time doesn’t increase linearly with the number of processes serviced. I am still trying to figure that out.

One thing that’s not helping concurrency: doing work in init means that start_link takes longer to return. Consider moving the code from ProcessEngine.init to a handle_continue callback.

Another thing that isn’t helping concurrency: init and execute both need to make calls to singleton processes (ProcessModelService and ProcessService)

With a nearly 100x increase based on 10x more input, I’d start by carefully looking at where data’s being collected in the code; all it takes is one List.append that’s called per-ProcessEngine to make things quadratic.

Some other random thoughts:

• terminate doesn’t do any formatting on the reason argument, so if you call Process.exit(some_pid, :shutdown) (eg) you’ll get :shutdown as an argument - and the ProcessEngine target will crash!

• harping on the same point: terminate is not guaranteed, there are lots of (admittedly uncommon) scenarios where it will not run before the process disappears. Consider checkpointing the state during intermediate steps, if “resuming” is important.

• consider extracting type-specific code like this to a per-step (or per-type) “callback module”

@al2o3cr

Oh Wow. That is very helpful. I appreciate your effort. I’ll get back to you. Thanks

One thing that’s not helping concurrency: doing work in init means that start_link takes longer to return. Consider moving the code from ProcessEngine.init to a handle_continue callback.

I did that.

Another thing that isn’t helping concurrency: init and execute both need to make calls to singleton processes (ProcessModelService and ProcessService)

What you say is true, but why is making calls to those servers an issue? Perhaps you are alluding to it below, but I’m not sure.

With a nearly 100x increase based on 10x more input, I’d start by carefully looking at where data’s being collected in the code; all it takes is one List.append that’s called per-ProcessEngine to make things quadratic.

Some other random thoughts:

• terminate doesn’t do any formatting on the reason argument, so if you call Process.exit(some_pid, :shutdown) (eg) you’ll get :shutdown as an argument - and the ProcessEngine target will [crash!]
  (mozart/lib/mozart/process_engine.ex at e345b047076065de8d6b04e5b6a3b6cb75220f8b · CharlesIrvineKC/mozart · GitHub)
• harping on the same point: terminate is not guaranteed, there are lots of (admittedly uncommon) scenarios where it will not run before the process disappears. Consider checkpointing the state during intermediate steps, if “resuming” is important.

I had read that a terminate call isn’t guaranteed. Pity, I thought. Wonder why that is? I’ll need to research what might cause the call to be skipped.

If I did checkpointing it would probably be done frequently and it might be a big hit to performance. Maybe an occasional missed terminate call might be worthwhile.

• consider extracting type-specific code like this to a per-step (or per-type) “callback module”

That sounds like a sensible thing to to and a good exercise also.

Really appreciate your feedback.

Because every OTP process processes messages one by one. If you have 100+ processes each wanting something from that “central” process then that’s an obvious and very major bottleneck.