Judging by your code, you’ll always handle a priority request in a freshly spawned process, while normal requests are serialized in the serializer GenServer. Given this, I don’t see the need for two GenServers. You could do the same thing if the priority request is handled in the client process. Doing that means you only need to deal with normal requests, so you only need one GenServer.
If however you want to limit concurrency and assign priorities to requests, then you need one GenServer which acts as a queue, and it will spawn separate processes to consume the items. Perhaps GenStage could be used for this, or otherwise plain GenServer together with monitored tasks will definitely work.
A few comments on your assumptions:
The normal calls don’t block GenServer (well, they do, but not for a long time), but they block the client.
Cast will indeed always succeed. However, if new requests arrive much faster than the serializer is able to process them, the serializer mailbox will grow, and you’ll experience some significant latency increases. Also, having a single serializer process means a lot of garbage is generated which adds to GC pressure. I’d rather start a separate task for each request, and use monitor to find out when the task is done. I explained this idea recently in this thread.
You’re basically on the right track. Your limiter GenServer will not wait until requests finish, so it will be able to queue more requests, even though some are currently being processed. The implementation is somewhat messy, so I suggest looking into the thread I linked for some pointers.
Assuming you’re not trapping exits in the limiter process, yes. However, there are some edge cases which might left your request processes dangling if the parent process terminates. Also, your structure is not friendly for user hot code upgrades. Finally, if any request fails, everything will fail. For example, if a single normal request crashes, all pending normal requests are lost, as well as all currently running priority requests.
To systematically address those issues, it’s usually good to keep all processes OTP compliant (aka special processes), and keep all parent processes supervisors. In your case this would mean something along the lines of having a separate task supervisor where you’ll start your request processors. The limiter GenServer would keep track of currently running tasks, as well as pending requests. It would also monitor started tasks, so it would know whenever some task terminates (normally or abnormally). When that happens, the limiter GenServer can start the next task, or even restart the failing task if you want to that. Keeping all tasks under a supervisor also mean you can stop them in a predictable fashion, and ensure proper process cleanup.
You can make such test predictable by introducing some message passing dance between request processor and test process. Basically, you want your normal request to block until you send it some message. Then, you can start a high prio test and verify that it succeeded. Finally, you send the “unlock” message to the blocked process and assert that it succeeded.
