We have a service which only purpose is to receive JSON in one endpoint, validate the fields using ex_json_schema and send to Google Cloud Pub/Sub system as fast as possible (using google-api_pub-sub). I load tested the service a bit and got only about 193 requests/s through (with correct data) running in n1-standard-1 virtual machine in Google Cloud (1vcpu, 3,75GB memory). Performance should be at least 100x better, hopefully more, but I’m stuck on which way to proceed. I would be happy to share the code if needed!
I would be happy to share the code if needed!
Yes, please.
If i remember correctly Goth.Token.for_scope(@gcloud_scope) is performing an http request each time you call it. If this is happening, will add quite a bit of latency for each publish.
Try benchmarking each part of the request separately. Maybe split them into functions and run benchee on them. Or use a profiler.
- goth
- ex_json_schema
- httpoinson
- jason
I’ve had problems with goth before (config is stored in a single genserver, it quickly becomes the bottle neck), it was too slow for me. Never used ex_json_schema. Some people have problems with httpoinson as well, fortunately, there are faster alternatives like https://github.com/lpgauth/buoy. jason should be fine.
Looks like it is caching tokens in a GenServer. But uses synchronous calls to fetch them. TokenStore
