Hackney exposes very detailed metrics GitHub - benoitc/hackney: simple HTTP client in Erlang · GitHub, Specially hackney.POOLNAME.queue_counter can be very useful in determining whether timeouts are due to pool size.
ananthakumaran
2
9mm
ashneyderman
you can remove pool and see what you get. you will certainly be slower but at least you will be getting to the amount actual sockets possible. You will also figure out what goes first - CPU, network bandwidth.
Pool usage in this case is probably a good thing for you as it provides for some pushback mechanism. Something that I do not think you built (better way to do it is to place those at the gates not in the backroom with the pool).
0
9mm
I tried without it and it was crazy low. are you saying thats the exact amount that its limited by an external resource?
0
ashneyderman
baldmountain
Hi,
Sorry a bit late here. I worked on the adserver at Yieldbot and it had no issues handling 10,000 QPS on 6 instances (or was it 9) using IIRC 8 cpus servers with each instance using about 1G of ram. This should be doable. The servers were CPU bound not ram or network bound.
We were running our servers in Docker containers managed by mesos and singularity so it’s a bit different deployment. One thing we did run into is that the erlang VM thought it had resources for the underlying machine rather than what docker had allocated it. If you are running multiple instances on a single server they may all be creating schedulers for 64 cpus rather than the number you allocated for them. You’ll have way more schedulers than you need. Instead run several smaller servers 8 or 16 cpus and only a single instance of the application on each server. Setup haproxy or similar to handle routing to the instances. We used Baragon to update haproxy config when instances were started. Because we were using mesos we had to make sure only a single instance of a server came up on each slave. Also, new instances have to ramp up their load. If you hammer them all at once the schedulers get overloaded and the instance will crash. IIRC there is a haproxy setting to manage ramping up a new upstream server.
Not sure how Kubernetes works, but in docker/mesos the cpu allocation is a suggestion, but if the process uses too much, mesos kills the process.
Also IIRC don’t use bridge networking use host networking.
Definitely setup pools with hackney. The defaults are not good for this kind of load.
I should mention that Phoenix templates are awesome. We didn’t cache ads. We generated them on the fly. The server rendered html, css and js (along with a lot of checks and other processing) on every request and had a ~12ms response time.
I think that was most of the gotchas we ran into. And I hope I’m remembering everything correctly.
Oh, the only other thing was we had to open two ports. One for the server and one for epmd. Even if epmd wasn’t ever going to connect to another instance. haproxy only routed to the server port.
3
9mm
Wow this is super advice. A few questions…
-
Were these outgoing requests (10K/S)? or incoming.
-
what http library did you use?
-
Did you have any wisdom for choosing the pool settings? My main confusion right now is how to calculate the max pool size per instance… like the CPU chart doesn’t clearly a non-fuzzy view of how that correlates to CPU usage. its hard as well because traffic is always changing so I can’t exactly predict what CPU is for the increased pool vs just higher traffic. I was hoping theres some other way to detect pool size max.
Thanks again, amazing info
0
baldmountain
You’re welcome. 
-
Incoming requests. We did make a request to a server on the local network on each incoming request so both. (As well as cache requests to scylladb.)
-
We used HTTPoison and Tesla. Both were configured to use hackney. (IIRC hackney handled https connections better, but that may be history more than current status.)
-
The pools had 256 for max_connections. But we had a pool for each outgoing service. Don’t forget to specify the pool in the options passed to get/post or you’ll get the default pool.
We found that the CPU limit was more from the scheduler being busy than requests. We knew it was time to scale when the message queues started to get behind.
Above I mentioned scylladb. We changed our caching from redis to scylladb. Performance was much better than redis.
One other thing I’ll throw out there is we wrote our phoenix controllers to do most everything in plugs. A controller would look like:
defmodule ServerWeb.FooController do
use ServerWeb, :controller
require Logger
alias Plug.Conn
plug :put_layout, false
plug PluOne
plug PlugTwo
plug PlugThree
plug :plug_four
plug :plug_five
plug PlugSix
plug :plug_seven
plug PlugEight
def index(%Conn{assigns: %{param: param}} = conn, _params) do
conn
|> put_resp_header("cache-control", "no-cache")
|> render("index.html", param: param)
end
end
All the business logic happened in plugs. We could test them in isolation develop them in small isolated pieces. Some controllers had as many as 26 plugs.
I hope I am remembering all this correctly. Yieldbot shutdown the middle of December so all of this is from memory.
2
9mm
scylladb looks amazing. very interesting, thank you. So basically when you see MsgQ backup that means schedulers are the holdup and not the actual server CPU/resources?
Very interesting… this is very helpful, in my project but also just general understanding
0