Limiting maximum number of concurrent outgoing TCP/HTTP connections?

The system ‘outgoing’ ports is significantly higher than 65k, you generally should not worry on outgoing connections.

However, to limit scale you can either have a single genserver that just holds, say, a map of active connections, waiting for messages about them, or use a connection pool like poolboy or so.

The short answer is that no, there is no other limit than the ones you already know about.

The longer answer is that you may be uncomfortably close to some weird performance degradation that I haven’t yet figured out. Or not. I’d tell you the exact conditions that cause it, if I’d figured that out

Have you run into this yourself? I’m not sure if you’re mirroring my sentiment (things might go wrong in unexpected ways!) or offering advice

If you are using HTTPoison which uses :hackney. Make sure that you set max_connections to a high number. The default is 50. (config :hackney, max_connections: 1000). I think even open ports count towards the nofile ulimit. We ran into the :emfile issue while opening a lot of http connections. Also make sure that you read the response body so that the socket is freed up properly. If you make a request and don’t read the response body properly it tends to create issues.

Offering advice based on experience. Benchmark carefully. Benchmark at twice the load you expect to see if it falls apart.

I believe I am having a similar issue, but I don’t see this being reported on HTTPoison nor hackney. I still need more time in debugging this but do you still face the similar issue with the latest HTTPoison or hackney?

My use case is I would pattern match the HTTPoison.Response like such,

with {:ok, %HTTPoison.Response{status_code: 200, body: body}} <- HTTPoison.get(url) do
 ...
else
{:ok, %HTTPoison.Response{status_code: code}} when code in [foobar] -> ...
_ -> ...
end

And I am getting a lot of weird :emfile error as well.

Are you making many concurrent requests? Also, are you on OS X? There is a 256 default limit for open file descriptors in OS X and a new connection in hackney, I believe, opens a new socket. I bumped into that problem recently.

Nope I am on Amazon Linux 2 (which is quite similar to CentOS 7).

An emfile error points to an incorrectly set ulimit (you can check your current ulimit by running ulimit -n, but make sure that you run it as the user that your app is running as) or you are not pooling your HTTP connections properly. Even with a default ulimit of 1024, you’ll hit the limit if you open up 1K http connections without a connection pool. So, to debug this, I would do the following:

1. Figure out if you are using connection pooling in hackney
2. Find out what the cause of the emfiles is: is it too many open http connections, too many open files?
3. Use proper connection pooling and benchmark

I am indeed using connection pooling in hackney and I did set my ulimit to 500K beforehand so those are out of the way. Actually my observation is that when I ran lsof I can see that my “main” beam.smp is holding say, 800 TCP connections (which is well below the limit) but I do have some other Erlang-associated process such as 1_schedul, 2_schedul, ... and 1_dirty_c, 2_dirty_c, ... etc each holding an identical amount of TCP connections as my main beam.smp (I know they are associated with my main beam.smp because lets say my beam.smp PID is 4469, the PID of say 1_schedul would be 4469 some_other_number) and all of these add up to huge numbers.