Tweet: Thread pool startvation affects everyone with an example of a DNS service

Chat / Discussions
, ,
1

Today I saw this tweet:

https://twitter.com/davidfowl/status/1362962967789072386

While it starts in the context of DOTNET:

Thread pool starvation affects everyone. Learnt about a service that went down because the dns server died and dns queries were hanging (there was no timeout). That in itself is a problem but it causes issues because dns resolution is synchronous… #dotnet

Dns resolution is historically blocking on every OS. See getaddrinfo. Modern operation systems have an asynchronous version of this API. Windows has GetAddrInfoExA/W https://docs.microsoft.com/en-us/windows/win32/api/ws2tcpip/nf-ws2tcpip-getaddrinfoexa and Linux has https://linux.die.net/man/3/getaddri

DNS resolution on windows is fully async on .NET if the operation system supports it. In .NET 6, we support it on linux as well. Both of these were external contributions!

Before .NET 6 and in most other platforms I’ve looked at, dns resolution is still blocking. This means that even if you use the async APIs, it’ll kick off a synchronous operation and block a thread pool thread.

I then moves to compare how it is done in NodeJS:

This lead me to look at what nodejs does here. Would it have run into the same problem? Nodejs uses libuv which is a libraries that tries to abstract various OS operations (file, socket, tty, etc) into a single unified API. But what does it do for dns resolution?

Turns out, for the APIs that aren’t truly asynchronous everywhere (file IO, DNS) nodejs uses a threadpool that runs work off the event loop. This means you aren’t blocking the event loop but non-event loop threads are being blocked with dns queries.

What happens when these threads are starved? Well node does this thing where it only uses 1/2 of the threads for this “slow IO”, the other work just queues up behind this blocking work. This results in a similar effect to thread pool starvation as the queue length just grows…

This would manifest as callback not being called and lots of work being scheduled (assuming lots of new DNS requests keep showing up) but never completing. I’m not sure what this is called in the node world, if anyone knows, please let me know.

Now he goes about comparing how its done in Golang:

Let’s see what the other languages do. Go blocks a goroutine and waits on a channel. It’ll eventually spin up a new OS thread if the goroutine blocks too long. Strangely, this logic uses cgo :thinking:

Due to nature of pre-emptive scheduler of the BEAM and the possibility of having millions of BEAM processes running at same time I have the impression that this would not be such a big issue, but now I am curious to understand what would be implications on the BEAM when running DNS service application.

So, what is your take about a DNS service running on the BEAM? Could it lead to OS threads starvation?

2 Likes
2

It’s not at all an issue unless your OS/kernel hit you with limits – in which case you can’t do anything.

BEAM’s philosophy is to basically allow you to have a million synchronous I/O requests and check which one is ready to go – and it does that really fast (thanks to Linux kernel magic like select / epoll). Golang’s core feature – the goroutines – and Rust libraries like rayon work exactly like your quote described:

Which is a way to be conservative about spawning threads because they are a more expensive operation in terms of CPU, RAM, context switches and what-have-you.

IMO just having a warm thread-pool ever since your program starts – like the BEAM does – is the best policy. Just have anywhere from N to N*4 threads ready (where N = number of CPU cores) when the program starts.

I would caution against using the BEAM for a DNS server; Erlang/Elixir are not known for raw muscle power. Using Rust’s async capabilities is IMO literally the best option of all. A well-written Rust program can saturate a 10Gbps network card on a mid-range server.

1 Like
3

Wasn’t there a blog post recently about a DNS service using Elixir/Erlang? Can’t find it now but do remember this - Erlang basically powers the internet :003:

90% of the worlds internet traffic goes through Cisco’s Erlang based routers:

Screenshot 2021-02-26 at 18.57.44
Screenshot 2021-02-26 at 18.57.442792×1372 219 KB

2 Likes
4

Through “Erlang controlled” not “implemented in Erlang”. Switches have highly optimised FPGAs as a “core” as even hand written assembly in generic purpose CPU would be way to slow for high throughput switching that is needed at the core of the internet.

And there is nothing about DNS in this talk.

2 Likes
5

Yes it was more an aside and I’d love for Cisco to blog more about how they use Erlang and how it might help in other related areas, particularly those that underpin the internet.

With regards to Elixir/Erlang and DNS itself, not being able to find that article is going to bug me now :lol: I’m sure they talked about how they went to handling thousands of additional requests a second with Elixir and iirc, it was a very recent article - like a few days or weeks old at most :neutral_face:

1 Like
6

You guys are probably talking about this:
https://elixirforum.com/t/are-you-using-phoenix-in-production-for-public-facing-traffic-lets-hear-your-story-if-you-want-to-get-interviewed-email-based-or-a-podcast-come-on-in/25825/76?u=derek-zhou

5 Likes
7

YES that is the one! Thank you Derek! It would have driven me nuts otherwise :044:

5 billion DNS queries a month with a huge requirement to have maximum uptime

2 Likes
8

I never said that Erlang isn’t good choice for DNS server. In fact I am thinking about writing PiHole alternative in Elixir/Erlang, as the UI of that is quite limiting (for example I can set only A records for local domains or it seems that there is no UI for DHCPv6). Maybe one day I will do that.

3 Likes
9

I already responded on twitter, but for the sake of completeness I’ll redo it here. From what I can tell, Erlang uses its own dns resolver, inet_res, which is written in plain Erlang, so this issue should not surface in Erlang.

More generally, if the runtime uses a potentially long-running syscall, the thread pool can be exhausted. My feeling is that the OTP team tries to avoid these scenarios, and that such issues would already surface in practice, but I don’t have any data to back this claim.

5 Likes
10

So, @sasajuric invited @garazdawi to the conversation and seems that now we have the answer:

7 Likes
11

Today I wake-up and looking to my previous post I am not sure If I totally understand the Tweet I linked above.

Some questions for @garazdawi:

You mean that Erlang can do OS blocking calls that make OS spin threads, thus making possible for the BEAM to starve the OS from threads?

You mean a DNS implementation? Can you link to both? If not can you expand on this?

1 Like
12

Yeah, I am also interested. Thought he also said that there is a choice between two implementations and the first one actually can scale much more than a normal pool of OS threads doing synchronous I/O.

1 Like
13

There is inet_res that @sasajuric posted earlier. DNS look up is just UDP client so I assume it is not too much trouble to do in erlang.

1 Like
14

Just to make one thing clear first. I am not an expert when it comes to DNS, nor really how it works in Erlang. I know some things, but not all. This is how it works to the best of my knowledge.

The tweet I was referring to was https://twitter.com/davidfowl/status/1365232349584052227?s=20

Erlang provides both a UDP-based implementation and “native” implementation. You can configure which is used here: Erlang -- Inet Configuration.

The UDP-based one is written completely in Erlang with the pros and cons of that. You can see the implementation here. This can be faster than the native one or not depending on what you are doing. There is work ongoing to make it better.

The “native” implementation is a port program called inet_gethost. You can see the implementation here. This program starts a pool of threads that call gethostbyname and communicates to erlang via stdin/stdout. inet_gethost was written long before nifs became a thing, which is why it is not a nif.

Both of these have been around for about 20 years with tweaks along the way, but the main point is the same. Sometimes you want the OS name resolution and something you don’t. It depends on what it is that you are doing.

9 Likes
15

Thanks a lot for the more detailed explanation.

I think that now I have a better understanding but I am still left with a question, that was the origin of all that discussion in Twitter and the reason for this post:

So, is Erlang capable of causing OS threads starvation, be it with DNS queries or any other OS blocking syscall?

2 Likes
16

Yes it can definitely happen. I do not think it is as likely as many other systems, nor is the effect as bad.

5 Likes
17

I think inet_res is there for speed, not for robustness. If your DNS server is not responding, you are screwed in multiple ways and os threads starvation may not be very high in the list.

1 Like
18

I am not really worried about DNS(it is just the example in the Tweet), instead I just wanted to know if it was possible for the BEAM to starve the OS from threads, and it seems that is possible as mentioned in the previous post by @garazdawi.

19

But as mentioned, Erlang can handle this kind of situation more gracefully, and I think that it’s important to remark.

2 Likes
20

Yes, I already have done exactly that in the Twitter thread:

1 Like