Will Erlang refactor out / optimize its use of `sscanf`?

In light of recent events that demonstrate that sscanf can have quadratic performance characteristics I was wondering if Erlang’s development is considering removing its usage of sscanf and replacing it with, say, strtof (if applicable) or a hand-rolled implementation even.

Here are the relevant articles for context:

Basically, using sscanf in a certain way (a really easy one to fall into) can severely hamper your performance.

As of today, OTP’s repository has 22 matches of the word sscanf in its GitHub repo: Search · sscanf · GitHub

Pinging @rvirding and @garazdawi with the question: do you guys think there’s still time to smuggle such performance reworks for the release of OTP 24.0?

6 Likes

If you can find a scenario where the use of sscanf is a problem, I’m all for replacing it. But I don’t think there is any reason to replace all without first seeing that it is a problem.

8 Likes

Sadly I can’t. My question was more about erring on the side of caution in a blanket manner. But I can understand that not being a good policy for code changes.

2 Likes

The article linked by @dimitarvp shows this:

So, to me this seems a problem, but I may be wrong… Can’t we try to replicate this in Erlang to see if we really got the issue?

Anyone who use 10meg C-strings deserve to run slow, or worse, OOM.

1 Like

You will not need to hit the 10 megs to get the speed penalty.

The algorithm used seems too slow, thus even if you have much smaller strings it will give you the penalty, just will be in seconds, instead of minutes :wink:

The non-usage of sscanf seems to net performance gains before that, it’s just that at 10MB and more it can actually become human-noticeable in one go. However, if you have a loop which uses a code path that uses sscanf 10-100 times, you can easily get hit by human-noticeable delays with much smaller inputs.

As said above, I can understand if it’s not deemed a priority but the HN threads of both articles have clearly demonstrated several other examples of workflows where performance problems become easy to notice.

1 Like

This is academic without reviewing where it is actually used:

We generally avoid C string functions because they’re deceptively complicated. Nearly all of them are locale sensitive and there’s no telling how awful the performance of that is, nor can we be sure that they do what we want: 4,3 can mean 4 or 4.3, \ can be a letter, and so on. To make matters worse these things can change at any time depending on what drivers or NIFs do and we just don’t want that headache.

Still, we use them in a few places where rolling our own hasn’t been worth it and the above isn’t a problem. Taking inventory, we use sscanf for:

  • Parsing command line options:

    • Online schedulers percentage" (e.g. +SP50:50)
    • Decentralized counter-/reader groups
    • Number of poll threads/sets
    • Meta tracing

    The command line is parsed once and is generally very small, so you’d have to go out of your way to trigger this issue.

  • Checking Linux kernel version when first starting the network driver. The check has been obsolete since kernel version 2.5 so we can just remove it altogether, but in either case we know the string is only a handful bytes long.

  • Checking zlib library version on Mac. This only done on the first call to zlib:inflateGetDictionary/1 on a string that’s only a handful bytes long.

  • Checking cgroup CPU quota. This is generally only done once on emulator startup, and only used on a few strings that are guaranteed to be short and hot in cache.

  • Grabbing memory information in os_mon. The worst-case scenario is that we sscanf 8 times in a buffer that’s smaller than 4K and guaranteed to be hot in cache. By default, this is called once per minute when memsup is started.

  • Grabbing CPU usage information on Linux as part of cpu_sup:util/0. We know that the string is hot in cache and shorter than 256 bytes.

  • Various debugging stuff that’s hidden behind #ifdef in normal builds.

None of these are performance-sensitive and their cost is negligible to begin with, so I don’t think this is a problem in practice.

26 Likes

Just for reference @dimitarvp pointed out where it’s used when he posted this topic:

1 Like

Thank you very much for your detailed analysis and to give us the peace of mind we were after :slight_smile:

2 Likes

Extremely helpful and informative, thank you!

5 Likes

whoops sorry I missed that @dimitarvp

2 Likes