Benchmarks - how important are they to you? To Elixir?

Hi Isaac,
thank you for coming here and being curious about our discussion of the project.
I haven’t written code for the debian Benchmarks Game yet. But yeah. I was looking at GMP. Not realizing that Erlang uses GMP under the hood and it also uses PCRE.

So it would probably be interesting seeing if we can bring the speed closer to the top.

It’s really hard making these kinds of benchmarks because each language has their own way of doing things that might not align with those kinds of benchmarks.

So I’ll probably try to contribute Erlang or Elixir examples if I can find some speed ups instead of complaining about them on a forum

I speak about telemetry, and You speak of other frameworks… I don’t see the relation.
Have You ever implemented telemetry in Elixir?
It is specific to the BEAM how it is done.

Yeah totally, like other languages Elixir has its take on metric gathering too.

Originally you mentioned the idea of telemetry is unique to the BEAM. I was just trying to say that lots of other languages have the ability to get various metrics out of their runtime and then display those metrics in easy to digest ways. It’s just not called telemetry in most other languages.

I meant telemetry could answer your question about request duration…

Most popular languages / frameworks have their own features and libraries to get similar stats.

For example with Flask, Rails, Laravel and Django there’s various forms of “debug toolbars” that give you stats like how long a request took, how many DB queries you had, how long those queries took, how much time was spent compiling a view, queries against Redis, etc..

These are drop in solutions requiring zero lines of code and are extensible to add your own custom stats. There’s also language specific tools for profiling function calls and getting break downs of how long it took to execute a specific part of a function, etc..

Then there’s also a bunch of APM tools out there for nearly every language that’ll let you get in depth stats about pretty much anything you could imagine. All results are persisted too, so you can compare and chart the results over time.

System specs like memory / CPU usage, etc. is also a solved problem with many dozens of tools. Often times you can get a moving chart (with weeks of persistence) of this straight from your cloud provider for free and without having to set anything up. DigitalOcean will do this for you.

What about Telemetry, something that is quite specifiic to BEAM ecosystem?

Absolutely. When I work with Elixir and Rust and I get torn between 2-3 implementations I simply write a benchmark to decide what to use in the current project.

Yep! Like every “game” where score is the only thing that matters, the very predictable result is that people will – ahem – game the system to get as much score as possible.

I too don’t take T.E. seriously these days. They don’t do enough code quality control.

Same, that’s what I want to see in benchmarks. People get too focused on low-level details!

50/50, obviously Rails is more mature but cachex is quite easy as well, plus many – myself included – find the hand-crafted approach with just a few more code lines preferable because it’s more explicit. That’s beside the topic though and I get your point, just not sure if your example is precise.

Generally agreed but “easiest” isn’t the only important metric. Simple isn’t easy but simple makes your code more future-proof. So I’d slightly modify your statement when it applies to me: “I prefer to benchmark the simplest solutions (even if they turn out not that easy to code) because that’s what makes a project easily evolvable and is thus adding to the ‘real world project’ definition”.

I’ll also link to The Real World Project which is IMO a very good starting point for more realistic benchmarks.

I get really annoyed with some of the benchmarks ( Benchmarks Game ) because in they are often using the exact same library for the computations…

Please tell us specifically which tasks you think show programs which use " the exact same library".

Please tell us how many other tasks are shown which you do not think show programs which use " the exact same library".

If all you mean is that many of the pidigits programs use GMP, and many regex-redux programs use PCRE then how does that effect comparison of say Erlang and Ruby programs?

In fairness to Actix, while they were doing stuff that isn’t considered best practice in Rust, it is (iirc anyway) being used in production by Microsoft. I think this further supports the need for different categories in such benchmarks, so one for what is typical of that language or framework, and another where you can hack things to your heart’s content to squeeze every last drop of performance This would help people see what they can get out of a system from the get go, and what the potential of it could be at a later stage.

I’s agree that for 90% of apps most languages would probably suffice, however, for some strange reason we seem to be attracted to the promise of more. I don’t think that’s necessarily a bad thing tho - having ambitious plans for a project will probably (imo) lead to a more polished product because the developers may be much more passionate about it

Edit:

I took a trip down memory lane via an old MetaRuby thread (which was funnily enough after looking for a thread to link to in this recent thread - funny how one thing can lead to another )) and this was one of the graphics that was part of the whole hype/excitement:

There are lots of comments about Elixir being like “Ruby on steroids ” etc …so I’m more certain than ever that they (/the performance aspect) plays (or played, at least back then) an important part…