I’ve been using Elixir for the past two years or so, and I’ve noticed that every single JSON library has it’s own benchmarks, that all say that they are the best. I wanted to know which one was the fastest currently, so I decided to build my own benchmarks. I’d love inputs from everyone to see what people think!
That repo also contains various benchmarks of Elixir core that I’ve used to find the fastest versions of different functionalities in the core Elixir language.
In most benchmarks Jiffy tends to be the fastest of the well known libraries. Jason is slower but has more use throughout the community and its pure elixir. We use a combination of jiffy and jason at work and benchmark against our specific payloads.
One of my side projects right now is building a web server that can get faster performance than cowboy (I know, it’s a pipe dream). Part of that was choosing the fastest JSON parser, and when I went exploring the popular libraries, I saw that in every benchmark there were wildly different results, each favoring whichever library I was looking at, so I made my own little benchmark to see which was the fastest.
I know that Jason is the current standard, I use it when I use Plug or Phoenix, it’s one of my favorite libraries.
This was just a quick side project of mine to get a more definitive benchmark, since every library typically advertises the benchmarks that show their library to be the best, and I wanted to know which one was without any bias.
I can’t speak for @michalmuskala (the creator) but IMO the selling point is that it’s pure Elixir and has no native dependency. And that it’s stricter.
Especially if you deploy to various Docker and VPS configurations this can be a life saver since some VMs and hosting providers can be finicky when compiling native code. Not having to deal with various GCC / clang versions and distro-specific patches can make a huge difference.
Agreed, I am not saying it’s always hard, I am only saying that not using it gets you rid of one more potential deployment problem that will bite you just at the right night at 2:00 AM.
As a guy who usually works without devops and sysadmins around, I appreciate this a lot. Other people’s workflows vary of course.
It’s the same question I had, especially after my benchmarks. Jiffy has such a massive performance boost here that it just makes sense to use it. And it’s not like it’s an unknown library, it’s been established for a while as THE library in Erlang.
For my own uses I simply wrote a little Elixir wrapper around Jiffy so that I can make it look more Elixir-y, and still use Jiffy. It makes it a little easier to use Jiffy, which at the end of the day really isn’t that difficult.
Out of curiosity, what kind of deployments have you done where libraries that use NIFs are problematic?
My company has a few Elixir services in production, and they use some Erlang libraries and some NIFs as well, and we’ve never had any issues in deployments using Docker.
I feel like this might come off as mean, and I want to stress that I am not trying to be mean, this is genuine curiosity because I want to know if I might run into these issues in the future.
It was a few years ago and was on an older Ubuntu and changing the server was non-negotiable. I can’t remember the details but I was left with some mild PTSD on the topic.
Since then, and when working with Elixir, I am OK with sacrificing some performance and go with NIF-free dependencies. I am sure things have improved quite a bit in the meantime.
I started my last couple of side projects because there is a kind of rivalry at my company between Go Programmers and Elixir Programmers, and the main point that the Go programmers continuously make is performance. Of course the performance they are talking about is by using extremely optimized low level Go libraries like fasthttp instead of the more established and simpler to use web frameworks web frameworks, so I decided to start looking at libraries that leverage NIFs better, like Jiffy, to see how big the performance boost is, in order to prove that when used write and written well, Elixir/Erlang can be just as performant.
You guys should not have a rivalry based on speed, but on:
Code readability and long-term maintainability. Dunno about you but I regularly had to mentally squash 30 lines of Golang code to “oh, this does map with filtering, and then does a reduce” in my head. It’s exhausting and is not helping me want to learn more.
Fault-tolerance, which people only appreciate when they see how their language of choice does not have it. I prefer an Elixir server maxing out 50k / sec users that never makes a mistake and keeps everybody’s latency to acceptable levels, to 400k / sec users in Golang and having to periodically look for the reason of yet another abrupt panic (which happened several times in my previous Golang contracting but hey, I might just be an idiot, that’s a fair possibility as well).
Brevity. This is related to the first point. Less lines of code == less bugs, that’s basically one of the very few universally proven rules of programming. Most of the functional languages are pretty terse and that’s an asset.
I still dig Golang for a number of reasons but its verbosiveness and bad long-running services story are not appealing. You can also read the gaming streaming service Twitch.TV’s blogs on their adventures with Golang which are mostly positive – but they had some pretty tough fights to win along the way.
Jiffy is generally faster on single-threaded benchmarks, the difference is much less pronounced when you start running concurrent processes all doing JSON stuff - Jason being all Elixir means it uses schedulers better - Jiffy being in C has a bit harder time dealing with that. At least those were my findings when I was benchmarking it around the initial release.
Also - the fact that it is in C is problematic, especially on Windows. It raises the entry bar considerably if you need to compile C on Windows, which many beginners use. I can only speculate this could be a reason for Phoenix sticking with a pure-elixir library as a default.
Additionally, especially with encoding, Jason can be much faster in practice. Jiffy can only deal with simple data types - strings, integers, maps - you need to encode everything down to that before handing it off to jiffy - especially annoying with things like DateTime or similar. This usually means two-step encoding - from your application to simple terms and from that to JSON. Because Jason is based on protocols, in can do all of that in a single, extensible pass. Furthermore, with deriving and the macros in Jason.Helpers module, you can do some additional work at compile-time to speed up the encoding even further. I wouldn’t be surprised if all of that noticeably outperforms jiffy.
I also have some ideas that would make it possible to do single-pass decoding into more complex data structures - in a similar way this should give an edge to Jason over jiffy if you don’t need to go over your data twice. At the same time it’s hard to find an API that would be both fast and extensible. As a last point when it comes to performance, when initially writing Jason, I had plans to include some optional native code to speed up the hottest loops - but there wasn’t that much demand for it so far, so it’s waiting for better times .
Just a note, this is more of a friendly rivalry than a real one. Our main core of the company is being written in Golang, but we have legacy core services written in Elixir.
My argument for choosing a language is always going to be code readability and developer happiness, because at the end of the day if you want the best performance than just write everything in assembly.
Unfortunately, right now a large amount of our services are being written in Golang, so it is seeming to be where we are moving to right now.
Thanks for jumping in with this! I always love hearing from the core team members in the language to see what they think.
I agree that because jiffy is written in C it could be problematic, especially for Windows users. This is one of the beauties of working with Docker/containers, that way you don’t have to worry about the system you are using locally, just the one you’ll be using in the field.
What kind of benchmarks would you run to try and get more accurate results here? Mine were pretty simplistic, it’s just running encode/decode on the same input and that’s it. I’d love to start setting up benchmarks to see how various libraries deal with multithreading and schedulers, more complex data types, etc. Do you mind sharing the benchmarks and files that you use for checking Jason?
The benchmarks are in the repo - https://github.com/michalmuskala/jason/tree/master/bench - it’s the same ones that Poison uses. With Benchee, you can also pass the parallel option to run concurrent benchmarks, but in that case the results are much, much less reliable. Running them on a real service is probably the only way to measure it reliably and check how it will perform in your use case.
We do what @michalmuskala suggests. We’ve benchmarked Jason and Jiffy with each of our individual services and we go with the one that makes the most sense.
But JSON is a garbage data format and we’re actively exploring alternative formats for our service communication layer.
.
