Huh, that’s weird that it only handles the dynamic flatbuffers when flatbuffers is all about the static schema (which would be a great fit for macro’s to generate the module!).
OvermindDL1
keathley
I haven’t personally. I typically go from json to msgpack if I just wanna get an encoding/decoding performance boost and smaller payloads. If I need schema’s (which is rare) then I use protobuf or avro. I’ve been aware of flatbuffers but they’ve never really fit a use case or problem I had. My default is to use msgpack whenever possible as its widely supported and is a “free” upgrade over json for 98% of use cases.
1
OvermindDL1
The benefit of flatbuffers is no encoding/decoding steps in most languages (and in fact barely one on the beam), and for languages that support it the data can be read directly with no translations at all, in addition the schema means that useless data like named fields don’t need to be specified as everything become positional thus more tightly packed. Overall it is significantly faster than protobufs or msgpack (at least in C++ where I use flatbuffers), I’m curious how that would translate to the BEAM when using a well made library for it.
1
keathley
Sure, I get how they work and why they’re useful. But for all practical purposes msgpack tends to be fast enough for a lot of my use cases and much more widely supported. If you’re going to use json you can almost certainly drop in msgpack instead and it’s an overall win. I also don’t need to squeeze every last microsecond out of my response times. At some point there’s diminishing returns :).
0
OvermindDL1
Oh I don’t use it for web stuff, I use it mostly for network transmission of packets that I need to both be as fast and tiny as possible, not just on the transmission time but on actually (de)serializing the packet. ^.^
HTTP overhead will swamp any real gain from it anyway. However, not everyone uses Elixir for just http stuff (I use it for a lot of non-http stuff).
1
dimitarvp
Do you have any benchmarks and/or observations on how much quicker msgpack is compared to JSON in your Elixir projects?
0
keathley
With our payloads encoding and decoding with msgpack is about 2x-3x faster than json and there’s about a 15% reduction in payload size.
1
dimitarvp
Do you have a set of anonymised payloads we could make a GitHub repo to benchmark with? I’d contribute a FlatBuffers section to the initial JSON / msgpack benchmarks.
0
keathley
I don’t but I can try to put something together. I suspect our payloads aren’t that unusual though so you could probably use any standard payloads for benchmarking
0
OvermindDL1
If you want to compare json with a couple things, flatbuffer’s benchmark includes those (the super fast rapidjson, which is a lot faster than anything in elixir) and so forth:
Flatbuffers, due to knowing the possible schema, is significantly faster than rapidjson in every single way in binary mode, and when encoding to/from flatbuffer’s json it is still significantly faster than rapidjson, just not as fast (still a lot faster than just 3x-4x speedup that msgpack supposedly gives), and flatbuffer’s binary format is significantly smaller message size too.
1