Protox - A 100% conformant protobuf library

[ahamez]

Hi everyone,

I’ve been working on this protobuf library for 3 years. We use it in the company I work for, EasyMile, to communicate with our autonomous vehicles.

As you can imagine, reliability is very important! Therefore, a lot of efforts have been put into making sure that the library is reliable. To this end, I committed to make it pass all the 1302 conformance tests that Google provides for binary serialization. This is on top of having a 100% code coverage, plus property based testing and dialyzer.

In order to assess the performance of protox, I benchmarked it with random and handmade inputs against the other available libraries, exprotobuf and protobuf-elixir. You can find these benchmarks here (zip archive of a benchee HTML export). They show that protox is always faster than exprotobuf and most of the time on par or faster than protobuf-elixir. Of course, the result might be different with another computer/OS and with different inputs, I’ll be happy to include new ones.

Also, I think that protox is easier to use as it let you work directly with Elixir structures (for instance, there’s no need to use a new function to create a protobuf message, which means that you’ll have a nice error if you use a wrong field name). Furthermore, there’s no need to generate any file.
But of course, I may be a little biased here .
[Edit] It seems that file generation is important for some users, so I guess I should find a way to make it possible after all!

You can find it here: GitHub - ahamez/protox: A fast, easy to use and 100% conformant Elixir library for Google Protocol Buffers (aka protobuf). Let me know what you think!

[Qqwy]

This is very cool!
I have not had the chance to use Protobuf myself so far, but I am super stoked about the huge amount of work that clearly went into this library.

[WFransen]

First of all, my utmost thanks for creating this library.

Just some extra information: I’m using your implementation in order to (try) to create a gherkin parser and the underlying message protocol is defined according to the ProtoBuf protocol.

And to be clear, your library works great. Just a double check regarding the compile time dependencies:
Does this mean that every person who wishes to use this library that uses your lib as a depency, have to install protoc? Similar to Argon2 (which causes a lot of problems on Windows, according to my students).

Is there a way to overcome this? I’ve tried using exprotobuf (I don’t mind it being slower if it compiles without the need for external applications), but too bad that there’s a small issue there (going to create a topic later on about that).

Hence I’m using your library. Just wanted to say thanks again!

[ahamez]

Yes, protox requires protoc to be available at compile-time. It seems you need a library that generates code as you could just add the generated code to yours. I have some ideas to implement this, but I don’t know when I’ll be able to try those.
In the meantime, if you’re in a hurry, you can use https://github.com/tony612/protobuf-elixir. It’s not 100% compliant with protobuf specifications, but it will certainly help you to move faster.

[WFransen]

Thank you for the information ahamez. I’m watching this topic and am eagerly waiting for that code generation feature!

P.S. : if you need ever anyone to test an experimental feature, feel free to contact me!

Best of luck and keep up the awesome work!

[ahamez]

So, I decided to give it a try, and it works . It’s a little crude as it’s performed via a mix task (thus you need to download the repository to generate files).
It’s now possible to generate Elixir files from protobuf messages:

MIX_ENV=prod mix protox.generate --output-path=/path/to/messages.ex foo.proto bar.proto

Thus, protoc is only required when generating files and project including those files just need to add protox as a dependancy.

Let me know if it works for you!

[WFransen]

After trying this out, it works perfectly, thank you!

ElixirLs does take a lot to format these, but with enough RAM this all works out

Once again thanks for implementing this!

[ahamez]

protox 1.0.0 has been released! As no new features have been added and the API has been stable for a long time, I thought it was finally the time to mark it stable.

So, as a recall: why would you want to use protox rather than the others libraries like protobuf-elixir which has more features (and users )? Well, It really depends on how much do you need a reliable implementation. As protox has been written for the case of autonomous vehicles, reliability was more important than features. Thus, it has been has been thoroughly tested using the conformance checker provided by Google, has a 100% test coverage and uses property based testing.

Don’t hesitate to give it a try .

[sanswork]

I don’t need this at the moment but thanks for continuing to contribute to the ecosystem.

[ahamez]

protox 1.2.1 is now available!
The main changes since 1.0.0 are:

• generation of one file per module, which speeds up the compilation if you have a lot of protobuf messages
• possibility to ignore unknown fields

And of course, it’s still 100% compliant with the Google conformance checker .

[ahamez]

protox 1.4 is now available !

Thanks to the awesome work of @sneako, protox now supports proto3 optional fields.

It’s now possible to use the optional keyword to track field presence (note that it requires protoc >= 3.15):

syntax = "proto3";
message Msg {
  optional int32 foo = 1;
}

To check if a field has been set, you just have to compare it to nil:

iex> %Msg{}.foo
nil

Many thanks to @sneako !

[ahamez]

Even though Erlang 24 has been released 1 month ago, I thought it would be nice to assess the impact of Erlang 24 JIT on protox.
I ran some benchmarks yesterday, comparing with Erlang 23, and they show that it’s constantly faster with JIT:

• Decoding is 1.17-1.49 faster (average: 1.31), using 1.02-1.08 more memory (average: 1.06)
• Encoding is 1.10-1.92 faster (average: 1.58), using 1.03-1.17 more memory (average: 1.08)

A nice improvement in my opinion !

(You can perform theses benchmarks yourself using the instructions at https://github.com/ahamez/protox#benchmarks)

[ahamez]

protox 1.5 is out!
Thanks to cheng81 (Francesco Zanitti) · GitHub, it’s now possible to specify multiple import paths.

[ahamez]

protox 1.6 is out!
It’s a significant update as it provides support for JSON encoding and decoding, including Google well-known types (except Any).
Of course, Google conformance tests have validated that protox behaves as expected .

[ahamez]

I forgot: if you plan to use JSON and if you are already using the file generation feature, you should regenerate the files.