Read Protobuf data in Elixir

Questions / Help
1

I’m trying to parse some strings in protobuf format using the @bitwalker’s exprotobuf, but I’m not sure how to do that. From the examples it seems like the library is designed to implement Elixir modules using .proto files but all I want to do is get the contents of these protobuf strings in readable format.

I know this library should be able to decode the proto strings before converting them into modules, but there isn’t any documentation for that. I’m currently trying a few things with Protobuf.Decoder.decode/2 to see if that works for me.

Please help!

iex(33)> inspect proto_data                       
"<<10, 204, 30, 18, 201, 30, 26, 207, 2, 40, 176, 165, 174, 251, 156, 42, 48, 5, 74, 89, 103, 112, 58, 65, 79, 113, 112, 84, 79, 70, 120, 70, 107, 114, 103, 78, 89, 48, 122, 117, 111, 106, 71, 73, 115, 67, 113, 95, 111, 119, ...>>"

iex(34)> inspect proto_data, binaries: :as_strings
"\"\\n\\xCC\\x1E\\x12\\xC9\\x1E\\x1A\\xCF\\x02(\\xB0\\xA5\\xAE\\xFB\\x9C*0\\x05JYgp:AOqpTOFxFkrgNY0zuojGIsCq_ow2M-ve0gElq6xNzW7ijGBeT_9gotFOHArnkh3xrfIhamGIGjY956kSnkrFPg\\x8A\\x02\\xE7\\x01\\n\\x1Cperson-108247076822183591710\\x12\\x15108247076822183591710\\x18\\x1C \\t*\\x0FShahzaib NaseerRe\\b\\x04*\\\\ .... .... ....."
2

Okay, so I’m now trying to read a .proto file, this is the error I get:

** (Protobuf.Parser.ParserError) 92('syntax error before: ', '\'{\'')
    (exprotobuf) lib/exprotobuf/parser.ex:68: Protobuf.Parser.parse!/2
    (exprotobuf) lib/exprotobuf/parser.ex:10: anonymous fn/3 in Protobuf.Parser.parse_files!/2
        (elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
    (exprotobuf) lib/exprotobuf/parser.ex:8: Protobuf.Parser.parse_files!/2
    (exprotobuf) lib/exprotobuf.ex:68: Protobuf.parse/2
    (exprotobuf) expanding macro: Protobuf.__using__/1
                 iex:14: GP (module)
        (elixir) expanding macro: Kernel.use/2

I get the error on the Identifier line:

message BookDetails {
    repeated BookSubject subject = 3;
    optional string publisher = 4;
    optional string publicationDate = 5;
    optional string isbn = 6;
    optional int32 numberOfPages = 7;
    optional string subtitle = 8;
    repeated BookAuthor author = 9;
    optional string readerUrl = 10;
    optional string downloadEpubUrl = 11;
    optional string downloadPdfUrl = 12;
    optional string acsEpubTokenUrl = 13;
    optional string acsPdfTokenUrl = 14;
    optional bool epubAvailable = 15;
    optional bool pdfAvailable = 16;
    optional string aboutTheAuthor = 17;
    repeated group Identifier = 18 {
        optional int32 type = 19;
        optional string identifier = 20;
    }
    optional bool fixedLayoutContent = 21;
    optional bool audioVideoContent = 22;
    optional bool isAgencyBook = 23;
}
3

I’ve got a Chromecast client working that uses Protobuf, I had to modify .proto file a bit to get it to compile.

Here’s the client https://github.com/NationalAssociationOfRealtors/chromecast/blob/master/lib/chromecast.ex

I had to remove this line from .proto file, https://github.com/minektur/chromecast-python-poc/blob/master/cast_channel.proto#L9

4

This might be an overkill but my current workaround is to use protobufjs:

def parse_protobuf(data) do
  binary_list = :binary.bin_to_list(data)

  {out, _exit} = System.cmd("node", ["-e", """
    var protobuf     = require('protobufjs');
    var builder      = protobuf.loadProtoFile('./proto/response.proto');
    var response     = builder.build('Response');

    var buffer       = new Buffer( #{ inspect(binary_list, limit: 10000) } );
    var decodedData  = response.decode(buffer);

    console.log(JSON.stringify(decodedData));
  """])

  out
  |> Poison.decode!
  |> ExUtils.Map.symbolize_keys
end

I rather come up with an Elixir/Erlang only solution, but this would have to do until I can figure out why ExProtobuf doesn’t work for my proto file.

5

Internally exprotobuf uses https://github.com/tomas-abrahamsson/gpb for parsing .proto files, encoding, decodin etc. Unfortunately gpb doesn’t support groups as stated on the features section of the README.md And as groups are a deprecated feature of protobufs, I doubt gpb is going to be adding support for them, see: https://developers.google.com/protocol-buffers/docs/proto#groups

Not 100% sure, but the suggested workaround described in the google documentation is converting that group into separate message, something like this:

message BookDetails {
  ....
  repeated Identifier identifier = 18;
  ....
}
message Identifier {
  optional int32 type = 19;
  optional string identifier = 20;
}
6

Doesn’t work for me. Now I get this when trying to decode

** (FunctionClauseError) no function clause matching in :gpb.decode_wiretype/1
    (gpb) src/gpb.erl:253: :gpb.decode_wiretype(3)
    (gpb) src/gpb.erl:259: :gpb.skip_field/2
    (gpb) src/gpb.erl:217: :gpb.decode_field/4
    (gpb) src/gpb.erl:348: :gpb.decode_type/3
    (gpb) src/gpb.erl:212: :gpb.decode_field/4
    (gpb) src/gpb.erl:348: :gpb.decode_type/3
    (gpb) src/gpb.erl:212: :gpb.decode_field/4
    (gpb) src/gpb.erl:348: :gpb.decode_type/3