How to Decode Protobuf/gRPC messages when I have only a JSON Schema available?

Questions / Help
,
1

Hey everyone!

I’m successfully receiving events from Salesforce’s PubSub API via gRPC, but I’m struggling to decode the actual payload.

Since Salesforce allows schema customization, they don’t provide predefined Protobuf definitions for the event data. Instead, events are wrapped in a generic message, containing:

  • A schema_id (e.g., “23456789asdf”)
  • A binary payload, which needs decoding

I can fetch the corresponding JSON Schema using:

Eventbus.V1.PubSub.Stub.get_schema(
  channel, 
  %Eventbus.V1.SchemaRequest{schema_id: "23456789asdf"}
)

I would love to have a flow like the following:

  1. :white_check_mark: Store known schemas (no problem)
  2. :white_check_mark: Check if a schema is known on receiving an event (no problem)
  3. :white_check_mark: Fetch missing schemas (no problem)
  4. :question: Compile missing schemas
  5. :question:Decode the payload using the schema

I’m stuck on how to go from a JSON Schema to something usable for decoding the Protobuf payload . Any guidance would be greatly appreciated! :+1:

2

Salesforce uses Avro Schemas: Event Data Serialization with Apache Avro | Use Pub/Sub API | Pub/Sub API | Salesforce Developers

So this library holds the answer: GitHub - beam-community/avro_ex: An Avro Library that emphasizes testability and ease of use.

{:ok, %{schema_json: json_schema}} = Eventbus.V1.PubSub.Stub.get_schema(
  channel, 
  %Eventbus.V1.SchemaRequest{schema_id: "23456789asdf"}
)

{:ok, avro_schema} = AvroEx.decode_schema(json_schema)

{:ok, payload} = AvroEx.decode(avro_schema, event.payload)
4 Likes