Protobuf NaiveDateTime type

Questions / Help
1

Hello,

what would be the best type for proto3 to handle NaiveDateTime from Elixir types.

message PostResponse {
    string content = 1;
    string id = 2;
    google.protobuf.Timestamp inserted_at = 3;
    google.protobuf.Timestamp updated_at = 4;
    string user_id = 5;
}

I think Timestamp is not the best one, I want to have the NativeDateTime type value as it is.
any help would be appreciated.

2

Well, NaiveDateTime is just a struct / map after all:

iex(1)> NaiveDateTime.now() |> Map.keys()
[:__struct__, :calendar, :day, :hour, :microsecond, :minute, :month, :second,
 :year]

The calendar field is usually the module/atom Calendar.ISO so you can strip it if you don’t deal with other calendars, and the special __struct__ field, and then end up with this:

iex(2)> NaiveDateTime.utc_now() |> Map.from_struct() |> Map.drop([:__struct__, :calendar])
%{
  day: 10,
  hour: 12,
  microsecond: {981284, 6},
  minute: 32,
  month: 3,
  second: 40,
  year: 2021
}

You can then also remove the decimal precision value from microsecond by using elem(..., 0) and then you’ll just have seven unsigned 32-bit integers (which you can encode with int32 since it uses variable-length encoding over the wire). That resulting map of seven integers can then very easily be described as a nested Protobuf structure afterwards.

Of course, you can also convert the NaiveDateTime to UNIX seconds / milliseconds / microseconds / nanoseconds and just encode the whole thing as a 64-bit unsigned integer:

naive_date_time_from_another_function
|> DateTime.from_naive!("Etc/UTC") # assuming your times are in UTC
|> DateTime.to_unix(:microsecond) # or :second, or :millisecond, or :nanosecond)
3

Thanks for the response @dimitarvp, this really helps, this way I can do it after fetching the data from DB and updated columns before passing it to PostResponse.new(), and works for limited data sets.

But the thing I have nested fields inside a list of a map and the list goes to 1000’s, is there a way where I could set this parser and a specified type globally so that can be applied all the timestamps field.

4

Well, Elixir is not one of the fastest languages (although it’s one of the fastest dynamic languages!) but without having sample data and some of your code, I can’t help further.

If you are feeling up for it, you can make a GitHub project with sample code and something like 3000-5000 of these records and we can both test implementations and benchmark. I’d like it and will help you with it.

But as for this:

…I am not sure I follow what you are saying. Can you clarify?

5

Hey @dimitarvp

I really appreciate your time for this, thanks for your response.

Here is what I meant,

message PostResponse {
    bool active = 1;
    bool anonymous = 2;
    string content = 3;
    repeated string group_id = 4;
    string id = 5;
    string user_id = 6;
    repeated MediaResponse media = 7;
    PollsResponse polls = 8;
    bool sensitive = 9;
    google.protobuf.Timestamp inserted_at = 10;
    google.protobuf.Timestamp updated_at = 11;
}

message MediaResponse {
    string src = 1;
    enum MediaType {
        IMAGE = 0;
        VIDEO = 1;
      }
    MediaType type = 2;
    int32 order = 3;
    string caption = 4;
    bool active = 5;
    string post_id = 6;
    google.protobuf.Timestamp inserted_at = 7;
    google.protobuf.Timestamp updated_at = 8;
}

message PollsResponse {
    bool active = 1;
    string id = 2;
    repeated PollOptionsResponse poll_options = 3;
    string post_id = 4;
    string summary = 5;
    string title = 6;
    google.protobuf.Timestamp start_date = 7;
    google.protobuf.Timestamp end_date = 8;
    google.protobuf.Timestamp inserted_at = 9;
    google.protobuf.Timestamp updated_at = 10;
}

message PollOptionsResponse {
    bool active = 1;
    string context = 2;
    string id = 3;
    string polls_id = 4;
    repeated PollVotesResponse poll_votes = 5;
    google.protobuf.Timestamp inserted_at = 6;
    google.protobuf.Timestamp updated_at = 7; 
}

message PollVotesResponse {
    string id = 1;
    bool active = 2; 
    string user_id = 3; 
    string context = 4; 
    string poll_options_id = 5;
    google.protobuf.Timestamp inserted_at = 6;
    google.protobuf.Timestamp updated_at = 7; 
}

and In elixir, the response looks like

%TwitterProto.PostResponse{
  active: true,
  anonymous: false,
  content: "Some content in here",
  group_id: ["547bd541-880b-4bd0-8b41-3e19e23f3064",
   "c2edaa11-1915-4212-8bf4-3fba1ca286a3"],
  id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902",
  media: [
    %Twitter.MediaResponse{
      active: true,
      caption: "",
      order: 1,
      post_id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902",
      src: "",
      type: :IMAGE
    },
    %Twitter.MediaResponse{
      active: true,
      caption: "",
      order: 2,
      post_id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902",
      src: "",
      type: :IMAGE
    }
  ],
  polls: %TwitterProto.PollsResponse{
    active: true,
    id: "2843684e-1d00-4208-a88d-ed980071d807",
    poll_options: [
      %TwitterProto.PollOptionsResponse{
        active: true,
        context: "Yes",
        id: "b7c20db7-09d2-4087-a55d-32bc356aa2d3",
        poll_votes: [],
        polls_id: "2843684e-1d00-4208-a88d-ed980071d807"
      },
      %TwitterProto.PollOptionsResponse{
        active: true,
        context: "No",
        id: "cefe92a1-a52e-436c-a48d-7da84af81fc4",
        poll_votes: [],
        polls_id: "2843684e-1d00-4208-a88d-ed980071d807"
      },
      %TwitterProto.PollOptionsResponse{
        active: true,
        context: "Maybe",
        id: "41e727ed-2c55-4d8c-a2bd-fb4fe66c5f3c",
        poll_votes: [
            %{}, %{}, %{} .... 10_00_0000
        ],
        polls_id: "2843684e-1d00-4208-a88d-ed980071d807"
      }
    ],
    post_id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902", 
    summary: "",
    title: "Are you willing to join ARMY??"
  },
  sensitive: false,
  user_id: "ad29e49a-5e61-43cf-ab85-438f2ee9fb16"
}

the above response will have multiple nested maps with timestamp() fields, so if I parse after fetching the data from DB, I have to loop through all the nested maps, and also I feel like it won’t be generic. Is there a way or any suggestion to achieve this in a generic way??

6

Yeah, doesn’t seem like it right now.

Which library are you using to [de]serialize the data?

7

I am using elixir-grpc/grpc and google_protos

8

Cool. So will you make a small GitHub project – with some sample data included – so we can iterate on this?

9

For sure, I will create Repo for this, will get back to you.