Protobuf NaiveDateTime type

Well, NaiveDateTime is just a struct / map after all:

iex(1)> NaiveDateTime.now() |> Map.keys()
[:__struct__, :calendar, :day, :hour, :microsecond, :minute, :month, :second,
 :year]

The calendar field is usually the module/atom Calendar.ISO so you can strip it if you don’t deal with other calendars, and the special __struct__ field, and then end up with this:

iex(2)> NaiveDateTime.utc_now() |> Map.from_struct() |> Map.drop([:__struct__, :calendar])
%{
  day: 10,
  hour: 12,
  microsecond: {981284, 6},
  minute: 32,
  month: 3,
  second: 40,
  year: 2021
}

You can then also remove the decimal precision value from microsecond by using elem(..., 0) and then you’ll just have seven unsigned 32-bit integers (which you can encode with int32 since it uses variable-length encoding over the wire). That resulting map of seven integers can then very easily be described as a nested Protobuf structure afterwards.

Of course, you can also convert the NaiveDateTime to UNIX seconds / milliseconds / microseconds / nanoseconds and just encode the whole thing as a 64-bit unsigned integer:

naive_date_time_from_another_function
|> DateTime.from_naive!("Etc/UTC") # assuming your times are in UTC
|> DateTime.to_unix(:microsecond) # or :second, or :millisecond, or :nanosecond)

Thanks for the response @dimitarvp, this really helps, this way I can do it after fetching the data from DB and updated columns before passing it to PostResponse.new(), and works for limited data sets.

But the thing I have nested fields inside a list of a map and the list goes to 1000’s, is there a way where I could set this parser and a specified type globally so that can be applied all the timestamps field.

Well, Elixir is not one of the fastest languages (although it’s one of the fastest dynamic languages!) but without having sample data and some of your code, I can’t help further.

If you are feeling up for it, you can make a GitHub project with sample code and something like 3000-5000 of these records and we can both test implementations and benchmark. I’d like it and will help you with it.

But as for this:

…I am not sure I follow what you are saying. Can you clarify?

Hey @dimitarvp

I really appreciate your time for this, thanks for your response.

Here is what I meant,

message PostResponse {
    bool active = 1;
    bool anonymous = 2;
    string content = 3;
    repeated string group_id = 4;
    string id = 5;
    string user_id = 6;
    repeated MediaResponse media = 7;
    PollsResponse polls = 8;
    bool sensitive = 9;
    google.protobuf.Timestamp inserted_at = 10;
    google.protobuf.Timestamp updated_at = 11;
}

message MediaResponse {
    string src = 1;
    enum MediaType {
        IMAGE = 0;
        VIDEO = 1;
      }
    MediaType type = 2;
    int32 order = 3;
    string caption = 4;
    bool active = 5;
    string post_id = 6;
    google.protobuf.Timestamp inserted_at = 7;
    google.protobuf.Timestamp updated_at = 8;
}

message PollsResponse {
    bool active = 1;
    string id = 2;
    repeated PollOptionsResponse poll_options = 3;
    string post_id = 4;
    string summary = 5;
    string title = 6;
    google.protobuf.Timestamp start_date = 7;
    google.protobuf.Timestamp end_date = 8;
    google.protobuf.Timestamp inserted_at = 9;
    google.protobuf.Timestamp updated_at = 10;
}

message PollOptionsResponse {
    bool active = 1;
    string context = 2;
    string id = 3;
    string polls_id = 4;
    repeated PollVotesResponse poll_votes = 5;
    google.protobuf.Timestamp inserted_at = 6;
    google.protobuf.Timestamp updated_at = 7; 
}

message PollVotesResponse {
    string id = 1;
    bool active = 2; 
    string user_id = 3; 
    string context = 4; 
    string poll_options_id = 5;
    google.protobuf.Timestamp inserted_at = 6;
    google.protobuf.Timestamp updated_at = 7; 
}

and In elixir, the response looks like

%TwitterProto.PostResponse{
  active: true,
  anonymous: false,
  content: "Some content in here",
  group_id: ["547bd541-880b-4bd0-8b41-3e19e23f3064",
   "c2edaa11-1915-4212-8bf4-3fba1ca286a3"],
  id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902",
  media: [
    %Twitter.MediaResponse{
      active: true,
      caption: "",
      order: 1,
      post_id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902",
      src: "",
      type: :IMAGE
    },
    %Twitter.MediaResponse{
      active: true,
      caption: "",
      order: 2,
      post_id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902",
      src: "",
      type: :IMAGE
    }
  ],
  polls: %TwitterProto.PollsResponse{
    active: true,
    id: "2843684e-1d00-4208-a88d-ed980071d807",
    poll_options: [
      %TwitterProto.PollOptionsResponse{
        active: true,
        context: "Yes",
        id: "b7c20db7-09d2-4087-a55d-32bc356aa2d3",
        poll_votes: [],
        polls_id: "2843684e-1d00-4208-a88d-ed980071d807"
      },
      %TwitterProto.PollOptionsResponse{
        active: true,
        context: "No",
        id: "cefe92a1-a52e-436c-a48d-7da84af81fc4",
        poll_votes: [],
        polls_id: "2843684e-1d00-4208-a88d-ed980071d807"
      },
      %TwitterProto.PollOptionsResponse{
        active: true,
        context: "Maybe",
        id: "41e727ed-2c55-4d8c-a2bd-fb4fe66c5f3c",
        poll_votes: [
            %{}, %{}, %{} .... 10_00_0000
        ],
        polls_id: "2843684e-1d00-4208-a88d-ed980071d807"
      }
    ],
    post_id: "65dfc815-62f3-4a8e-b31d-05b87ea0e902", 
    summary: "",
    title: "Are you willing to join ARMY??"
  },
  sensitive: false,
  user_id: "ad29e49a-5e61-43cf-ab85-438f2ee9fb16"
}

the above response will have multiple nested maps with timestamp() fields, so if I parse after fetching the data from DB, I have to loop through all the nested maps, and also I feel like it won’t be generic. Is there a way or any suggestion to achieve this in a generic way??

Yeah, doesn’t seem like it right now.

Which library are you using to [de]serialize the data?

I am using elixir-grpc/grpc and google_protos

Cool. So will you make a small GitHub project – with some sample data included – so we can iterate on this?

For sure, I will create Repo for this, will get back to you.