Aggregating Data in mnesia and saving a snapshot with potentially million rows.


I am trying to make a selfie contest where users would be able to post their pictures (only one) online and other users would be able to upvote/like them. I am expecting the number of users to be 0.1 million.

For the above load I believe mnesia would be a good fit. I am planning to store each vote as a record in mnesia. I am not planning to scale horizontally but use only one node.
One requirement is to show popular pictures to users. For this i would need to aggregate the votes per picture. With potentially millions of rows of votes this operation is going to be expensive.

Can I use Flow to perform this aggregation.
If yes

  • How do i stream data from mnesia into Flow
  • How do i take a snapshot of calculation and put it back into mnesia and start after the snapshot in the next aggregation cycle.
  • Once I have the aggregated data can I sort it using Flow or can i do it using Enum.sort_by . But this dataset might also be large (potentially as large as the number of users).
    If no
    Please advise.

Also would using Ecto andPostgres be able to handle this load. I am relatively comfortable with Ecto and Postgres but am skeptical about the load.

I am new to elixir and am very excited about it. I am thankful to community for being of so much help. Your advise in the above matter would be very helpful.

1 Like