I’m writing and app that will broadcast sport events during a match with Phoenix channels to many users, users can react to the event and create a response to this event that will be stored in the DB, I think the database might not handle the inserts and be the bottleneck, I’m thinking of using GenStage for this problem, I would like some advice to solve this problem.
How many thousands of writes per second do you imagine you’ll have? Postgres trivially writes 1000+ per second and can be made to handle much more than that. A simple gen_server that batches these reactions and does inserts with Ecto.Repo.insert_all might help, though I have no practical knowledge of how batching with insert_all helps with performance.
I thought about using a GenServer but I will need between 20000 to 40000 or more inserts per second, I can’t use batch insert since each user will select different data for an event.
Curious to hear the answers as well. I was planning to use Elixir to build a payment gateway which will process millions of payments on a single day.
Maybe timescaledb could be useful for your use case; its an extension to Postgres. The whitepaper suggests ingestion rates well above your requirement is ok and more-or-less linear as the database grows. See pages 5 and 6.
1 Like
Really nice, just using timescaledb would be more than enough for the amount of writes that I require, thanks for pointing the pages on the white paper
