I’m a student and I decided to use Elixir (and Phoenix) for a project.
I have to receive between 7,500 and 75,000 messages per second (containing some data, like a few floats and integers).
For now, my classmate and I decided to use Phoenix’s channels to receive them and put them in a database.
The problem is I’m not quite sure on how to achieve that, because I’m still learning Elixir (and computer sciences) and I discovered Phoenix like a month ago.
For the database (PostGreSQL), I think I can manage it, but this will make us use multiple inserts of ~100’s rows each;
Actually I thought about a distributed phoenix service, where each receiver, on incoming message, will use a process from Poolboy to put the data in a list, and when the list size reach 100 (or with some kind of timer in case we don’t receive message anymore) the process will do the inserts and then clear the list.
Is there any better way, or at least, some optimisations i can do to achieve this goal?
Thanks for your reading !
EDIT : Added some separator