How to insert 75,000 incoming messages in a database

Phoenix Forum Questions / Help
, ,
1

Hi!

I’m a student and I decided to use Elixir (and Phoenix) for a project.

Some context
I have to receive between 7,500 and 75,000 messages per second (containing some data, like a few floats and integers).

For now, my classmate and I decided to use Phoenix’s channels to receive them and put them in a database.

The problem is I’m not quite sure on how to achieve that, because I’m still learning Elixir (and computer sciences) and I discovered Phoenix like a month ago.

For the database (PostGreSQL), I think I can manage it, but this will make us use multiple inserts of ~100’s rows each;

My solution
Actually I thought about a distributed phoenix service, where each receiver, on incoming message, will use a process from Poolboy to put the data in a list, and when the list size reach 100 (or with some kind of timer in case we don’t receive message anymore) the process will do the inserts and then clear the list.

Is there any better way, or at least, some optimisations i can do to achieve this goal?

Thanks for your reading !

EDIT : Added some separator

2 Likes
2

Hello and welcome,

What You might want is a back pressure mecanism… You might have a look at GenStage, Flow or Broadway.

It is like putting the events in a buffer queue before saving to db. You achieve isolation.

You can get concurrency by specifying the number of producer/consumer.

Also, when saving to db, You can use insert_all to batch saving. For example, saving 10’000 messages in one db call. Instead of 10’000 calls saving one record :slight_smile:

4 Likes