I’ve a local subscriber database with about 2million records, I will connect to 4 remote servers with a TPS of 5 on each. These servers contain real time subscriber data usage. I need to monitor subscriber data usage and notify them when the usage reaches a certain threshold. Below is how I plan to execute this task.
- Create 20 worker processes using poolboy and maintain a MINT connection in these processes
- A process to fetch the records from the database (more processes to fetch records asynchronously)
- Send the records across to worker processes.
Kindly advise if this is the most efficient way to implement the notification server.
Well you don’t mention where the data is stored that you are accessing, but if it were in, say, PostgreSQL, then I’d write a trigger to send a NOTIFY message to a listening Elixir process from the PostgreSQL side when the data usage has hit whatever you are watching for.
Access to the remote servers where the real time data usage resides is via an api call, I only have a local database with the subscription details.
Ooo that sounds painful… I hope you can batch receive as much data as possible on each individual API request! Assuming the API has some kind of streaming output like any good GraphQL API or so then a single connection would be perfectly sufficient. I’d probably still do the data processing for it in a local database with a NOTIFY call back to elixir when it’s time to notify someone.
Unfortunately I can’t batch the call, more of a legacy upstream server, however I’ve a TPS of 5 on 4 servers - meaning I can process about 20 records per second. I want the most effective way to implement this.
Well you are going to be throttled by the remote server regardless, so however many tasks you can spin up to request data before the server starts slowing them all down due to it being overloaded would be the most ‘efficient’, and that you could only figure out by testing it. ^.^;
1 Like
