Our company has a running production app that needs to send over 800,000 push notifications for one user action.
For example, one famous person posted a content → send push to all users subscribed to that person.
The current implementation is in Python that would take up to one minute delay from the first user to receive the push vs the last user. Upon researching how to optimize this situation, I came across Elixir.
Currently, I’m in the process of learning the basics of the language and I like it a lot (coming from Python & JS). The community is more open to others.
Real question is, would utilizing Elixir reduce delay up to 3 seconds (business requirement)?
Is Phoenix the right framework?
I would need an endpoint that receives a signal(POST) from our Python backend, then process the push notifications in Elixir.
Increasing the number of pods and DB scalability wouldn’t matter much as we can scale it decently within our budget. However, I also think that the beauty of Elixir is that it can run thousands of processes without a problem, so ideally, I won’t have to increase the number of pods.
How are you currently pushing the notifications out to your users?
Take us through the high-level steps to see how the process could be made more concurrent in Elixir.
Also, welcome to the community.
And to answer your main question, Elixir would be a great choice to bring the total time down to < 3s with a combination of batched workers across a number of connected nodes, exactly how depends on the specific guarantees you need at each part of the system.
Are you using the Python AdminSDK API to send each subscribed user an individual message or are the 500 tokens sent with the MulticastMessage feature?
Have you tried creating an FCM publisher topic per user, subscribe each interested client to the topic, and then you send 1 message to the famous person’s publisher topic and let FCM deal with pushing it to the configured subscribers.
For 800,000 messages you’re probably running into FCM’s fanout throttling limits, in which case no backend system will help you and it would be best to stay with Python for now and let management know that the bottleneck is on FCM’s side. They may have enterprise options that can increase those limits at a cost.
However if you want to replace the system completely with your own custom Phoenix application and things like the JavaScript WebPush API, you will definitely be able to shrink the 1min delay by at least a few orders of magnitude at the cost of having to develop and maintain your own system.
From the looks of it, it doesn’t seem like we’re using Topics.
I’m in the position to refactor so I’m not 100% sure why the original implementation did not use Topics.
I’ll have to ask tomorrow. If Topics can be used, it’d be better utilizing that than adding Elixir micro service (although I personally want to)
I added more context above, but with Phoenix Channels, Oban, and some pretty simple WebPush service workers you should be able to implement a PoC parallel system quite quickly to test out the throughout for a single server.
If you have complete autonomy then why not? Just depends on the expected longevity of the system, whether others will be able to maintain, refractor, and extend the Elixir part of things, budget and time constraints etc.
I think we might have to do some ground work regardless of the language.
Since we’re using React Native for mobile, it seems like using FCM is unavoidable.
However, in the long run, it might be necessary to build our own system, the delivery speed is crucial for our application and users have been complaining about it
Since you’re using React Native then it shouldn’t be very hard to create a client-side notification component that interfaces directly with an Elixir microservice through Phoenix Channels, either through the bundled JavaScript library, or platform-specific implementations SwiftPhoenixClient on CocoaPods.org and JavaPhoenixChannels.
Then the decision of how to implement the pub/sub topic notification architecture between Python and Elixir is up to you. Through REST routes with a Phoenix.Endpoint or leveraging the same Channels communication strategy directly from Python with phxsocket · PyPI, while you slowly replace each backend feature with a shiny Elixir one.
