I need to build an internal app, and I think I can finally use Elixir for this which I’ve been waiting to do for a long time.
Unfortunately I forgot all of what I learned because I never used it, so I was wondering if someone could help me outline a basic structure of how I would build this:
The basic purpose of this server is to receive batches of tokens to send push messages to, along with the message contents. That should add them to some sort of ‘waiting’ queue. Then I want 100K workers who will churn through this queue, hitting a 3rd party HTTP API endpoint (1 API request per 1 message)
So, I need to send 10M pushes 4 times a day, so the incoming requests to elixir might look like this:
/send/batch (max limit of say… 5000 messages per request)
[["111", "push message 1"], ["222", "push message 2"], ... 4,998 more ]
[["333", "push message 3"], ["444", "push message 4"], ... 4,998 more ]
[["555", "push message 5"], ["666", "push message 6"], ... 4,998 more ]
... all the way until 10 million
So thats incoming messages that need to get sent, along with the token it gets sent to. That is what has the ‘backlog’
Then I need 100K workers (or some defined # of workers)
to make an HTTP request asyncronously in its own green thread (1 thread per message, so 100K HTTP requests happening at once, at all times):
POST to 3rd party endpoint: https://example.com/sass/send-message
{body: {token: 'aaa', body: 'test message 1'}}
… if HTTP request fails, retry in its own thread using exponential backoff (this isnt really required if its a huge pain… maybe it can just wait a random amount of time or something to make it easy).
So the most that needs to be in the backlog would be about 10M.
Each token + message is about 200 bytes, so 10 million messages in the queue is about 2GB in memory.
I’m not sure really how to go about doing this , or what libraries to use (HTTPoison I suppose). I could use phoenix to do the core API portion of it… or maybe I need cowboy only (it’s a very simple app, it doesnt need other endpoints)
Do I build a “genserver” or?
