I have an application with predictably variable workloads. Hundreds of thousands of requests each evening, and no activity from after midnight through around 2PM.
To make matters worse, the application is stateful – making extensive use of GenServers, and it uses WebSockets.
Right now I’m paying for the full resources required to scale all day – however, I know that if this were a stateless service that didn’t need sockets (basically if it were structured the way I would have structured it if it were a NodeJS app), I could put it behind a dynamic load balancer or use a service like Google App Engine for auto-scaling, or use Lambda/GCloud Functions/Azure functions.
What is the best thing to do in this circumstance for an Elixir app?
If you don’t want to refactor your application to remove some of the statefulness, you could make use of Elixir’s distribution featues.
If your application can handle keeping the Websockets alive with fewer resources, you could spin up a second node with more resources and make them take care of the requests.
Yeah, the application is already running on multiple distributed nodes – I’m wondering what type of service and/or library would enable me to dynamically add and remove nodes in response to increases in traffic.
I played with an idea of doing it from within the applications using the provider’s API to create / delete virtual machines. I didn’t really encounter any major problems with it, so maybe you could do something similar.
When some metric (resource utilization or users online count per node) goes above some threshold, send an API request to your provider to spin up an additional vm, and when that same metric goes below, do the opposite – destroy one of the nodes.
Since that scaling logic (thresholds) is written in an actual programming language, it provides enough flexibility to do almost whatever you want …
