I’m looking for thoughts on how I could prototype a leaderboard service with elixir, (+phoenix? +genstage?). It would have many concurrent leaderboards, each one with a couple hundreds competing users. Each user would update the leaderboard every second for 20 minutes with his score, and get an updated ranking of all players assigned to this leaderboard in response.
It feels like a compelling case to introduce elixir in my company and show a quick, fast and elegant alternative to our usual java/memcache/aws toolbox.
It feels like websockets for client server communications could be a good fit. Having no experience with GenStage, I also wonder if this would be a good fit. My initial impression is that it’s tailored to process a large amount of data like what my analytics team would do.
Bonus point if the solution doesn’t need a cache. Extra bonus point if it doesn’t need a database (without making the whole thing more complex of course).
Each leaderboard can be a supervised process, so that when you update a user score, you can recalculate concurrently (i.e. one leaderboard calculation doesn’t block the other ones).
Each leaderboard data can be stored in a dedicated ets table, which would give you great performance for concurrent reads.
Apart from Phoenix, you could do all of this with Elixir 1.4 standard library tools, which I think it’s a very compelling argument.
Yes, I’d say that one of the most compelling points would be that in the end, it could be packaged to a simple release which can be put and run on any webserver without needing any external dependencies: Everything is inside the BEAM. And for some real mind-blowingness you might start a second webserver and connect their BEAM nodes together, and then show them what happens if you shutdown one webserver violently, bring it back, then shutdown the other webserver violently and bring it back.
For leaderboards, ets could be used, or even a simple list of {score, name}-tuples.
My practical introduction to Concurrency in Elixir presentation actually uses leaderboards as an example to explain what is going on with the Actor Model behind the scenes.
I agree with @cloud8421 that GenStage is not something you need here. You don’t have a waterfall of data that comes in which needs to be partitioned and redirected. You have very compartimentalized high scores which follow a very simple and restricted way of updating leaderboards.
Thanks @cloud8421@Qqwy and @jur0 for the very helpful messages or links. This last stackoverflow thread is really comforting me regarding the performance which could be achieved.
Now I worry a bit about the resilience of a such a system. On one hand one leaderboard crashing with N users won’t affect the rest of the users. On the other hand, I now lost the leaderboard data of these N users and they won’t be happy at all. The customer support team will not like this situation either.
Last but not least, a machine crashing or deploying a new version of the service would loose all the scores and rankings. What’s the Erlang/Elixir approach to deal with this problem compared to a stateless service with a NoSQL datastore like AWS Dynamo?
You can regularly dump a leaderboard data to an external data storage if needed and restore from there on node startup. This would allow you to control the write pace (e.g. you can queue writes).
