Hey guys, I’m developing Pleroma and preparing it to scale. Of course the bottleneck is not Elixir, but the Postgres database.
It runs an SQL query to generate each user’s home feed, and this query is expensive and slow.
In the Ruby world, Mastodon solves this problem with Redis. A background worker pre-generates feeds for each user and stores them in memory, ready to be served at any time.
I’m wondering what your opinions are on options in Elixir. Is it reasonable to use ETS to solve this problem?
1 Like
It definitely is but it really depends on the shape of the data and how quickly does it expire and it’s especially dependent on how well can you segment this data – I mean in terms of unique caching keys.
Have you looked at Cachex?
If you are thinking about having multi-node caches I feel you’d be going against the wind here and maybe should just use Redis indeed.
When you say that the query is expensive/slow, to me it seems that you have identified your problem and it’s in the database. Perhaps improving the query it self could solve the problem? Maybe using database views? Trying to optimize the application layer with caches before looking for a solution for the problem itself feels like you will encounter more problems with invalidating caches and so on.
