alexgleason
Pre-generating user feeds in a Twitter-like microblogging platform
Hey guys, I’m developing Pleroma and preparing it to scale. Of course the bottleneck is not Elixir, but the Postgres database.
It runs an SQL query to generate each user’s home feed, and this query is expensive and slow.
In the Ruby world, Mastodon solves this problem with Redis. A background worker pre-generates feeds for each user and stores them in memory, ready to be served at any time.
I’m wondering what your opinions are on options in Elixir. Is it reasonable to use ETS to solve this problem?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 2 of 2 Posts
dimitarvp
It definitely is but it really depends on the shape of the data and how quickly does it expire and it’s especially dependent on how well can you segment this data – I mean in terms of unique caching keys.
Have you looked at
Cachex?If you are thinking about having multi-node caches I feel you’d be going against the wind here and maybe should just use Redis indeed.
gpopides
When you say that the query is expensive/slow, to me it seems that you have identified your problem and it’s in the database. Perhaps improving the query it self could solve the problem? Maybe using database views? Trying to optimize the application layer with caches before looking for a solution for the problem itself feels like you will encounter more problems with invalidating caches and so on.