JILL24
Sharding of ets table
suppose a case where single ets is storing users and users start to grow too much in ets table. now i want to shard ets table in the same node means local node.
is it possible? and if it is possible, how?
note - Don’t use 3rd party library. just custom built.
Trending in Discussions
Other Trending Topics
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First Post!
benwilson512
How many users do you have? What are your access patterns? What problem are you running into with a single ets table?
Most Liked
jhogberg
I think you should take a step back and ask yourself why you’re having this problem to begin with.
Do you require that both reads and updates are executed sequentially?
If you don’t require that at all, then the server process is pointless and you’re much better off letting the clients access the table directly, which will drastically improve performance.
If you only require that updates run sequentially, then the server process is only necessary for updates and reads can access the table directly, which will greatly improve performance.
If you require that both reads and updates run sequentially, then I’d love to hear why. Either way you need to take a long hard look at your application’s architecture as routing this much traffic through a single process is not normal.
The concurrency options generally make the table faster for concurrent operations at the cost of making each individual operation slower. If there are no concurrent operations, such as in your case with a single server process being the only one to touch the table, then they make things slower for nothing.
LostKobrakai
Registryalso uses sharded ets tables for scaling purposes.hst337
etssharding is used not to increase size, it is used to increase speed of parallel lookups/updates to reduce lock contention. Do not use ets sharding unless you have a reason to do so.Last Post!
Nicoo
You received to much XY responses. Sounds like you just desperate to get the answer to your question.
One solution (maybe not ideal) is to make a logical separation.
Example:
Define a field “data_table” in the “user” table.
Let’s imagine that the table you want to shard is “user_data”.
You have 4 tables (shards) of “user_data”: user_data1, user_data2, user_data3, user_data4.
For each user, you can assign a shard to
user.data_tablefield (user.data_table = "user_data1").This approach will improve some cases, surely make others worse. It all depends on your case.
In a multi-tenant logic, it can do it.
This is not the ideal answer to your question but just to advance in the reflection for sharding