Sharding of ets table

How many users do you have? What are your access patterns? What problem are you running into with a single ets table?

it is just an assumption, do you know, discord also store their guild members in ets table, and when it starts to too much like 250k then he have distribute into small chunk of tables. now i am asking same scenaruo what if users start to grow then it is possible to create small small table of same ets table?

Good question.

I don’t know but here are some inspirations for a beginning of an answer:

• Transparent and out-of-box sharding support for ETS tables in Erlang/Elixir. | Carlos Andres Bolaños
• GitHub - cabol/shards: Partitioned ETS tables for Erlang and Elixir · GitHub

Registry also uses sharded ets tables for scaling purposes.

yay, Registry or GenRegistry also create local ets table in each node if we start it in main supervisor. but here i am telling about a chat group like discord which handle that much guild members in each ets table. i am talking about breaking the big ets table into small small ets table in same node by sharding not in different node.

is it possible? i know it is possible but how?

ets sharding is used not to increase size, it is used to increase speed of parallel lookups/updates to reduce lock contention. Do not use ets sharding unless you have a reason to do so.

i saying the same thing i want to increase the speed of parallel lookup because hypothetically if one ets table contains 250k users data which contains many complex fields then each time if i have to lookup or update then it will become problematic. so, i want to use sharding for breaking ets table(250k users) into small small ets table in same node.

Did you measure your lock contention on ets table? Did you measure your access time? Did you optimize your access patterns? Did you tune *_concurrency flags? Unless you do all of this, you have no reason to shard tables. ETS is build with concurrency in mind and it has pretty smart lock system which allows parallel access to separate chunks of the table, so sharding is required only when you have some exotic access pattern and you can guarantee on your level that the chunks you’ll split the table into will increase performance

This question doesn’t really work as a hypothetical. Sharping ets is a tactical performance optimization, the details have to be married to how it is actually used.

i measure acess time and read_concurrency and it gets slower when more process like 50k process tried to connect 1 genserver for lookup from 1 ets table which contains 250k users. by using update_counte to limit number then aldo it gets slower and slower.
now i think i have no option but to use sharding, what do you think?