Are you routing all ets table access through a gen server or are the client pods allowed to access the ets table directly?
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
JILL24
client sending request to genserver but i know if this much try to send request to 1 genserver then it’s queue will explode so i am using update_counter to limit client to send request at time. but problem lies here when ets table grow to 250k(i am giving a number but actually it is 1 million) users details as tuple. then it is very slow to lookup if i set read_concurrent true then also it is very slow to lookup or update.
0
jhogberg
Erlang Core Team
I think you should take a step back and ask yourself why you’re having this problem to begin with.
Do you require that both reads and updates are executed sequentially?
If you don’t require that at all, then the server process is pointless and you’re much better off letting the clients access the table directly, which will drastically improve performance.
If you only require that updates run sequentially, then the server process is only necessary for updates and reads can access the table directly, which will greatly improve performance.
If you require that both reads and updates run sequentially, then I’d love to hear why. Either way you need to take a long hard look at your application’s architecture as routing this much traffic through a single process is not normal.
The concurrency options generally make the table faster for concurrent operations at the cost of making each individual operation slower. If there are no concurrent operations, such as in your case with a single server process being the only one to touch the table, then they make things slower for nothing.
3
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
Are you conducting a synthetic benchmark or do you actually have a real application with 1 million users?
0
JILL24
here we are talking about distributed system.
suppose here A is client genserver and B is ets table genserver.
we require B because for B ets table is in local node and he can retreive data but if i allow all client to direct acess table by using :rpc.call then it will increase overhead. so for that i am using two genserver and rereive data or update data.
0
JILL24
is it possible to do sharding of one ets table(which contains 1 million users data in tuple) into multiple small small table in same node?
0
jhogberg
Erlang Core Team
Again, please take a step back and ask yourself: why have you designed the application this way?
0
JILL24
yes it is only way to handle this much traffic, it have many backend logic like hash ring etc.
my only simple question is this
Q. is it possible to sharding one big ets table into smaller smaller ets table in same node? if yes then how?
0
jhogberg
Erlang Core Team
I doubt that, but I’m not going to argue.
@Nicoo already posted a link that should be more than enough to get you started, so I don’t see what you’re on about. We’re just trying to help you.
0
JILL24
you are correct, this link can help me to create custom sharding for ets table.
thank you.
0