Events - checking nested maps for match

Questions / Help
1

Each user would have their own genserver holding their entire state(loaded from db)

%{
  tree_one: %{
    keyword_one: %{
      domain: "elixirocks.com",
      twitter: "some_lame_twitter_link.com",
      score: 5,
    }
  }
}

There would be some 20+ trees and up to 10000 keywords total inside these “trees”.

Now let’s suppose some X process in the system gets updated score for our elixirocks.com domain, what would be the most efficient way to update it inside all the genservers that contain that domain?

Does it make sense to broadcast something like “topic:domain_update” + listen with all genservers and then check inside maps(state) for a match ? I am afraid how this will perform if I have say 1000 genservers(users) and each has some 10k entries and there is 1 update every 5 seconds :slight_smile:

Those genservers would then be responsible for broadcasting this changes further to the users via phoenix channels…

Reading last 2 paragraphs back, makes me cringe a little… Maybe just ditch this design for shared state (ets/mnesia) or even just stick with damn postgres(trying to avoid this and have elixir processes be main source of truth) :slight_smile: ?

2

The best strategy is to only have data at the location where you need it. In your example, it seems like you might be copying the information for all websites across all processes (one per user). Maybe this really is required for your application, but maybe this information is only partially related to each user, as it seems like the data is relatively static (i.e. the same between many users).

If it really is the case that this information is not really special per-user, you could store it elsewhere (such as, indeed, in an ETS table) and only request it when neccessary. This way you will always have the most up-to-date information, and have a lot less data copying.

In the case that the data really is different per user, then there is nothing you can do except to send a message to each of the GenServers that should update their data. This still wouldn’t be a problem for 1000 GenServers with each 10_000 entries that need to update every five seconds(Map access is O(log n)), but it definitely is slower than above approach.

1 Like
3

Even if special per-user ETS might be best too (with a janitor process if necessary too).