GenServer/ETS/Rustler performance questions

Hisako1337

Good morning fellow alchemists!

I have a question about quite a low level implementation topic regarding copy/concurrency semantics, let me explain the situation briefly first:

I have some (~10) ETS tables (set/unordered) that contain a few 100k records each, where the primary keys are either straight integers or tuples of integers (mimicking compound keys)
the values in the tables mostly are HashSet of integers (“IDs”)
the tables are owned by dedicated GenServer processes, so that only the owning process can update data within its ETS table, but all other processes can read in parallel from it
the owning processes subscribe to phoenix PubSub events in order to update individual ETS records in realtime as needed
there is a general “search” function that contains several keys to look up within the ETS tables, each returning a HashSet of IDs, and the final result set is the intersection of all these IDs in the hashsets (meaning IDs that are in all returned lists) + some business logic in between
this is already a performance optimization on precomputed data in memory, since the database chokes on this calculations (we optimized as much as we could before), therefore “use SQL” is not an option here
this search function is also a hot code path throughout our whole application and must be as fast as technically possible (few miliseconds, tops)
it worked quite good so far but now that the amount of records in the tables (and the lenght of the result sets) go up over time, this “algorithm” is becoming noticeably slower
we don’t want to introduce additional infrastructure components like search engines

so this is the situation right now. the precomputation of data and holding it in ETS tables already brought us a great performance boost, but it seems the “intersection” logic with custom rules is the new bottleneck now.

What we already use is Rustler for a different problem elsewhere in the project, and it basically “solved” another complicated computational problem. But here I think the actual problem is the “copying” of increasing amounts of data (few dozen megabytes) between BEAM processes, plus the computation afterwards.

Now my idea right now is to move the precomputation-code into a Rustler NIF, maybe even store it all in a single Map structure instead of multiple ETS tables, and on request use this map as a parameter + the search parameters to calculate the resulting list of IDs also in the NIF and return it.

The three questions I have right now (couldn’t find details so far):

Can I pass (and receive!) data back and forth into the NIF via pointer/reference instead of copying it to eliminate any overhead here?
Can I hold the precomputed data in a Genserver and make it accessible in a concurrent way similar how ETS can be configured for parallel read access? Without too much copying?
Can I modify parts of the data or replace the whole data container from within Rust to speed up things?

AFAIK a GenServer can not handle true concurrency (it iterates on its single linear inbox), and Task creates new processes that copy memory into it, even ETS copies memory on querying but at least updates are “in-place”.

Thanks in advance!

15 comments

#genserver

2 1187 15

2024-06-27 15:37:20 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

dimitarvp

As a start, do you want that cache to be multi-node or single-node?

Post #1

cpud36

In Rustler you can use resource/ResourceArc to pass a reference into Elixir code. The documentation is quite lacking, but you can look at this test.

If you want to have mutable resource, you must manually synchronise on the Rust side. E.g. you can use RwLock, or an rcu.

Beware, that resource references cannot be shared across nodes AFAIK.

Post #2

Hisako1337

single node, as in every node in the cluster has its own local cache, but every node in the cluster has the cache

Post #3

Hisako1337

oh cool, that already helps a bit, thanks!

Post #4

filmor

Resource references can be shared across nodes, but they can only be used (in a NIF) on the node that created them initially and only if it’s still alive when accessed. E.g. if you generate a resource object on your node from a process, send it to another node, let the process die and receive the answer back, the resource object will not “work” anymore.

I’m currently working on refactoring resources (and now that you say it, @cpud36, the docs for this one are indeed very much lacking) as they are using a mechanism that will produce warnings and later errors in rustc.

Data is always passed by reference into NIFs as they run in the same process environment as the calling process. You could (as an alternative to using a resource object) also store the data in a binary object (e.g. using something like flatbuffers). If it’s larger than 64 bytes, its data will not be stored directly in the process environment and will not be copied either.

Post #5

dimitarvp

My Rust chops have eroded a little bit since I haven’t used it actively in like half a year now but last I remember I have used DashMap successfully before in an application that had a very heavy usage of read and write caching. It’s single-node.

Sadly this was predating the times I became obsessed with telemetry so no clue if it’s doing better than ETS but I’d think since it doesn’t copy data that it should be faster.

Others have already pointed you at how you can create a resource inside the Rust code and return a reference to it in Elixir.

Post #6

andyleclair

If you’re precomputing the data, you could look into using :persistent_term but I’m not sure the viability of updating that data, since you’d force a global GC on every update. How frequent are the updates?

Post #7

Hisako1337

Data is always passed by reference into NIFs as they run in the same process environment as the calling process

this is hugely valuable information for me already, thanks for that!

so this would solve part of my problem already, passing the data blob into rust for further computations is “free”!

Post #8

Hisako1337

I have used DashMap successfully before in an application that had a very heavy usage of read and write caching.

this looks nice indeed, but my problem around it is still: where to store that data in Elixir land? a GenServer afaik does not allow for truly concurrent reads

Post #9