Attempt at adapting KuzuDB's Rust crate into Elixir via NIF, any tips/thoughts?

Hi, great job with starting the project, @bgoosman

I’m wondering what your plans are in regard to the repo’s future; thanks!

I think integrating it with the Explorer library would be nice. i.e. pass dataframes from Elixir to/from the NIF, to be loaded into kuzudb, or fetched from kuzudb. I tried without much success.

WDYT?

Thanks for reminding me about this. I just updated the kuzu crate from 0.6.0 to 0.7.0 and improved the Elixir unit test.

We can do some pretty useful things with just run_query, like create a kuzudb using COPY FROM commands.

Besides that, feature parity with the API.

Thanks for updating your repo @bgoosman . Yes passing dataframes would be a good idea.

I’m wondering how the BEAM process management maps onto Rust threads, i.e. can we achieve proper query concurrency using KuzuDB via Elixir?

Are you using KuzuDB in production with your integration? What is the use case, without getting into any trade secrets

Not sure about BEAM & Rust threads. Here’s KuzuDB’s concurrency model https://docs.kuzudb.com/concurrency.

You got me thinking about the connection to the db. Right now run_query will create a new connection every time it’s called, which doesn’t sound ideal. I wonder how to manage a persistent connection.

We’re a graph viz company, so having an embeddable graph engine like KuzuDB would be an excellent improvement over a server based graph engine like Neo4j.

That’s my goal as well - being able to do in-mem Cypher queries.

Maybe @filmor @hansihe or @evnu could point us in the right direction in terms of how the mapping between BEAM processes and threads in Rust works.

I was curious so I took a peek, and I think this function might benefit from being scheduled on a dirty scheduler: kuzu_nif/native/kuzu_ex/src/lib.rs at f94c2e243983650db26c4b19bb0f73835fd52ec5 · bgoosmanviz/kuzu_nif · GitHub and re: “mapping processes to threads”, I think usually libraries for wrappers around embedded databases create a NIF resource per database connection and then Elixir / Erlang manage them however they want, no 3rd-party threads involved. Looking at kuzu_nif/native/kuzu_ex/src/lib.rs at f94c2e243983650db26c4b19bb0f73835fd52ec5 · bgoosmanviz/kuzu_nif · GitHub, it seems like the API might be similar to DuckDB wrappers, since they also have a notion of “databases” and “connections” (SQLite only has “databases”). Two example DuckDB wrappers in Elixir:

• GitHub - AlexR2D2/duckdbex: The Library embeds C++ DuckDB database into you Elixir application. · GitHub
• GitHub - ruslandoga/duckdb_nifs: DuckDB bindings for Elixir · GitHub

So I think the API could be:

# NIF resource, DB is closed when the underlying reference refcount goes to 0
# can be a Dirty IO NIF
db = Kuzu.open() 

conns =
  for _ <- 1..:erlang.system_info(:dirty_io_schedulers) do
    # also NIF resources, since Kuzu is C++
    # they probably reference and keep DB alive too,
    # so it's easy to just wrap them info resources individually.
    # When none of them are referenced anymore, Erlang would "garbage collect"
    # them and call NIF resource destructors, which would destroy Kuzu connections.
    # once Kuzu connections are destroyed, Kuzu would probably close DB as well
    # since it would no longer be referenced either.
    # can be a Dirty CPU NIF or a plain NIF, depending on what Kuzu does here
    Kuzu.connect(db)
  end

{:ok, pool} = GenServer.start_link(SomePool, conns)
{ref, conn} = GenServer.call(pool, :checkout)

result =
  try do
    # since it might trigger disk access, it would be safer if it was a Dirty IO NIF
    Kuzu.query(conn, query)
    # or it can be a Kuzu.prepare followed by a Kuzu.execute
    # this would allow statement caching and query params
    # Kuzu.prepare can be a Dirty CPU or a plain NIF
    # Kuzu.execute would be a Dirty IO NIF for the same reasons as Kuzu.query
  after
    GenServer.cast(pool, {:checkin, ref, conn})
  end

# can probably be a plain NIF
Kuzu.to_arrow(result)

I think this would be useful in the upcoming AoC