Need help with Exqlite segfaulting when timing out

Is there a way I can have valgrind with an asdf installation or am I going to need to use some other erlang / elixir version manager to do this?

I’ve debugged my share of C NIFs in the past, so I hope I can provide some help. I pulled your branch and reproduced the segfault. Locally I added some fprintf-based logging to exqlite_close and noticed something that made my raise an eyebrow, so I thought I’d share. If I’m off base here, please let me know.

An fprintf statement in exqlite_close prevents the segfault on my system, so there is definitely a timing element here (race condition). I printed the memory addresses for conn and conn->db and saw many duplicate addresses, which made me wonder if exqlite_close is being called multiple times on the same resource, resulting in a double-free-like segfault when running with high concurrency.

BTW, thanks for this library. I am a fan.

This was my observations as well. Which makes me believe this has something to do with the garbage collector.

I’m not familiar with DBConnection, but I’ve written similar libraries in the past. When you do DBConnection.start_link, is it calling your connect one time and then sharing the result among the pool of 50? If that’s the case, I wonder if creating a single enif resource and allowing it to be “disconnected” from multiple concurrent processes in the pool is the source of the race condition. The DBConnection README says it calls disconnect automatically:

DBConnection also takes all of the care necessary to handle
failures, and it shuts down the connection and the socket
whenever the client does not check in the connection to avoid
recycling sockets/connections in a corrupted state (such as a socket
that is stuck inside a transaction).

(It looks like DBConnection was written with network sockets in mind. Due to the nature of SQLite, obviously there is no network socket, so the disconnect behavior may not be optimal for your use case.)

In my experience, it’s safest to ensure that an enif resource gets a dedicated BEAM process so that you can carefully control the access to it. At the very least, I assume you only want a maximum of 1 exqlite_close on a given resource; You can build that guarantee by serializing calls to Connection.close on a dedicated BEAM process. Otherwise, you can’t guarantee there are not 2 scheduler threads freeing the memory simultaneously.

We get spoiled by the BEAM scheduler and garbage collection, but when it comes to NIFs sadly we have to worry about all these issues again. I don’t believe the BEAM garbage collector is causing any problems in this case.

Edit: in my haste I probably jumped to some wrong conclusions about DBConnection. I’m going to spend some more time and try to get more concrete results.

The way I’ve written it is that every time a new connection to the pool is created, a new sqlite instance is opened and no sharing of sqlite databases between connections.

Is db_connection even necessary in SQLite’s case? When I was trying to get my library off the ground a few years ago I planned on using :poolboy or, as I learned lately, the :jobs Erlang library.

Plus if you open only one handle and set it up as fully multithreaded you can have only one stateful OTP process per SQLite connection string and return it each time a connection is checked out – and only close it if the processes is terminated – which you can do manually with an API e.g. Exqlite.close(handle) or simply when the app itself shuts down.

no sharing of sqlite databases between connections

Yes, I missed the mark on that detail. I see now that the pool is starting a process per enif resource.

However, the DBConnection docs sound as if they’re exposing the “socket” (in your case the enif resource handle) to client processes as an optimization. The lib is pretty sophisticated, so it’s challenging for me to grok quickly, but the docs suggest this is the case.

Other database libraries would send a request
to the connection process, perform the query in the connection
process, and then send it back to the client. This means a lot of
data copying in Elixir. DBConnection keeps the socket in the
holder and works on it directly.

If I have this right, then you may find that multiple processes (the client procs) are calling exqlite_close concurrently on the same enif resource.

Not necessarily, but I tied this pretty closely to being used by ecto_sqlite3.

I could use poolboy or nimble pool for pooled resources, but when it comes to Ecto, you have to implement the DBConnection interface in order to have it play nice with ecto_sql.

This certainly could be the case. I just don’t have a good way to prove that a double close is being called.

I get it and I’m not judging, simply saying that if I encounter something like this and it takes me more than a few hours to troubleshoot I’d likely just eliminate the problem by drastically changing approach.

I even have most of the Rust code that maintains a per-connection-string SQLite handles cache (a parallel hash map); IIRC I only needed to add tests there but it’s when I stopped everything due to personal and work life turmoil.

I will be starting a new job soon and after I settle a bit I’ll finally return to making my library workable. Maybe you can take inspiration from my Rust code; I’ll be happy to help when I have some more free time as well.

Feel free to ping me anytime!

Oh. I haven’t looked into that. I definitely will now, thanks.