I would recommend not overlooking the magic of an in memory bloom filter such as Blex as a first pass membership test.
You may even conclude that building SQLite indexes is not the best approach with a space efficient memory based bloom filter.
adw632
scohen
We’re using ets presently. Next ls uses SQLite
I really want to keep lexical 100% pure elixir.
2
scohen
adw632
If you do decide to look at space efficient membership filters that work in constant time independent of number of elements, far better than log(n) and not requiring an external database dependency like SQLite.
I would suggest that for your use case a variant of bloom filters known as cuckoo filters would be most appropriate as you need to update the index as code changes.
There is a maintained Elixir/Erlang implementation here:
GitHub repo here: GitHub - farhadi/cuckoo_filter: High-performance, concurrent, and mutable Cuckoo Filter for Erlang and Elixir · GitHub
If you don’t understand what a probabilistic membership filter is then I think it would be worth understanding how they can make the otherwise impossible, possible. Why introduce a whole lot of IO in database file when you don’t have to?
0
scohen
adw632
Fair enough I don’t presume to know all the problems you’re dealing with but I do know that keeping a database in sync with a changing code base will be quite IO heavy, hence expressing some reservations and possible alternatives than introducing a database.
Please take my posts as merely trying to create awareness of possible alternative approaches and nothing more than that.
0
adw632
Yes there is a misunderstanding I misread the sqlite statement.
Clearly I need more sleep
2