Hi! Your fdb project was very helpful in me getting the erlfdb binding tester back up and running in GitHub CI, so thank you!
One thing that’s unique about developing on top of FDB is that many flowers can bloom, so-to-speak. Love seeing others with interest in the tech. 
I don’t use gh but I will reply here if you don’t mind.
The only prior art I’m aware of for vector indexes on FDB is this project, which was posted on the FDB forums but as far as I can tell they never actually ported it to FDB before abandoning it. It’s not clear to me from the README what type of index they used, but the same org is a maintainer of a very popular SIMD vector search implementation (usearch) so it’s possible they were taking the approach I considered above.
SPANN is interesting, I haven’t read the paper.
The main reason I’m tempted by the quantize+brute-force approach is that it’s enticingly simple, and I am (naturally, as a programmer) very lazy 
The complexity of HNSW and co make me very uncomfortable, especially since it doesn’t really seem like they perform that well. Contrast with a BTree, for instance, which is much easier to understand and yet has excellent performance (for a different use-case, of course).
It could very well turn out that vector search is a problem best solved by brute-force in parallel (like most of OLAP). I don’t think anyone knows for sure yet.
@a3kov A quick update on your question.
erlfdb: The erlfdb 0.2.2 release is in a real production app, working well, with a significant workload. I will consider a 1.0 release, if that is meaningful to anyone.
ecto_foundationdb: EctoFDB is still experimental, but it is now driving a small and low-stakes production app at https://livesecret.link (GitHub)
Cheers!
The inability to rename a field without rewriting the whole DB is a bit worrying… I’m not sure how much of an issue it may become, but it’s exactly the type of things that may block wider adoption of ecto_fdb.
That’s a fair take. However, I believe it glosses over the difficulty in safely renaming a column in other dbs as well, as detailed in safe-ecto-migrations | Renaming a Column.
Certainly the ecto_fdb project has a long way to go – deleting a field is table stakes, for example, and it’s not there yet. Thanks for your interest!
From the linked article:
There is a shortcut: Don’t rename the database column, and instead rename the schema’s field name and configure it to point to the database column.
Ecto_fdb could do the same trick I believe ? But yeah, I totally forgot about this. In 99% of cases we care about the model (schema) field names, and not about the DB column names.
Keep in mind the context in which I brought this up: FDB is designed to operate as a large, distributed system with a potentially enormous amount of data. The “database” layer (here, ecto_fdb) is designed to be stateless.
In this context, you have to take special care to use indirection for schema changes without breaking the consistency guarantees of the database and without rewriting the entire table (because, again, the table could be 100TB).
This is not easy, but it’s not impossible or anything. I don’t think it’s substantially more difficult than, for example, building indexes online (which was already implemented).
This “trick” is essentially doing the indirection in the application layer. It’s inelegant because, ideally, the schema should be stored along with the data it represents. But it would work, I suppose.
I actually believe the FDB design decisions work quite well for small and moderately sized databases as well. For me, it’s more about the kind of workload and the operations story. I wouldn’t want people to come away thinking that they need 100TB of data in order to use it – LiveSecret is essentially a 0-Byte database running on a couple GB of memory (please don’t try to DOS it though 
). I’m pretending FDB is an embedded database by using ex_fdbmonitor , instead of SQLite, even though I am a big fan. Of course, please check FDB’s system requirements for a production app. Running with low system memory does go against the grain of a recommended FDB deployment. (8GB per fdbserver process recommended)
To date, the largest value add for me is never having to write DDL and never having to worry about migrations while I’m doing additive changes (new schemas, new fields, new indexes), which is 99% of development anyway IME.
We wouldn’t know until we tried it, to be honest. Some Ecto features come for free, and some require the adapter to meet specific contractual details. But if it doesn’t work, I imagine it would be possible to add support. Let me know if you find out!
Happy to report there is progress on the main branch, not yet in a release. Specifically, there is a new CLI module, a guide, and a test for deleting or renaming a field safely, in the spirit of Safe Ecto Migrations. Renaming a field in this way actually requires 2 full table rewrites, which you may find unfortunate, but at least now it’s possible to do it 
.
Other progress on the main branch includes refactoring the IndexInventory into a general purpose metadata store. This paves the way for richer metadata such as a schema field map (like an atom table) that could allow for table operations without rewrites, similar to what’s been described in this thread. That may come in future work – I’m still considering the implications.
100% agree. Even for a smaller database you get high availability/durability out of the box and very strong correctness and consistency guarantees (much better than Postgres, for example). Plus if you ever do need to scale, you know it won’t be a problem.
What I was getting at is that any solution for schema operations needs to be able to scale to 100TB+ scale in order to get full value out of FDB. Rewriting a table is fine for a (very) small DB, but not so much as you grow. Especially since FDB encourages a multitenant approach, and that complicates things further.
I have never actually operated FDB, but I struggle to think of anything in the architecture which would necessitate 8GB per process for a small DB. I would imagine this requirement is for operating at scale.
If the DB is small then everything is small: shard map/txnstatestore, byte sample, serverdbinfo, mutations in memory on the tlogs and in the storage ptree, etc will all be small.
Another thing to keep in mind is that for a large DB rewriting a table, even in the background, is actually physically expensive. FDB is a huge distributed btree, and btrees have a lot of write amplification. Rewriting the rows in order will reduce that amplification considerably (because FDB naturally batches disk updates due to the in-memory MVCC design) but even then rewriting every key is naturally going to wear out your SSDs, which is not free. On the cloud they will charge you iops instead, so same problem.
After some thought I think this design would work:
To create a field: add a new field key with a new (monotonic) integer id. It is easy to allocate ids transactionally because migrations are rare so there is no contention. You can store the next_id in the schema.
To rename a field: simply point the new name to the id previously used by the old name, and then delete the old name.
To delete a field: well, delete the field! The id will remain “allocated” because they are monotonic.
So, to run a migration, we perform the above operations and then update the “schema version” to whatever Ecto provides, similar to how it works with Postgres. There should be no problem submitting this as a single transaction to FDB, as it won’t be large.
Naturally, you can then store rows with integer keys instead of string keys, where the integers are the above ids. You could even encode them as varints for even more space savings since there are only as many ids as fields have existed (i.e. not many).
Finally, for correctness you would have to read the “schema version” in every transaction to ensure it matches what the client expects. This would create a bottleneck for a large DB, but you could use the \xFF/metadataVersion key for cache invalidation (increment it on every migration). That key was specifically designed to do exactly this.
You can cache the %{field => id} and vice-versa mappings in an ets table (or persistent_term, perhaps, would be better) along with the Ecto version and the metadataVersion.