zteln

zteln OP

@garrison Thanks for your reply! I have rewritten a large part of the code, focusing on making consistency and durability guarantees simpler and better. Main changes:

  • MemTable implemented as an ETS table and the store putting SSTs into an ETS table, allowing concurrent readers
  • removed all conflict detection between writers and instead implemented one writer at a time and any pending writers are placed in a write queue. This is true serial execution.
  • the WAL syncs after every append

Some other changes: CRC verification on SSTs on start, and data in SSTs are compressed for levels larger than 1.

Check the new version out if you have time, any more feedback is welcome and greatly appreciated!

1
Post #11
garrison

garrison

Those changes all sound good to me. A single writer is a reasonable tradeoff for an embedded DB.

One thing you might want is read-only transactions that don’t have to take the write lock. You have a multi-get operation, but that’s not enough to do pointer chasing on a consistent snapshot:

key1 = "foo"
key2 = get(key1)
key3 = get(key2)

This pattern is needed for many data structures that one would implement over a KV store. Without a transaction these reads will be performed at different versions, which will cause consistency issues.

If the transactions are read-only you don’t need any further concurrency control. All you need to do is get the latest seqno of the most recent committed write txn and perform reads at that version. That is enough for strict serializability (along with the single writer).

You do also need to be able to hold a snapshot of the manifest open (blocking SST deletion) for the duration of the read transaction, which we discussed previously. You will need that for backups as well.

2
Post #12
zteln

zteln OP

Thanks for the feedback! I’ve released v0.3.0 now incorporating your suggestions:

  • Read-only transactions (can be obtained by providing the option read_only: true to Goblin.transaction/2)
  • Added Goblin.export/2 which exports a snapshot of the database into a .tar.gz file.

The internal manifest format was changed for these changes. Let me know what you think!

1
Post #13
zteln

zteln OP

Small update (v0.3.2):

  • Flush and compactions are triggered by checking byte sizes instead of amount of keys. This changes the start options to Goblin.start_link/1. The default values are mirrored from RocksDB default values.
  • Fixed some tests, working on covering more tests
  • Working on optimizing as well.

Let me know if anything doesn’t work!

2
Post #14
zteln

zteln OP

Yet another release (0.4.0) with some larger rewrites. Goblin has been redesigned almost from the ground up with consistency guarantees in mind. Some noticable changes:

  • Goblin.read/2 is introduced as a read-only transaction (instead of previously providing read_only: true to Goblin.transaction/3, which is now deprecated in favour of Goblin.transaction/2).
  • Goblin.is_flushing/1 and Goblin.is_compacting/1 have been renamed to Goblin.is_flushing?/1 and Goblin.is_compacting?/1, respectively.
  • The file format which Goblin saves data to has been changed, making previous versions non-compatible with the new version.

Any feedback is welcome!

1
Post #15
dimitarvp

dimitarvp

Just to make you aware: in the Elixir community people would do either is_flushing or flushing? but not both at the same time.

2
Post #16
Asd

Asd

+1 to that comment.

  • names ending with ? (like flushing?) are functions returning a boolean
  • names starting with is_ (like is_number) are macros that compile to guards

You need to name functions just flushing?

1
Post #17
zteln

zteln OP

Ah, thanks for the clarification!

0
Post #18
— All posts loaded —