Why so Serious?

I am aware of this project and I do like it. They clearly share my values about all of this rugpull nonsense and for that alone they have my full support.

However, and this is what I was getting at with the MinIO point, there is a bit of a technical issue here. For the trend I was talking about specifically, weak consistency guarantees are kind-of a problem, as is high latency. This is why AWS introduced their “S3 Express One Zone Storage Class” (seriously they need to fire their entire marketing dept), which to be clear is actually not S3 at all (other than by name) but an entirely new system. This system targets a class of products which has been emerging lately: hosted cloud services where you “bring your own S3”. There are many examples of this, like Snowflake (data warehouse), Quickwit (full-text search), Turbopuffer (vector search), s2 (some queue thing I guess), Neon (literally Postgres pages shoved into S3), and so on.

The reason AWS had to essentially build a new S3 to enable these products is that, fundamentally, object storage with its high latency and poor consistency guarantees is kinda a bad primitive for building databases. People were just using it because it was there, so they have adapted.

The “alternatives” like MinIO and Garage and others (SeaweedFS is another) have not really caught up, though. They are classical object stores and make poor primitives.

It makes a lot more sense to start with a strongly consistent system, because strong consistency is very difficult to achieve but very easy to give up.

And, as you may have noticed, this is exactly what I have been doing :slight_smile:

3 Likes