Distributed Databases - options/thoughts on storing data in a distributed manner?

4 Likes

What’s the TLDR Michal? Or was there anything in particular you wanted to discuss? :101:

Guess we’ve got https://en.wikipedia.org/wiki/Mnesia :003:

4 Likes

Nothing particular just wanted to gather information how to store data in distributed way :slight_smile:

2 Likes

AntidoteDB might also be relevant…

https://syncfree.github.io/antidote/

3 Likes

OrbitDB is based on IPFS, which is a distributed filesystem

Distributed databases interest me, since we can use the web then how it is intended

Completely distributed :smile:

4 Likes

Until very recently big pain even for companies with unlimited resources and very capable engineers. In the end it makes sense for them to create a solution that looks and behaves like ACID SQL RDBMS e.g. Spanner, CockroachDB. I hope CockroachDB at some point becomes a viable option but would not bet on it.

3 Likes

You can try TiDB

Looks interesting thank you, for me to be comfortable with using something for actual data store on a project has to be really mature tech. and in wide spread use. I do like to play with things though just to learn :slight_smile:

1 Like

Not really about a database tech, but sagas seem to be an interesting way to solve transaction consistency when your data is spread across services in a microservice architecture:

1 Like

I also think its fine, that you can connect it to a K/V store in Rust.
:slight_smile:

I think I would prefer to put data in apache kafka isntead saga pattern :slight_smile:
https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/

1 Like

??? not sure what you mean :slight_smile: I am fine dealing with a young language but integrity of a data store is not something I am comfortable compromising.

2 Likes

You can connect TIDB to this K/V store

I guess hard its new to you that both, TiDB and CockroachDB are written in golang?

Since otherwise would you not complain about a database which is written in Rust :slight_smile:

How is a database which is written in a language with manual memory allocation more save as Rust, which is fully memory save, nearly impossible to compile with run time errors and in stable API since years?

Not to say anything about a full OS, which is written by 1-2 guys within 2 years in it.

When it comes to data security is ZFS very helpful since it allows you to basically roll back and forth to any state you find useful :wink:
SmartOS is great for that, same as freeNAS and a casual Linux. :slight_smile:

??? Still confused why is hard for me that someone is using Go to do a project? It would be hard for me if I had to do a database in Go. ZFS is wonderful battle proven tech that is pretty much useless for snapshotting state of geographically distributed (multi-node database). SmartOS is very cool project as pretty much anything Bryan Cantrill has done my only worry is that Samsung might not care much about open source side of things going forward.

2 Likes

OK spent time digging around CockroachDB does look promising very strong team and appears to be relatively well funded I guess 3-4 more years might be a usable option.

2 Likes

And nice part is some sort PostgresSQL compatible.

2 Likes

I also found myself digging deeper into ‘more fancy’ databases like dgraph, while I again and again stumble about the fact that most projects have written plain old SQL drivers and nothing else. :slight_smile:

Which means, this design decision of both, TiDB and CockroachDB might play a major role when it comes to adoption?

I see TiDB a bit forward here since a) MySQL drivers are more common b) Postgres devs are more conservative and c) Postgres is the more advanced technology

How you see that?

What I think data all distributed storage technology right now migrate to some sort SQL interface.
For example Apacha Kafka now have KSQL (https://www.confluent.io/product/ksql/)
I think people get tired of pure no sql databases where they need to put all logic in application itself.

2 Likes