Anybody actually use Litestream (the SQLite backup/syncing layer) with Elixir?

Questions / Help
1

The creator is being supported by Fly.io (hired) to work full time on Litestream.
I have a new project I’m ramping up and it’s going to be an e-commerce project.

I’m wondering how Ecto Sqlite3 layer can work with Litestream, in an e-commerce platform.
Because if that works, then I can:

  • use S3 as a very cheap backup solution
  • auditing
  • provide fast user experience because no need for network hop (like when using Postgres)
  • it seems like this is somewhat of a next generation data store architecture

Thanks in advance!

2

Litestream is a backup solution, you have to deploy S3 separately or pay for it.

“Litestream has limitations. I built it for single-node applications, so it won’t work well on ephemeral, serverless platforms or when using rolling deployments. It needs to restore all changes sequentially which can make database restores take minutes to complete.” - Ben Johnson

Why not use a distributed Rqlite?!?

When you absolutely must not lose any of your data.

rqlite is an easy-to-use, lightweight, distributed relational database, which uses SQLite as its storage engine. rqlite is simple to deploy, operating it is very straightforward, and its clustering capabilities provide you with fault-tolerance and high-availability.

It also has an in memory database mode with max 2 gb per node capacity! Which won’t loose data on restarts, since the Raft log is the authoritative store for all data, and it is stored on disk by each node, an in-memory database can be fully recreated on start-up from the information stored in the Raft log.

Also don’t be afraid to use the right tool for the right job.

Like Meilisearch for faster text search.
Or running MinIO if you want object storage.
Or CockroachDB if you want distributed Postgres like DB.

I’m using all 3 and still going to add Rqlite for storing data close to application, in memory. For config management that doesn’t belong on main db.

By no means thats a complete list of databases you will need.

You will end up needing ClickHouse for OLAP or Time series data storage, prometheus for monitoring, loki for logging, a key value store for something & who knows what, eventually.

P.S. Please be wary of the GitHub issues section.

I have been burned by jumping the gun and boarding the hype train, only to reach the end of the road.
Big teams can afford to migrate away from a tech debt, but you will be left hanging.

There’s a reason people prefer Postgres, and now I know why!!

3 Likes
3

I use it on a personal project that needs a sizeable database that’s predominantly read only, for which I want to minimise the number of moving (and costly) parts. It works well! But I’d stick with hosted Postgres for most things.

1 Like
4

… I like that!

1 Like