aschrijver

aschrijver

Sqlite in Production

(Moved to separate topic from: https://forum.elixirforum.com/t/lower-the-initial-barrier-by-using-ets-and-not-postgresql-for-phoenix-examples/53287 )

FWIW I’ve seen sqlite and related projects (e.g. litestream) getting rave reactions on HN the past 1.5 year or so (even today there’s such discussion), and what I find intriguing in many of the comments I read was people using it in production. So much discussion in the tech community comes from startups and Big Tech who need hyperscale, and vendors playing into this. Many average web apps don’t need all that. I am interested myself in small technology, and sqlite is a great fit there. I think more sqlite focus would not only be attractive wrt onboarding, but for actual use as well.

Most Liked

tj0

tj0

I’ve been using sqlite in production for read-only datasets for 1.5 years now. It has performed much better than postgresql in this use-case and FTS5 has actually been faster on full-text search.

I’ve been planning a complete migration for all workloads from postgresql when litefs is a bit more mature, but it has been close to a drop in replacement after some minor changes.

For those looking to migrate from postgres to sqlite:

# Dump the db and rename some tables from "public"
pg_dump --data-only --inserts db_dev -h localhost -U user > backupName.sql
sed -i 's/public\.//' backupName.sql

# sqlite doesn't support boolean true/false, so you'll have to manually change in an editor
0,.s/true,/1,/
0,.s/false,/0,/

Regarding litefs, https://litefs-liveview.fly.dev/ is multi-region application using an early version and a custom library to forward write requests. ~sheertj/elixir_litefs - sourcehut git is a elixir specific version of fly_rpc and fly_postgres_elixir which is used to forward db requests to the primary on fly. However, I believe litefs may include write forwarding soon, which could make this library mostly redundant.

Regarding write performance, to be honest, I would expect most applications to be fine. I did a quick write benchmark, but unfortunately don’t remember if it was 1k, 10k, or 100k writes per second. My application is very read heavy with between 200k-500k pageviews/events per month (not so easy to measure without instrumenting everything), but only 50k write queries / month. That works out to about 0.02 queries per second which much, much, much, much less than what sqlite can handle. HCTree is not required. :slight_smile:

When litefs finishes solving streaming backup, db migration, and write forwarding, there will be very little reason to be using postgresql in production for a large number of applications.

Edit: hctree: Thread Test
Stock sqlite does between 4k and 40k writes on a single thread. HCTree looks like it is roughly double the performance.

dimitarvp

dimitarvp

SQLite has its own backup functionality as linked by @LostKobrakai – it’s not just copying the files though – and a lot of energy has been expended to prove that it works stably.

Additionally, it also has the concepts of patch-sets and change-sets (section 2.1) – somewhat to the confusion of us in the Elixir space due to the naming clash – that allow you to accumulate changes since a last known snapshot, in a dedicated file.

SQLite has a strong backup apparatus even without litestream. The latter seeks to make backups over the network more or less transparent. The rest is well-covered by sqlite itself.

dimitarvp

dimitarvp

That’s already covered. You issue a backup command and SQLite itself makes new file(s) while you’re using the same session. It’s fully atomic.

People get needlessly scared there, their team is very aware of the potential issues and took care of them a while ago.

Where Next?

Popular in Discussions Top

CharlesO
Erlang :list.nth simple, but 1 - based nth(1, [H|_]) -> H; nth(N, [_|T]) when N > 1 -> nth(N - 1, T). Elixir Enum.at … coo...
New
mmport80
I have put far too much effort into Dialyzer over the last year or so - and basically - I doubt it’s worth the effort. It’s not as easy ...
New
tmbb
This is a post to discuss the new Phoenix LiveView functionality. From Chris’s talk, it appears that they generate all HTML on the serve...
342 18243 126
New
Markusxmr
Since Drab has been developed for a while in the open, introducing the Liveview functionality in a way it happend appears to undermine th...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39467 209
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
cvkmohan
The upcoming Phoenix 1.6 release looks very interesting. Became a habit to watch the commits - and - what they are bringing in. phx.gen...
New
Rustixir
Hi everyone, im working on find best language/framework/system for high concurrency, high performance and stable performance after wor...
New
AstonJ
Can you believe the first professionally published Elixir book was published just 8 years ago? Since then I think we’ve seen more books f...
New
100phlecs
Are there any downsides, like perf issues, to putting all functional components in CoreComponents as long as you prefix it with the conte...
New

Other popular topics Top

jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39467 209
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement