andzdroid
I saw this bit of text on the cubdb repo’s readme (GitHub - lucaong/cubdb: Elixir embedded key/value database · GitHub):
Snapshots come at no cost: nothing is actually copied or written on disk or in memory, apart from some small internal bookkeeping.
In relation to this code:
{x, y} = CubDB.with_snapshot(db, fn snap ->
x = CubDB.Snapshot.get(snap, :x)
y = CubDB.Snapshot.get(snap, x)
{x, y}
end)
I looked at the with_snapshot function and all it does is make a genserver call. Can somebody explain how this doesn’t copy anything in memory?
def snapshot(db, timeout \\ 5000) do
GenServer.call(db, {:snapshot, timeout}, :infinity)
end
My understanding of elixir was that messages across processes are always copied. Is my understanding wrong? Am I missing something?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
LostKobrakai
You’re correct. The message
{:snapshot, timeout}is copied when being sent. That doesn’t mean the database itself (as in the data your stored in there) is copied when you do a snapshot though – which is what the documentation is trying to say.andzdroid
Wouldn’t the whole db be copied in the reply? The function is trying to get a snapshot of the whole database, which is sent back by the genserver call.
LostKobrakai
That expects that “the snapshot” is “the data you store in the db”. It probably isn’t. It’s likely just a pointer or reference to the actual data stored elsewhere.
joey_the_snake
If it was a pointer or a reference then it wouldn’t really be a snapshot because then you could see other people’s modifications. Otherwise it would have to be a pointer or a reference to a copy.
I believe the docs are implying no copying outside of data transfer between processes.
LostKobrakai
That depends on the datastructure used and what information makes up the pointer. E.g. if the datastructure is an append only log you can create a “snapshot” just by having the id of the last item in the log, which is still part of the snapshot. CubDB afaik uses some kind of tree structure to store data.
joey_the_snake
Looking at the source, it seems like a tree structure is copied to the caller. And this tree structure is walked when trying to read particular keys.
LostKobrakai
Yeah, but that tree structure does in turn refer to a data store and that data store (file by default) holds the actual data. So yes, some stuff is copied (the internal bookkeeping mentioned), but it’s not the database nor the values it stores.
joey_the_snake
Yeah true. I do see the OP’s point though. I wouldn’t necessarily expect the full tree structure, even if it doesn’t have all the data, based on the description.
LostKobrakai
I’d imagine that in the grand scheme of things the tree is probably tiny compared to what would need to be moved around if values would be affected – especially when the db reaches sizes where you actually start caring for it not being copied.
lucaong
@LostKobrakai is essentially right.
CubDBuses an append-only Btree: changes are appended to the database file, like in a log file. Basically, every new change causes a new Btree root to be appended to the database file. The root points at the old nodes as well as to some new ones containing the data that changed. Existing Btree nodes are never updated in place. It works very similarly to an immutable data structure, like the ones used by Elixir. This means that if one has a reference to an old Btree root, they can traverse the tree and see the database exactly as it was when that root was the current one.Obtaining a snapshot then is as simple as holding a reference to a Btree root: new changes can be written to the database, but they won’t change what is pointed by that old root. Therefore, there is no database copying, as in CubDB does not have to make a copy of the database to give you an immutable snapshot. The Btree root is a small data structure, only holding a reference to other nodes down the tree: it doesn’t contain any data. Taking a snapshot is a very fast
O(1)operation. In fact, it is internally used on every read operation, to ensure consistency in presence of concurrent writes.The bookkeeping mentioned is simply because occasionally, in order to avoid growing the database file indefinitely as more changes accumulate, CubDB performs a compaction, which discards the data which is old and not referenced anymore. CubDB has to keep track of what’s referenced by the existing snapshots to avoid removing it, a bit like a garbage collector.
The details of compaction are a bit more involved, as compaction can proceed in parallel with reads and writes, but in essence when you take a snapshot CubDB does not copy any data: it merely remembers which Btree root your snapshot points to.
When you read from the snapshot, of course data will be copied from disk into memory. But that’s just the data you read, not the whole database. And, again, that is when you read from the snapshots, not when you take a snapshot.
I hope this clarifies it