michalmuskala
Another small library today.
PersistentEts
Hex: persistent_ets | Hex
GitHub: GitHub - michalmuskala/persistent_ets · GitHub
Ets table backed by a persistence file.
The table is persisted using the :ets.file2tab/2 and :ets.tab2file/3 functions.
Table is to be created with PersistentEts.new/3 in place of :ets.new/2. After that all functions from :ets can be used like with any other table, except :ets.give_away/3 and :ets.delete/1 - replacement functions are provided in this module. The :ets.setopts/2 function to change the heir is not supported - the heir setting is leveraged by the persistence mechanism.
Like with regular ets table, the table is destroyed once the owning process (the one that called PersistentEts.new/3) dies, but the table data is persisted so it will be re-read when table is opened again.
Example
pid = spawn(fn ->
:foo = PersistentEts.new(:foo, "table.tab", [:named_table])
:ets.insert(:foo, [a: 1])
end)
Process.exit(pid, :diediedie)
PersistentEts.new(:foo, "table.tab", [:named_table])
[a: 1] = :ets.tab2list(:foo)
Where Next?
Trending in Announcing
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Bringing learning, work and compliance together since 1987.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 13 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
lessless
Thank you for insights!
I tried to look into the sources of
tab2filebut didn’t have enough patience to dig deep down into what they’re using there for file manipulationI just saw that they’re using message passing and that was enough for me to understand that there will definitely be overhead.
Exadra37
As far as I know nothing in Elixir that writes to the disk can give you that level of guarantee:
As per Erlang docs :
I would love to be proved wrong
lessless
Does it mean that it can guarantee that an entry will be saved on disk if the insertion function returned ok?
How will it behave in the presence of always active clients?
It’s impossible to postpone shutdown indefinitely.
OvermindDL1
That is basically how it is with mnesia as well, using dirty reads/writes is near the same as calling ETS straight, except it can still serialize the data out in the back-ground after a write instead of needing dumping periods.
Although could not get to 10gigs with it, at least on a 32-bit system. ^.^
JEG2
The process is just for managing persistence. You don’t need to send messages to the process to interact with the table.
JEG2
This persistence model is what Redis uses, so I would say you could use this in many scenarios where you would otherwise reach for Redis.
aseigo
Data access, sure. But the performance of PersistentEts can not be the same as Ets: Ets does not include persistence. So while data access is surely the same via the :ets api, the overall performance of PersistentEts is still interesting imho..
I don’t find it easy to understand what those numbers mean without knowing the hardware involved, and also not overly meaningful without understanding how often persistence occurs and in which circumstances. Looking at the code, by default it writes the table to disk once per minute, regardless of changes to the table. (.. as well as on table owner exit) Am I reading that correctly?
What is the intended use case for this? It can’t be for “valuable” data, as any changes to it within the persist timeout will be lost. It can’t be for large tables, since hitting disk for such a time period could be pretty undesirable if there is other I/O happening, not to mention messages to the PersistentEts process which may be piling up behind it?
Looking at the code, I’m also unsure what happens if it so happens that PersistentEts.new is called when another PersistentEts is busy writing out a table … ?
So .. perhaps this is intended for “well behaved” applications with “small” datasets in ets tables?
I’m sure there must be a good use case for this, just trying to understand what it is. (So please don’t take the above as too critical .. just walking through my thoughts as I look over the code)
michalmuskala
Performance of PersistentEts should be the same as performance of Ets itself - it is Ets.
It persists periodically - the default is every minute. It also triggers when the owner terminates. You can also trigger persistence manually.
Yes each time the table is fully dumped. In my quick tests, dumping a 10GB table takes 20-30s. A 100MB table takes ~200ms.
OvermindDL1
Is there a benchmark of it compared to Mnesia with duplicate_bag tables using dirty read/writes (basically ETS that is DETS backed at that point) and similar settings for PersistentEts?
Does it only persist to disk ‘on occasion’ or after every write? Does it do it when the owner process is terminated? I’m guessing via the file2tab and such that it is serializing out the entire ETS table every write out instead of only the differences?
hubertlepicki
Copy paste that to readme now.