To give some more context on top of what @lucaong already said: How I have it configured is by starting CubDB using a database location fetched from the application environment (i.e. Application.get_env(:my_app, :myimportant_db_location) which I have configured to point to data/myimportant_db in the config/config.exs but to point to /root/data/myimportant_db in config/target.exs. This means that CubDB will work normally (and self-contained to the application we’re working on) when running it on the host machine, but will properly work by using a subdirectory of /root when running it on the Nerves device as well.
1 Like
Hi,
Thank you for the pointers. I only have one CubDB process.
I removed all Logger references, including require Logger calls, and the problem is still there.
What is strange is that there is this error about sys.beam, but why the BEAM would load this file once again, since the module is loaded at the VM boot …
All the calls to CubDB are made by 4 processes, all using get_multi and get_and_update_multi either to load data or to write data.
Reading:
CubDB.get_multi(cub, keys, :NOT_FOUND)
When writing we give no key:
CubDB.get_and_update_multi(cub, [], fn %{} ->
{events, puts, delete_keys}
end)
edit
I have found that the error happens because I’ve set auto_compact: true. If I set it to false, everything works well. So maybe it is an expected behaviour ?
Everything seem to work well, as I changed your code to add IO.puts calls and have the following logs:
IO.puts("compaction_completed")
send(caller, {:compaction_completed, btree, compacted_btree})
defp can_compact?(%State{compactor: compactor}) do
case compactor do
nil ->
IO.puts("compaction will run")
true
_ ->
IO.puts("compaction is pending")
{false, :pending_compaction}
end
end
compaction is pending
compaction_completed
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction will run
compaction is pending
compaction is pending
compaction is pending
compaction_completed
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
compaction is pending
Very interesting find about the auto_compact: true, Thanks @lud !
Only one compaction operation can run at any time, so when a compaction is already running, writes do not trigger another one. The pending compaction should only create one single file (and clean up after itself), but it might be that something is going wrong here. This is already a good start for an investigation.
One question: if you ls the data directory, which files do you see? If all is fine, you should see the .cub file and, possibly, a .compact file if the previous compaction did not complete (this file will be cleaned up when the next compaction starts). If you see many .compact files or other strange behavior, let me know.
I have two .cub files:
test//db/Elixir.Gem.EventsTest/D27.cub
test//db/Elixir.Gem.EventsTest/D26.cub
I think it is normal to have two because the app crashes so maybe CubDB cannot remove the old file. I have no .compact files.
Exactly as you say.
It looks like the compaction completed correctly, caught up with the updates occurring during the compaction, and then the VM crashed before the old file was cleaned up. That’s not a problem in itself, as CubDB is designed to handle this, and will clean up the old file at the next available opportunity.
That said, I still wonder what was causing the emfile or the sys errors. I will go through the code and see if I can find something suspect. I will also try to reproduce this. Let’s see.
If you happen to have a short example that reproduces this problem, please share it with me 
Good news for you, it seems to be related with my mutex and not with CubDB.
I tried with the following code. If you remove all references to Mutex, it works perfectly, if you keep it as-is it will crash.
I added some tests (locally) to the mutex suite and I can have 4 processes trying to lock the same key concurrently 10K times without seeng the error.
I also tried to remove all Logger references to the mutex but my code using CubDB keeps crashing.
defmodule CubDbCrashTest do
use ExUnit.Case
@db_dir "test/db/#{__MODULE__}"
@db_name Module.concat(__MODULE__, Repo)
@mutex Module.concat(__MODULE__, Mutex)
setup_all do
File.mkdir_p!(@db_dir)
db_opts = [auto_compact: true, auto_file_sync: false]
gen_opts = [name: @db_name]
start_supervised(%{
id: __MODULE__.DB,
start: {CubDB, :start_link, [@db_dir, db_opts, gen_opts]}
})
start_supervised(%{
id: @mutex,
start: {Mutex, :start_link, [[name: @mutex]]}
})
:ok
end
defmodule Account do
defstruct id: nil, balance: 0
end
defp put_account(%Account{id: id} = account) do
CubDB.get_and_update_multi(@db_name, [], fn %{} ->
{account, [{{Account, id}, account}], []}
end)
end
defp get_account(id) do
[account] = CubDB.get_multi(@db_name, [{Account, id}], :NOT_FOUND)
{:ok, account}
end
defp withdrawal(account_id, amount) do
Mutex.under(@mutex, {Account, account_id}, fn ->
{:ok, account} = get_account(account_id)
if account.balance < amount do
{:error, :not_enough_money}
else
account = Map.update!(account, :balance, &(&1 - amount))
put_account(account)
:ok
end
end)
end
defp deposit(account_id, amount) do
Mutex.under(@mutex, {Account, account_id}, fn ->
{:ok, account} = get_account(account_id)
account = Map.update!(account, :balance, &(&1 + amount))
put_account(account)
:ok
end)
end
@account_id 1234
test "Massive concurrency" do
account = %Account{id: @account_id, balance: 0}
put_account(account)
IO.puts("Launching commands")
iterations = 10_000
withdrawal = exec_command_n({:withdrawal, fn -> withdrawal(@account_id, 12) end}, iterations)
deposit_1 = exec_command_n({:deposit, fn -> deposit(@account_id, 4) end}, iterations)
deposit_2 = exec_command_n({:deposit, fn -> deposit(@account_id, 4) end}, iterations)
deposit_3 = exec_command_n({:deposit, fn -> deposit(@account_id, 4) end}, iterations)
IO.puts("Awaiting commands")
Task.await(withdrawal, :infinity)
Task.await(deposit_1, :infinity)
Task.await(deposit_2, :infinity)
Task.await(deposit_3, :infinity)
case get_account(@account_id) do
{:ok, %Account{balance: balance}} ->
assert(balance == 0)
other ->
raise "Unexpeced result"
end
end
defp exec_command_n(command, iterations) do
Task.async(fn ->
# Process.sleep(100)
exec_retry_n(command, iterations)
end)
end
defp exec_retry_n(command, 0),
do: :ok
defp exec_retry_n({name, fun} = command, iterations) when iterations > 0 do
case fun.() do
:ok ->
# IO.puts("Command succeeded: #{name}")
exec_retry_n(command, iterations - 1)
{:error, reason} ->
# IO.puts("error running command #{name}\n reason: #{inspect(reason)}")
Process.sleep(50)
exec_retry_n(command, iterations)
end
end
end
1 Like
Thank you for looking into it too.
I will not look further into this because at this level of concurrency, using a Mutex is not a good solution. It was just a dummy speed test.
Would you mind run my project juste once to see if you have the same issue ? If yes I’ll upload it to github this week.
Edit: problem disapeared when upgrading to CubDB 0.13, even though reading the changes I cannot guess why.
One comment: if you want to atomically check the balance and then update it, you can use get_and_update/3. That way, you can avoid using the Mutex:
defp withdrawal(account_id, amount) do
get_and_update(@db_name, {Account, account_id}, fn
nil ->
{{:error, "No account with id: #{account_id}"}, nil}
%{ balance: balance } when balance < amount ->
{{:error, :not_enough_money}, account}
account ->
{:ok, Map.update!(account, :balance, &(&1 - amount))}
end
end)
end
Similarly, deposit can be implemented like this:
defp deposit(account_id, amount) do
get_and_update(@db_name, {Account, account_id}, fn
nil ->
{{:error, "No account with id: #{account_id}"}, nil}
account ->
{:ok, Map.update!(account, :balance, &(&1 + amount))}
end)
end
Yep, I know !
The point was to have many concurrent updates, doing heavy jobs between fetching and saving.
I believe the fun runs inside the CubDB process and I did that in order to avoid serializing multiple updates. Am I wrong ?
Also I may use different repositories (ETS, CubDB, or Ecto) so the mutex is a simple way to queue updates to a common entity.
I see now, you want to avoid blocking when another update is occurring, but on a different account.
If you end up with some interesting benchmark I’d be curious to see the results
Also, your mutex solution skips the write when nothing needs to be changed (e.g. when the balance is too low). This gave me me an idea for an improvement to CubDB: currently there is no way to tell get_and_update to “bail out” and avoid updating. Such an option would be useful in this case, and in all cases when one is implementing some sort of “compare and swap” semantics.
… but on a different account.
Exactly.
Now for the benchmarks I think my solution would be interesting only with heavy works inside the transaction. I am building games (hobby project) so we update a big game state. I bet having only one GenServer call (to get_and_update_mutli) is faster for many simple updates (especially for simple additions like bank account balances obvioulsy – that is just test stuff).
If I find time for that I will setup a scenario and calculate different pairs of concurrency/update-time where a mutex would be better.
1 Like
Hello! Thanks for the very interesting product, I’m going to use it with my current project.
Could you please clarify:
This example doesn’t work, does it? nil is greater than any number because it is a special atom. In a case with integer IDs select/2 options should be min_key: {:users, 0}, max_key: {:users, nil, nil} I believe. But this example should reliably work with any other term beside a number and an atom.
An atom case is a kind of tricky: nil is also greater than any atoms prior to :nil:
iex(48)> nil > :a
true
iex(49)> nil > :aaa
true
iex(50)> nil > :nik
true
iex(51)> nil == :nil
true
iex(52)> nil < :nim
true
Sorry for so obvious remark but I didn’t pay attention to this fact and spent some time trying to get why this example doesn’t work in my case.
1 Like
Hi @heathen,
you are right, re-reading my post above I made some mistake, not sure what I had in mind.
If you have keys like {:foo, pos_integer}, then the correct way to select them all is min_key: {:foo, 0}, max_key: {:foo, nil}, leveraging the fact that nil is bigger than all integers.
A more detailed explanation is in the how to section of the documentation.
1 Like
Also, as this thread is now updated again, I take the chance to announce that the release candidate for CubDB 1.0.0 is now on Hex as v1.0.0-rc.1 
. It is the result of running CubDB in production on embedded devices for the past year, and introduces a few improvements that make the API more solid. The notable changes are:
- The database is 100% compatible with the previous releases (I have no intention to break compatibility there).
- Auto compaction and auto file sync are now the defaults: I decided to go for safer defaults, that should be good for the vast majority of cases, and let users tune it for maximum performance in special cases. More info about compaction and file sync are here
- The timeouts (for example in
selectorget_and_update_multi) are now enforced on the callee side too, freeing up resources immediately when a timeout elapses. Their API is also slightly different, astimeoutis now passed as an option, instead of a separate argument.
For most users, the changes needed to update to v1.0.0(-rc.x) are minimal: just review if the new defaults are ok for you (or set them explicitly), and, only in case you are using explicit timeouts, adapt calls to select and get_and_update_multi. I will write a proper release post on this forum when 1.0.0 is out, but I wanted to inform in advance people that are reading this thread and that expressed interest in CubDB.
13 Likes
Yeah - benefit of using a separate lib like snappy is marginal and makes it much harder to build your DB because contributors now need to grok a native code library as well as your code.
2 Likes
Thanks very much for making CubDB available, @lucaong. I’ve just added it to a new Phoenix project that will read and locally store data from an external CMS, and am looking forward to getting it into production over the next few weeks.
In case it helps anyone else get started slightly quicker, I found that @Qqwy’s example of how to have CubDB started as part of the supervision tree with just MyApp.DB wasn’t working (I assume CubDB’s start_link function has since changed), and that changing it to the following worked:
defmodule MyApp.DB do
def child_spec(_) do
%{
id: __MODULE__,
start: {CubDB, :start_link, [
Application.get_env(:my_app, :my_db_location, "data/my_db"),
[
auto_file_sync: true,
auto_compact: true,
name: __MODULE__
]
]}
}
end
end
4 Likes
Thanks a lot for the kind words and for the working example @BrightEyesDavid !
As a side note, I am soon going to release version v1.0.0, after I allowed the latest release candidate quite a lot of time running in the wild to detect potential issues. It will be exactly identical to the latest release candidate, so no code changes necessary in order to upgrade.
3 Likes
According to the docs I can select with a select function and min/max keys.
Would it be possible to implement a match-spec for the keys like this ?:
Like “I want all keys that are a 3-tuple where the first element is
__MODULE__, the second element is:some_key, and with any 3rd element”
The 3rd element in the tuple can be a number or a string. I don’t know how I would set max key for that.
I’d like that as well. I recently toyed with keys like {:products, uuid} and the first thing bigger than {:products, binary} is {:productt, smallest_number}, which is a bit awkward.
