<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="220045" data-post-id="220045">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lucaong" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/120/21794_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lucaong
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi <a class="mention" href="/u/exadra37" rel="nofollow">@Exadra37</a> , thanks for your observation. <code>CubDB</code> does not use delayed writes, so when setting <code>auto_file_sync: true</code> a file sync is programmatically triggered at the end of each transaction. As you noted, this strong durability guarantee comes at a performance cost. Setting <code>auto_file_sync: false</code> conversely trades durability for performance.</p>
<p>When persisting configuration, it makes sense to keep durability, as performance won’t be a problem with sporadic writes. When using <code>CubDB</code> for data logging, or other high volume write use case, it might make sense to choose performance instead.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220045" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/93">Post #92</a>
	                </div>
	            </div>
              <div id="likers-container-220045" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220045"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #92"></div>
  </section>
</div>
    <div class="postbit" id="220047" data-post-id="220047">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Sebb" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Sebb/120/21818_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Sebb
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="lucaong" data-post="92" data-topic="23397">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/48/21794_2.png" class="avatar"> lucaong:</div>
<blockquote>
<p>what you describe looks exactly like what I wanted to implement for automated stress testing.</p>
</blockquote>
</aside>
<p>I will set the hardware up and give it a try if you are willing to help implement the test-cases.</p>
<aside class="quote no-group" data-username="lucaong" data-post="92" data-topic="23397">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/48/21794_2.png" class="avatar"> lucaong:</div>
<blockquote>
<p><code>CubDB</code> data structure is an append-only B+tree saved in a single file</p>
</blockquote>
</aside>
<p>This assumes, that you can’t corrupt data already written when writing new data. Are you sure thats the case? I could image sth like this:</p>
<p><code>D</code> - data already written<br>
<code>F</code> - “empty”<br>
<code>E</code> - new data</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">(1)
--- flash page start ---
DDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDD
FFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFF
--- flash page end ---
</code></pre>
<p>Now we append to the file, therefore the flash controller reads the begin of the page into ram, deletes the page (prepare for write), resulting in</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">(2)
--- flash page start ---
FFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFF
--- flash page end ---
</code></pre>
<p>and now appends the data</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">(3)
--- flash page start ---
DDDDDDDDDDDDDDDDDD
DDDDDDDDDDDDDDDDDD
EEEEEEEEEEEEEEEEEE
FFFFFFFFFFFFFFFFFF
--- flash page end ---
</code></pre>
<p>If the power fails after (2) there is a problem.<br>
<strong>I’ve no knowledge of how flash controllers really work</strong>, but you could build a controller this way.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220047" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/94">Post #93</a>
	                </div>
	            </div>
              <div id="likers-container-220047" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220047"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #93"></div>
  </section>
</div>
    <div class="postbit" id="220048" data-post-id="220048">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lucaong" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/120/21794_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lucaong
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p><a class="mention" href="/u/sebb" rel="nofollow">@Sebb</a> it’s also worth noting that the test suite includes some tests that truncate the end of the data file in random ways, simulating a partial write because of a power failure, and assert that the database can still be opened and traversed.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220048" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/95">Post #94</a>
	                </div>
	            </div>
              <div id="likers-container-220048" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220048"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #94"></div>
  </section>
</div>
    <div class="postbit" id="220049" data-post-id="220049">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lucaong" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/120/21794_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lucaong
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>As long as the file is corrupted <em>at the last page</em> <code>CubDB</code> can recover it. Since the file is only appended, only the last page can be affected by a corruption.</p>
<p>Flash drives typically can delete no less than a full block, while they can write a single page (smaller than a block), therefore it would not make sense for a disk controller to erase before append-writing.</p>
<p>The details of how appending to a file works at the low level are down to the specific disk controller. Some will append writes to the next page, leaving “holes” if the previous page is incomplete. Some might work differently, but they should still write a page atomically. In the end though, at this level, the responsibility of ensuring sanity of written pages resides on the disk controller, not on the application code.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220049" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/96">Post #95</a>
	                </div>
	            </div>
              <div id="likers-container-220049" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220049"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #95"></div>
  </section>
</div>
    <div class="postbit" id="220050" data-post-id="220050">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Sebb" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Sebb/120/21818_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Sebb
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>For my use case it is OK to lose configuration done directly before power-fail.<br>
But it is not OK to lose configuration that once worked.<br>
Say I configure the device today, and it works. The Config ends up in the first half of a flash page.<br>
Tomorrow I do some more configuration but power fails and now my configs from yesterday may be lost. That is not acceptable.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220050" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/97">Post #96</a>
	                </div>
	            </div>
              <div id="likers-container-220050" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220050"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #96"></div>
  </section>
</div>
    <div class="postbit" id="220051" data-post-id="220051">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="NobbZ" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/NobbZ/120/27235_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  NobbZ
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Sounds well thought, though let me give you some words of warning! Some filesystems don’t cope well with that approach.</p>
<p>I’ve seen ext4 truncating files to zero bytes when they’ve been actively written to at the time of an power outage.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220051" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/98">Post #97</a>
	                </div>
	            </div>
              <div id="likers-container-220051" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220051"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #97"></div>
  </section>
</div>
    <div class="postbit" id="220055" data-post-id="220055">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Sebb" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Sebb/120/21818_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Sebb
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Do you have any insight why this happens?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220055" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/99">Post #98</a>
	                </div>
	            </div>
              <div id="likers-container-220055" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220055"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #98"></div>
  </section>
</div>
    <div class="postbit" id="220056" data-post-id="220056">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lucaong" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lucaong/120/21794_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lucaong
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Thank you all for the good points. At the end of the day, one can strive to recover from anything that is recoverable at the level of the library code, using the file system primitives appropriately <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> the responsibility of ensuring durability and avoiding corruption at the low level ultimately resides with the file system and disk controller.</p>
<p>If a particular disk controller or file system does dangerous stuff like truncating a file to zero bytes upon write or non-atomically updating a disk page, I am not sure how any approach could possibly ensure durability.</p>
<p><code>CubDB</code> is designed around a simple principle that is logically understandable and easy to debug. My opinion is that more complicated approaches would introduce more possible failure modes while not solving those aforementioned degenerate cases.</p>
<p>The approach that <code>CubDB</code> follows is the same of CouchDB, so the latter experience can serve as an example. Note that earlier pre-1.0.0 versions of <code>CubDB</code> did get user reports of file corruption in corner cases, which were addressed: some users do have high volume write use cases.</p>
<p>Stress testing like proposed by <a class="mention" href="/u/sebb" rel="nofollow">@Sebb</a> would be in my opinion the best approach to ensure that the approach is sound. Such stress test would still depend on the particular file system and disk used though.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220056" data-batch-url="/posts/batch_likers">
                        6
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/100">Post #99</a>
	                </div>
	            </div>
              <div id="likers-container-220056" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220056"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #99"></div>
  </section>
</div>
    <div class="postbit" id="220368" data-post-id="220368">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="mgibowski" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/mgibowski/120/21221_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  mgibowski
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Hi <a class="mention" href="/u/lucaong" rel="nofollow">@lucaong</a>,<br>
Firstly congrats on your project - CubDB looks good. However, I think there is an issue to be addressed.</p>
<p>My context is that I am advising a friend regarding the architecture of a Nerves-based project. The device will be accumulating data, and it will be very important that data is not lost and that it’s easy to perform backups. The data will not be relational and will be required to be accessible for long period (few years) because of legal reasons.</p>
<p>As this thread came up in the last few days, I considered CubDB for the persistence. However, after taking a closer look, there is one problem.</p>
<p>Currently CubDB uses <code>:erlang.term_to_binary</code> and <code>:erlang.binary_to_term</code> for serialization/deserialization.<br>
Unfortunately, it is not guaranteed data serialized in one OTP version can be correctly deserialized in another one.<br>
RabbitMQ once encountered <a href="https://bugs.erlang.org/browse/ERL-431" rel="nofollow">this situation</a>.</p>
<p>How this could be addressed? My suggestion would be to expose a serializer’s behavior in your library and allow alternatives. Just created a GitHub <a href="https://github.com/lucaong/cubdb/issues/38" rel="noopener nofollow ugc">issue</a> for that.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220368" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/101">Post #100</a>
	                </div>
	            </div>
              <div id="likers-container-220368" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220368"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #100"></div>
  </section>
</div>
    <div class="postbit" id="220374" data-post-id="220374">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="Sebb" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/Sebb/120/21818_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  Sebb
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<blockquote>
<p>It is fine if the encoded value is then decoded using <code>binary_to_term</code>, but is not fine if the binary is used as an identifier.</p>
</blockquote>
<p>would this problem concern CubDB after all?</p>
<p>You don’t need to always update an embedded system to the latest version immediately.<br>
Seems like there was a solution in the final version, right?<br>
I don’t think OTP will release sth that will break term-to-binary forever.</p>
<p>Serializer could be done, but will increase complexity and you’ll not find anything that can compete with term-to-binary in terms of robustness, speed and ease of use.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="220374" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/cubdb-a-pure-elixir-embedded-key-value-database/23397/102">Post #101</a>
	                </div>
	            </div>
              <div id="likers-container-220374" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="220374"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #101"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/23397/load_more?page=11">Load more posts (24 remaining)</a>
</div></template></turbo-stream>