<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="381138" data-post-id="381138">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lcabrini" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lcabrini/120/32329_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lcabrini
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Oh dear, did I call it configuration? I tried so hard to stick to “settings”, to make sure it was NOT configuration I meant. Oh, well, my bad I guess.</p>
<aside class="quote no-group" data-username="derek-zhou" data-post="30" data-topic="73831">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/derek-zhou/48/19943_2.png" class="avatar"> derek-zhou:</div>
<blockquote>
<p>Any caching layer is a data consistency risk; and it appears that you do not need to access those information so frequently that it will materially hurt performance.</p>
</blockquote>
</aside>
<p>Yes, from the example I gave that is true. My question was more about how it would be done, not if I need it or not. I was asked to provide an example so I did. That seems to have been a mistake because it has shifted the discussion from “how would you achieve X?” to “do you need to do X?”.</p>
<p>I have been given a few pointers along the way and am looking into them.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381138" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/32">Post #31</a>
	                </div>
	            </div>
              <div id="likers-container-381138" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381138"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #31"></div>
  </section>
</div>
    <div class="postbit" id="381141" data-post-id="381141">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lcabrini" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lcabrini/120/32329_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lcabrini
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I know very little about SQLite when it comes to web development. I’ve mostly just used it for smaller projects, like a podcatcher I built for KDE many years ago. IIRC, org-roam for Emacs uses and SQLite database to maintain indexes of documents and tags. I may be wrong, I don’t use org-roam myself, since I prefer denote. But these are both examples of SQLite being used by a single user at a time. For that I don’t doubt it makes a lot of sense to use SQLite. But is it really viable for a site where you’d need multiple connections open?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381141" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/33">Post #32</a>
	                </div>
	            </div>
              <div id="likers-container-381141" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381141"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #32"></div>
  </section>
</div>
    <div class="postbit" id="381143" data-post-id="381143">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>What does “multiple connections” mean? If you just mean multiple (Elixir) processes then it’s fine. There is no write concurrency but you wouldn’t even notice that until you’re in the hundreds of TPS.</p>
<p>The biggest advantage of SQLite is that it’s just way easier to manage out of the box. No installation, no setup, no upgrades, no server, and so on. Backups are easy with Litestream.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381143" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/34">Post #33</a>
	                </div>
	            </div>
              <div id="likers-container-381143" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381143"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #33"></div>
  </section>
</div>
    <div class="postbit" id="381145" data-post-id="381145">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lcabrini" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lcabrini/120/32329_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lcabrini
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Well, they wouldn’t have to be Elixir processes, necessarily. psql also spawns an (OS) process. I basically mean the ability for mutliple processes, heavy or light, to be able to communicate with the database and the server ensuring that it is done in an orderly manner so that data does not get destroyed.</p>
<p>As I said, the only experience I have with SQLite is a few desktop and command-line utilities I wrote that used SQLite like in a single-user context. I never stated that SQLite cannot be used for more complex scenarios, merely that I was not aware of that.</p>
<p>I do remember that the Amarok developers decided to use MySQL instead of SQLite back in the KDE3 days, which kind of sucked because mysql became a hard requirement to use KDE (or at least Amarok). I don’t remember the exact reasoning, but I’m quite sure it had to do with accessing SQLite from mutlipled threads.</p>
<p>That being said, this is quite some time ago now. I’m sure things have changed now. Things change. Again, no offence meant against SQLite.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381145" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/35">Post #34</a>
	                </div>
	            </div>
              <div id="likers-container-381145" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381145"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #34"></div>
  </section>
</div>
    <div class="postbit" id="381149" data-post-id="381149">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>SQLite is generally not something you would use from multiple (os) processes. There are some cases where you would; Litestream is a separate process that backs up the database, for example.</p>
<p>On the BEAM the nif you’re using (e.g. Exqlite) will serialize the transactions through a “connection” for you. There is definitely a point where this approach will tap out performance-wise, but modern hardware is just so fast that you’re unlikely to get there.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381149" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/36">Post #35</a>
	                </div>
	            </div>
              <div id="likers-container-381149" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381149"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #35"></div>
  </section>
</div>
    <div class="postbit" id="381160" data-post-id="381160">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lcabrini" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lcabrini/120/32329_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lcabrini
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>I might be completely out of my depth here, but isn’t an SQLite “database” just a single file? If so, why the need for a backup tool? Isn’t <code>cp</code> all you need?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381160" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/37">Post #36</a>
	                </div>
	            </div>
              <div id="likers-container-381160" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381160"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #36"></div>
  </section>
</div>
    <div class="postbit" id="381161" data-post-id="381161">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lcabrini" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lcabrini/120/32329_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lcabrini
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="garrison" data-post="36" data-topic="73831">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>but modern hardware is just so fast that you’re unlikely to get there.</p>
</blockquote>
</aside>
<p>I come from a different era where hardware was THE problem. Hence, I tend to spend far too much time optimizing.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381161" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/38">Post #37</a>
	                </div>
	            </div>
              <div id="likers-container-381161" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381161"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #37"></div>
  </section>
</div>
    <div class="postbit" id="381162" data-post-id="381162">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>It is a single file (or a couple with WAL mode), but if the database is being actively written there is no guarantee that you will be reading a consistent snapshot. As the copy goes on you will be reading some old parts and some new parts and this will corrupt the backup.</p>
<p>Most databases (including SQLite) have built-in functionality to take a consistent backup that reflects a single point in time, and you could of course just use that. Litestream is a fancy tool that will stream a continuous backup to S3 (or another machine) so that you always have an indestructible backup which is no more than a few seconds out of date.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381162" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/39">Post #38</a>
	                </div>
	            </div>
              <div id="likers-container-381162" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381162"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #38"></div>
  </section>
</div>
    <div class="postbit" id="381204" data-post-id="381204">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="lcabrini" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lcabrini/120/32329_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  lcabrini
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>This makes a lot of sense. I’m just a bit confused about one thing. If I treat the database as the source of truth (which makes sense), why would I bother with cache at all? If I were to use a cache I’d assume I would read from cache, write to database and force cache to be reread. And if I’m reading from cache, would that not at least be accepting that cache is a temporary representation of truth? Or am I just not understanding something very basic?</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381204" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/41">Post #40</a>
	                </div>
	            </div>
              <div id="likers-container-381204" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381204"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #40"></div>
  </section>
</div>
    <div class="postbit" id="381233" data-post-id="381233">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="cmo" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/cmo/120/19618_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  cmo
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Unless someone edits the database directly, the cache shouldn’t get out of sync with the database in that design.</p>
<p>I would not bother with the cache just yet unless you’re reading these values rapidly. These reads are going to be very quick.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="381233" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/what-would-be-the-proper-way-to-implement-a-simple-persistent-key-value-store/73831/42">Post #41</a>
	                </div>
	            </div>
              <div id="likers-container-381233" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="381233"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>