<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="354143" data-post-id="354143">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dimitarvp" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/120/38664_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dimitarvp
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="silverdr" data-post="9" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/silverdr/48/23771_2.png" class="avatar"> silverdr:</div>
<blockquote>
<p>And I kind of hoped that I can use that “model” as a custom type, where I would implement the callbacks (especially <code>dump/1</code>) serialising the map into compact-ed JSON to send it to DB this way.</p>
</blockquote>
</aside>
<p>Don’t get me wrong, you can and you should. My main point is that you probably should not go for a singular God field.</p>
<p>Consider these sum types (expressed as Rust enums):</p>
<pre data-code-wrap="rust"><code class="lang-rust">enum SocialNetwork {
  GitHub,
  GitLab,
  BitBucket,
  LinkedIn,
  // ...
}

enum UserProfile {
  NormalUser {
    first_name: String,
    last_name: String,
    email: String
  },
  SocialNetworkUser {
    profile_id: String,
    social_network: SocialNetwork
  }
}
</code></pre>
<p>I’d advocate for you to have two DB fields: <code>type</code> and <code>content</code>, whereas type would be <code>"normal_user"</code> or <code>"social_network_user"</code> and contents – everything else.</p>
<p>What I was trying – unsuccessfully – to convey to you is to separate at least types and contents, multiple times if necessary. You already mentioned these types of data will be in flux for a while so maybe a singular God field is the best option and you should just focus on app-level validation. Up to you. Maintainability should be the first priority.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="354143" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-schema-ecto-changeset-optional-conditionally-present-fields/68817/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-354143" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="354143"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="354160" data-post-id="354160">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="silverdr" data-post="11" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/silverdr/48/23771_2.png" class="avatar"> silverdr:</div>
<blockquote>
<p>Still I don’t feel like using associated table for it because a) for 1:1 relation that means unnecessarily more complex and less performant queries, and b) the config schema/structure is being actively developed and iterated upon</p>
</blockquote>
</aside>
<p>I think this is very reasonable. Don’t take my reductive opinion <em>too</em> seriously <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I agree config data tends to be further on the “unstructured” side of the spectrum, though I don’t have any formal theories as to why (it really just is). Personally I’ve taken to storing such data as pure key/value pairs in the database (one KV per row) and then validating them (and choosing defaults) at runtime. This was the only approach that offered enough flexibility for me, and I think it’s in line with most configuration systems in general (it is very common to store configs as yml or json files for non-web apps, which are very unstructured formats).</p>
<p>I did try an embedded schema approach first (as you have), but there is actually another performance problem lurking which you should probably consider: it is impossible to selectively update the JSON blobs in an RDBMS (I’m using Postgres, probably you as well).</p>
<p>So every update has to rewrite the entire config. I wanted to keep the option open to store a <em>lot</em> of config pairs if needed (it’s somewhat involved but my configs are often per-object), and I didn’t want to have to rewrite all of the other options each time. Plus, there’s the fact that Ecto dumps all the null keys into the blob too (which you have, of course, discovered).</p>
<p>Even worse, Postgres has a habit (due to its ancient and bad MVCC storage) of duplicating tuples every time you write a new one, so this is even more costly than it seems. They will eventually be vacuumed of course (which is not free either!). Note that this is not necessarily as bad as it sounds <a href="https://www.postgresql.org/docs/current/storage-hot.html" rel="noopener nofollow ugc">because of HOT</a>, which should kick in because I doubt you are indexing your config blobs!</p>
<p>But anyway, if you imagine having a hundred or a thousand config keys, one big blob starts to sound like a pretty bad idea.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="354160" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-schema-ecto-changeset-optional-conditionally-present-fields/68817/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-354160" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="354160"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="354189" data-post-id="354189">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="silverdr" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/silverdr/120/23771_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  silverdr
                      <span class="op-star" title="Thread Starter">
                        <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                      </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="dimitarvp" data-post="12" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/48/38664_2.png" class="avatar"> dimitarvp:</div>
<blockquote>
<p>Don’t get me wrong, you can and you should</p>
</blockquote>
</aside>
<p>The question is “how”. I don’t see any straightforward path towards that, unless I drop the <code>embeds_one</code> from my “parent” model.</p>
<aside class="quote no-group" data-username="dimitarvp" data-post="12" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/48/38664_2.png" class="avatar"> dimitarvp:</div>
<blockquote>
<p>My main point is that you probably should not go for a singular God field.</p>
</blockquote>
</aside>
<p>I see, roger.</p>
<aside class="quote no-group" data-username="garrison" data-post="13" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>So every update has to rewrite the entire config</p>
</blockquote>
</aside>
<p>Thanks for pointing out. This doesn’t bother me that much though as this config is read-heavy. If it’s get completely replaced once in a while doesn’t matter so much as reading it “all the time”.</p>
<aside class="quote no-group" data-username="garrison" data-post="13" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>Plus, there’s the fact that Ecto dumps all the null keys into the blob too (which you have, of course, discovered).</p>
</blockquote>
</aside>
<p>Yup, precisely that.</p>
<aside class="quote no-group" data-username="garrison" data-post="13" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>I doubt you are indexing your config blobs!</p>
</blockquote>
</aside>
<p>No, I don’t <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> I index data, by which I want to retrieve (possibly aggregated) information. I don’t need to aggregate any information on configs as of now <img src="https://forum.elixirforum.com/images/emoji/apple/wink.png?v=15" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="garrison" data-post="13" data-topic="68817">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/g/3bc359/48.png" class="avatar"> garrison:</div>
<blockquote>
<p>But anyway, if you imagine having a hundred or a thousand config keys, one big blob starts to sound like a pretty bad idea</p>
</blockquote>
</aside>
<p>It does. While I am not in the thousand keys range and don’t expect to ever get there. The hundred one doesn’t seem completely improbable. Which is the main reason I started this thread – how do I store only what is needed for a given configuration rather than just “everything”, including 90% of noise.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="354189" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-schema-ecto-changeset-optional-conditionally-present-fields/68817/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-354189" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="354189"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-most-liked cat-most-liked" title="One of the top 3 liked posts in this thread!"></div>
  </section>
</div>
    <div class="postbit" id="354214" data-post-id="354214">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="garrison" src="/assets/icons/user-9f439610.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  garrison
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>To expand a bit on what I did, and to be clear I’m not necessarily suggesting <em>you</em> do this (it depends on your use case), but I essentially decided to bypass Ecto for my config storage. I wrote a Config module that directly interfaces with the underlying key/value rows and then just wrote my own load/dump functions to encode whatever types I want (I think I still only have strings and bools - maybe some day I’ll need numbers, but probably not much more than that).</p>
<p>Throw in a function to grab multiple keys at once (to avoid round trips) and that’s about all I ended up needing for basic config functionality. And again, for my use case I really like that I can write keys <em>individually</em> without having to re-encode the blob. Validation is done on the way out which means I can, for example, set a default with a simple <code>Config.get(user, :string, "key", "default")</code> and the data is still sparse in the DB.</p>
<p>This is just one path you could take of course, and if you’ve already shipped something then you probably wouldn’t want to bother migrating. Just offering another perspective <img src="https://forum.elixirforum.com/images/emoji/apple/slight_smile.png?v=15" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="354214" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/ecto-schema-ecto-changeset-optional-conditionally-present-fields/68817/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-354214" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="354214"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-last-post cat-last-post" title="Last post!"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <span class="all-loaded">— All posts loaded —</span>
</div></template></turbo-stream>