<turbo-stream action="append" target="posts_list"><template>    <div class="postbit" id="206832" data-post-id="206832">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="luizdamim" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/luizdamim/120/458_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  luizdamim
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Are the SSE-C keys a requirement from your customers or your own architecture? And how many concurrent downloads are you expecting?</p>
<p>A crazy idea, but it’s worth sharing… (only valid depending on your responses above)</p>
<p>You can spin up new S3 buckets on demand, copying files using your full security model and deleting those buckets after the download ends or after a regular interval.</p>
<p>This way, the clients can download files using a much more relaxed security model (but still secure) pre-signed URLs.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206832" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/12">Post #11</a>
	                </div>
	            </div>
              <div id="likers-container-206832" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206832"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #11"></div>
  </section>
</div>
    <div class="postbit" id="206839" data-post-id="206839">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="evadne" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/evadne/120/13949_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  evadne
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>There is a default limit of 100 buckets per account unless additional quota is requested. Creation of a new S3 bucket takes a few seconds to a few minutes.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206839" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/13">Post #12</a>
	                </div>
	            </div>
              <div id="likers-container-206839" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206839"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #12"></div>
  </section>
</div>
    <div class="postbit" id="206840" data-post-id="206840">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="josevalim" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josevalim/120/1787_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  josevalim
                  </h3>
		          </div>
						
			          <div class="user-title">
									<span>Creator of Elixir</span>
			          </div>
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="evadne" data-post="6" data-topic="38211">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/evadne/48/13949_2.png" class="avatar"> evadne:</div>
<blockquote>
<p><code>ExAws.S3.download_file/4</code>: Defaults to a concurrency of 8, chunk size of 1MB, and a timeout of 1 minute.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="ryanwinchester" data-post="11" data-topic="38211">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ryanwinchester/48/9472_2.png" class="avatar"> ryanwinchester:</div>
<blockquote>
<p>Streaming <strong>2.4GB</strong> <code>zip</code> from S3 through locally running app, through browser:</p>
</blockquote>
</aside>
<p>The concurrency of 8 for download_file is going to increase the memory size (and likely the amount of data references). If the goal is to stream, I would try to go with Finch (and/or Mint) + Range queries.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206840" data-batch-url="/posts/batch_likers">
                        4
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/14">Post #13</a>
	                </div>
	            </div>
              <div id="likers-container-206840" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206840"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #13"></div>
  </section>
</div>
    <div class="postbit" id="206869" data-post-id="206869">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="luizdamim" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/luizdamim/120/458_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  luizdamim
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>You can increase the maximum limit to 1000 by sending them an email (at least that’s stated in the documentation).</p>
<p>Depending on the implementation, you can work in the background and notify the user when everything’s read. I know it’s far from ideal, but it’s a possible solution given the constraints.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206869" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/15">Post #14</a>
	                </div>
	            </div>
              <div id="likers-container-206869" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206869"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #14"></div>
  </section>
</div>
    <div class="postbit" id="206909" data-post-id="206909">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="philss" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/philss/120/20089_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  philss
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<aside class="quote no-group" data-username="ryanwinchester" data-post="11" data-topic="38211">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ryanwinchester/48/9472_2.png" class="avatar"> ryanwinchester:</div>
<blockquote>
<p>Streaming <strong>2.4GB</strong> <code>zip</code> from S3 through locally running app, through browser:</p>
</blockquote>
</aside>
<p>Cool! I made a small PoC using <code>aws-elixir</code> and <code>Finch</code> that kept the memory usage <a href="https://youtu.be/BVwg0Ecm7nQ" rel="noopener nofollow ugc">even lower</a> (ends in <a href="https://youtu.be/BVwg0Ecm7nQ?t=244" rel="noopener nofollow ugc">4m05</a>). It’s not a fair comparison because it’s not streaming to the client.</p>
<p>Here is the code: <a href="https://github.com/philss/aws-s3-stream-download-poc/blob/main/lib/download_manager.ex" class="inline-onebox" rel="noopener nofollow ugc">aws-s3-stream-download-poc/lib/download_manager.ex at main · philss/aws-s3-stream-download-poc · GitHub</a> (there is a custom <a href="https://hexdocs.pm/aws/AWS.HTTPClient.html" rel="noopener nofollow ugc"><code>AWS.HTTPClient</code></a> implementation using Finch inside the project).</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206909" data-batch-url="/posts/batch_likers">
                        10
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/16">Post #15</a>
	                </div>
	            </div>
              <div id="likers-container-206909" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206909"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-solved cat-solved" title="Marked as solution"></div>
  </section>
</div>
    <div class="postbit" id="206963" data-post-id="206963">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="ryanwinchester" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ryanwinchester/120/9472_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  ryanwinchester
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>This is excellent!</p>
<p>I’m definitely going to try this out soon, too.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206963" data-batch-url="/posts/batch_likers">
                        1
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/17">Post #16</a>
	                </div>
	            </div>
              <div id="likers-container-206963" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206963"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #16"></div>
  </section>
</div>
    <div class="postbit" id="206990" data-post-id="206990">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="alexandre" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/alexandre/120/678_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  alexandre
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Some months ago I wrote a stream <a href="https://gist.github.com/alexandremcosta/29109fd65f413aeb02b6211ee7707576" rel="noopener nofollow ugc">downloader</a> with Mint.</p>
<p>I had the opposite problem: stream from clients to S3. I needed to control the chunk size since S3 limits to 5MB.</p>
<p>That was accomplished with:</p>
<pre data-code-wrap="elixir"><code class="lang-elixir">    url
    |&gt; Downloader.stream_body!()
    |&gt; Downloader.chunk_bytes(5_000_000)
    |&gt; ExAws.S3.upload(s3_bucket, filename, opts)
    |&gt; ExAws.request!()

</code></pre> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="206990" data-batch-url="/posts/batch_likers">
                        6
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/18">Post #17</a>
	                </div>
	            </div>
              <div id="likers-container-206990" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="206990"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #17"></div>
  </section>
</div>
    <div class="postbit" id="207031" data-post-id="207031">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="dimitarvp" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dimitarvp/120/38664_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  dimitarvp
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Really cool, thanks for sharing this.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="207031" data-batch-url="/posts/batch_likers">
                        2
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/19">Post #18</a>
	                </div>
	            </div>
              <div id="likers-container-207031" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="207031"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #18"></div>
  </section>
</div>
    <div class="postbit" id="209032" data-post-id="209032">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="ryanwinchester" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ryanwinchester/120/9472_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  ryanwinchester
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Is this how I use <code>AWS.S3</code> with SSE-C?</p>
<p>          <a href="https://p196.p4.n0.cdn.getcloudapp.com/items/6qu8DNz0/1f8970fb-4667-4788-8ee2-a39efe75a389.png?v=8386fbe1d8dba3a3c7444bfd99cba83e" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://p196.p4.n0.cdn.getcloudapp.com/items/6qu8DNz0/1f8970fb-4667-4788-8ee2-a39efe75a389.png?v=8386fbe1d8dba3a3c7444bfd99cba83e" width="690" height="204">
          </a>
</p>
<p>I’m getting a <code>403</code>, and can’t tell if I’m using the function arguments wrong or if I’m deriving the key incorrectly, so I don’t know where to waste my time <img src="https://forum.elixirforum.com/images/emoji/apple/sweat_smile.png?v=15" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="209032" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/20">Post #19</a>
	                </div>
	            </div>
              <div id="likers-container-209032" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="209032"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #19"></div>
  </section>
</div>
    <div class="postbit" id="209101" data-post-id="209101">
  <section>
    <div class="post-wrap">


					<div class="post-header">
		        <div class="user-avatar">
		          <img alt="ryanwinchester" src="https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ryanwinchester/120/9472_2.png" width="120" height="120" />
		        </div>
					
						<div class="user-details">
		          <div class="user-name">
		            <h3>
                  ryanwinchester
                    <span class="op-star" title="Thread Starter">
                      <img alt="OP" class="op-star-icon" src="/assets/thread-icons/thread-icon-thread-starter-df91e872.png" />
                    </span>
                  </h3>
		          </div>
						
						</div>
					
					</div>

	        <div class="thread-main">
	            <div class="post-body" data-turbo="false">
								<p>Update: Modified to use <code>get_object</code> so I can see a better error message and it’s a signature mismatch error.</p> 
	            </div>

	            <div class="base-line">
	                <div class="thread-counters">
	                    <span class="thread-count count-likes js-likers-trigger" title="Likes" data-post-id="209101" data-batch-url="/posts/batch_likers">
                        0
                      </span>
                      <!-- <span class="thread-count js-solved-indicator" title="Marked as solution"></span> -->
	                </div>
	                <div class="go-to-post">
	                  <a title="Go to post" alt="Go to post" href="https://forum.elixirforum.com/t/streaming-from-s3-to-client-with-constant-memory/38211/21">Post #20</a>
	                </div>
	            </div>
              <div id="likers-container-209101" 
                   class="likers-container"
                   data-first-post="false"
                   data-batch-url="/posts/batch_likers">
                   <div class="likers-placeholder" 
                     data-likers-post-id="209101"
                     data-batch-url="/posts/batch_likers">
                  <div class="post-likers"></div>
                </div>
              </div>
	        </div>
			

    </div>

    <div class="triangle-top-right type-standard-post cat-standard-post" title="Post #20"></div>
  </section>
</div>
</template></turbo-stream><turbo-stream action="replace" target="load-more-container"><template><div id="load-more-container" class="load-more-container">
    <a class="load-more-button" data-turbo-stream="true" href="/topics/38211/load_more?page=3">Load more posts (1 remaining)</a>
</div></template></turbo-stream>