ryanwinchester
Streaming from S3 to client with constant memory?
Greetings fellow humans,
First, I’ll tell you why I’m not using pre-signed URLs: because this is for sensitive files, that I want to control access to at a much more granular level than signed URLs allow.
I’ve been looking at aws-elixir and ex_aws_s3.
For now, I’m planning to use cloud functions, and want to keep memory limits as low as possible.
So, my objective is to stream a file from S3, to the client, while keeping memory usage constant and not running out.
It looks like with ex_aws_s3 I can do:
ExAws.S3.download_file(..., :memory)
|> ExAws.stream!()
|> Enum.reduce_while(Plug.Conn.send_chunked(), &()...)
but I’m worried about not being able to control the memory usage it could be a problem?
Another option @philss mentioned in aws-elixir in this issue is using a range option, which I guess would give more control.
What do you fine elixir folks think I should do?
Marked As Solved
philss
Cool! I made a small PoC using aws-elixir and Finch that kept the memory usage even lower (ends in 4m05). It’s not a fair comparison because it’s not streaming to the client.
Here is the code: aws-s3-stream-download-poc/lib/download_manager.ex at main · philss/aws-s3-stream-download-poc · GitHub (there is a custom AWS.HTTPClient implementation using Finch inside the project).
Also Liked
alexandre
Some months ago I wrote a stream downloader with Mint.
I had the opposite problem: stream from clients to S3. I needed to control the chunk size since S3 limits to 5MB.
That was accomplished with:
url
|> Downloader.stream_body!()
|> Downloader.chunk_bytes(5_000_000)
|> ExAws.S3.upload(s3_bucket, filename, opts)
|> ExAws.request!()
josevalim
The concurrency of 8 for download_file is going to increase the memory size (and likely the amount of data references). If the goal is to stream, I would try to go with Finch (and/or Mint) + Range queries.
philss
I’m getting a
403, and can’t tell if I’m using the function arguments wrong or if I’m deriving the key incorrectly, so I don’t know where to waste my time
The usage is correct. The problem was a bug in the signature algorithm, which was not related to SSE-C 
I managed to fix it on master: Fix signature bug when headers are similar · aws-beam/aws-elixir@306fe41 · GitHub
This week we may release a new version of “aws-elixir” to Hex.pm.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"