Streaming from S3 to client with constant memory?

You can use a pre-signed URL with a signature that has an insanely short period of validity. Sign the URL every time. Should be much simpler. Having traffic go through your host can create a massive headache when the time comes to pay your bills!

Another issue with the pre-signed URL is when using SSE-C (custom keys for server-side encryption) requires headers to be sent to AWS with the request, so you can’t just redirect them to the URL or give the link out (and also exposes the key to the downloader, although that’s not the end of the world).

I haven’t found a good way to initiate that in the browser without some janky web workers acting as a proxy or something, but my JS expertise is severely lacking these days. I’ve fallen so far behind on frontend stuff

Another idea

1. Generate pre-signed URL but keep it server-side
2. Serve a stream to the client
3. When the client wants more data, you go and download a few megs from AWS and serve that down the stream

This way you can use whatever that reads chunks off HTTP and you can have more control over how big each chunk sent to the user should be, etc.

Also it should be possible to specify the key even in a pre-signed GET request for S3.

I’ll pay $200 USD to anybody who can tell me how to get a working pre-signed GET URL with SSE-C that requires no headers. I’ve wasted so much of my spare time on this already

TBF…

ExAws.S3.download_file/4: Defaults to a concurrency of 8, chunk size of 1MB, and a timeout of 1 minute.

So you should be ok. The stream isn’t started until the customer starts downloading anyway.

If you want to avoid the server-side round-trip, then you will have to expose the headers and start the download on the client with some JavaScript. You can use XMLHttpRequest to arrange the headers and get a blob back, then make a Data URI out of the blob, and serve it to the user… this works as long as the file is not large.

Can try GitHub - eligrey/FileSaver.js: An HTML5 saveAs() FileSaver implementation · GitHub for the saving part too

I’m probably going to start with a cap of 1GB filesize.

Related to FileSaver, I was looking at GitHub - jimmywarting/StreamSaver.js: StreamSaver writes stream to the filesystem directly asynchronous · GitHub but thought that a service worker seemed a bit janky. Maybe it’s better than streaming through my own server, though.

The bottom line IMO is whether you MUST use SSE-C and if it is acceptable in your use case to expose the headers to the client.

• [] [Re: SSE-C] It’s a must.
• [] [Re: headers] It’s not ideal, but I think it’s acceptable in this case.

Not an ideal solution, but use Nginx as a reverse proxy for S3 and extract header values from the URL params (or any other part of the query string).

Streaming 2.4GB zip from S3 through locally running app, through browser:

Screen Shot 2021-03-15 at 5.28.30 PM1836×1336 173 KB

Using

ExAws.S3.download_file/3
|> ExAws.stream!/1
|> Enum.reduce_while(Plug.Conn.send_chunked/3, &(...))

The memory usage wasn’t frightening, so might be okay, for now.

This is on a 6-core 2019 iMac with 32GB RAM. Not sure if there are any significant tradeoffs happening with memory usage if I go to a 1-2 vcpu environment…