How to stream file from aws to client through elixir backend

Are you using ExAws.S3 ?

In the past I did a service like wetransfer, using S3. So, if you want to use your application just for authorisation you can use a presigned url.

client → (your backend generate a presigned url) —> redirect the client to this url —> client downloads

In this way the client downloads directly from S3, without using bandwidth of your servers.

I was about to suggest presigned urls as well, but for the timespan those are valid they’re indeed shareable. I’m not sure if @benonymus wants that or does really need authentication on each request to the resource.

Thanks for both of the answers, we decided not to use presigned urls, so I am interested in a different solution
I was told that it is possible to stream the data from amazon through the backend using http chunking, but I dont know that how will that give a new url to the client
So far I got this:

HTTPoison.get!(x, %{}, stream_to: self())

and this gives this as a result

%HTTPoison.AsyncResponse{id: #Reference<0.528039844.3377725442.160394>}

The only other way I’d see, is to download in a stream and pipe the bytes down to the client, but this would mean that you have to pay the bill, twice!

yes, so basically instead of aws → client we do
aws → backend → client
right?

Yes. That’s the way you would do it.

it depends if they have or not the servers on aws. Outbound traffic between EC2 instances and S3 is free. But sure, from EC2 to client it’s paid (and quite expensive)

@benonymus why you avoid presigned urls?

Sorry guys for the “withdrawn” posts. I hit reply while I was trying to quote your posts.

Elixir makes this really easy thanks for concurrency. Do you have everything on AWS? Or do you have the servers with a different cloud provider?

everything is on aws yes, so the thing is that I am not sure how to go about it technially, is there any good article on it?

Presigned urls (requires methods to maintain url’s and refresh them, can result in weird caching problems when url’s expire, this was the conclusions we came with this option