benonymus
How to stream file from aws to client through elixir backend
Hey there,
I want to make my bucket closed, so people can’t get anything form it even with the links,
so I want to authenticate through the app (already done) and if that is passed I want to return the file with a new link, how would one go about this?
I read that chunking can be useful, but I don’t get the whole picture, that how will I have a new link for the client?
Most Liked
alvises
Are you using ExAws.S3 ?
In the past I did a service like wetransfer, using S3. So, if you want to use your application just for authorisation you can use a presigned url.
client --> (your backend generate a presigned url) —> redirect the client to this url —> client downloads
https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#presigned_url/5
In this way the client downloads directly from S3, without using bandwidth of your servers.
alvises
I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream.
defmodule HTTPDownload do
def stream!(url) do
Stream.resource(
fn -> start_request(url) end,
fn ref ->
case receive_response(ref) do
#returning the chunk to the stream
{:ok, {:chunk, chunk}} ->
HTTPoison.stream_next(ref)
{[chunk], ref}
{:ok, msg} ->
IO.inspect(msg)
HTTPoison.stream_next(ref)
{[], ref}
{:error, error} ->
IO.puts("ERROR")
raise("error #{inspect error}")
:done -> {:halt, ref}
end
end,
fn ref -> :hackney.stop_async(ref) end
)
end
defp start_request(url) do
{:ok, ref} = HTTPoison.get(url, %{}, stream_to: self(), async: :once)
ref
end
defp receive_response(ref) do
id = ref.id
receive do
%HTTPoison.AsyncStatus{code: code, id: ^id} when 200 <= code and code < 300 ->
{:ok, {:status_code, code}}
%HTTPoison.AsyncStatus{code: code, id: ^id} ->
{:error, {:status_code, code}}
%HTTPoison.AsyncHeaders{headers: headers, id: ^id}->
{:ok, {:headers, headers}}
%HTTPoison.AsyncChunk{chunk: chunk, id: ^id}->
{:ok, {:chunk, chunk}}
%HTTPoison.AsyncEnd{id: ^id}-> :done
end
end
end
So in this way you can get a stream from a http response and you can use it like this:
HTTPDownloader.stream!( url_to_my_s3_file )
|> Enum.each(fn chunk-> send_chunk_to_client(client_conn, chunk) end)
So, if you don’t want to deal yourself with AWS HTTP API, you can get the presigned url using the ex_aws_s3 library, and use the url in your backend to get the stream of chunks to send to the client.
LostKobrakai
I was about to suggest presigned urls as well, but for the timespan those are valid they’re indeed shareable. I’m not sure if @benonymus wants that or does really need authentication on each request to the resource.







