benonymus
How to stream file from aws to client through elixir backend
Hey there,
I want to make my bucket closed, so people can’t get anything form it even with the links,
so I want to authenticate through the app (already done) and if that is passed I want to return the file with a new link, how would one go about this?
I read that chunking can be useful, but I don’t get the whole picture, that how will I have a new link for the client?
Most Liked
alvises
Are you using ExAws.S3 ?
In the past I did a service like wetransfer, using S3. So, if you want to use your application just for authorisation you can use a presigned url.
client → (your backend generate a presigned url) —> redirect the client to this url —> client downloads
In this way the client downloads directly from S3, without using bandwidth of your servers.
alvises
I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream.
defmodule HTTPDownload do
def stream!(url) do
Stream.resource(
fn -> start_request(url) end,
fn ref ->
case receive_response(ref) do
#returning the chunk to the stream
{:ok, {:chunk, chunk}} ->
HTTPoison.stream_next(ref)
{[chunk], ref}
{:ok, msg} ->
IO.inspect(msg)
HTTPoison.stream_next(ref)
{[], ref}
{:error, error} ->
IO.puts("ERROR")
raise("error #{inspect error}")
:done -> {:halt, ref}
end
end,
fn ref -> :hackney.stop_async(ref) end
)
end
defp start_request(url) do
{:ok, ref} = HTTPoison.get(url, %{}, stream_to: self(), async: :once)
ref
end
defp receive_response(ref) do
id = ref.id
receive do
%HTTPoison.AsyncStatus{code: code, id: ^id} when 200 <= code and code < 300 ->
{:ok, {:status_code, code}}
%HTTPoison.AsyncStatus{code: code, id: ^id} ->
{:error, {:status_code, code}}
%HTTPoison.AsyncHeaders{headers: headers, id: ^id}->
{:ok, {:headers, headers}}
%HTTPoison.AsyncChunk{chunk: chunk, id: ^id}->
{:ok, {:chunk, chunk}}
%HTTPoison.AsyncEnd{id: ^id}-> :done
end
end
end
So in this way you can get a stream from a http response and you can use it like this:
HTTPDownloader.stream!( url_to_my_s3_file )
|> Enum.each(fn chunk-> send_chunk_to_client(client_conn, chunk) end)
So, if you don’t want to deal yourself with AWS HTTP API, you can get the presigned url using the ex_aws_s3 library, and use the url in your backend to get the stream of chunks to send to the client.
LostKobrakai
I was about to suggest presigned urls as well, but for the timespan those are valid they’re indeed shareable. I’m not sure if @benonymus wants that or does really need authentication on each request to the resource.
Last Post!
benonymus
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









