benonymus

benonymus

How to stream file from aws to client through elixir backend

Hey there,
I want to make my bucket closed, so people can’t get anything form it even with the links,
so I want to authenticate through the app (already done) and if that is passed I want to return the file with a new link, how would one go about this?
I read that chunking can be useful, but I don’t get the whole picture, that how will I have a new link for the client?

Most Liked

alvises

alvises

Are you using ExAws.S3 ?

In the past I did a service like wetransfer, using S3. So, if you want to use your application just for authorisation you can use a presigned url.

client --> (your backend generate a presigned url) —> redirect the client to this url —> client downloads

https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#presigned_url/5

In this way the client downloads directly from S3, without using bandwidth of your servers.

alvises

alvises

I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream.

defmodule HTTPDownload do

  def stream!(url) do
    Stream.resource(
      fn -> start_request(url) end,
      fn ref -> 
        case receive_response(ref) do
          #returning the chunk to the stream
          {:ok, {:chunk, chunk}} -> 
            HTTPoison.stream_next(ref)
            {[chunk], ref}
          {:ok, msg} -> 
            IO.inspect(msg)
            HTTPoison.stream_next(ref)
            {[], ref}
          {:error, error} -> 
            IO.puts("ERROR")
            raise("error #{inspect error}")
          :done -> {:halt, ref}
        end
      end,
      fn ref -> :hackney.stop_async(ref) end
    )
  end


  defp start_request(url) do
    {:ok, ref} = HTTPoison.get(url, %{}, stream_to: self(), async: :once)
    ref
  end

  defp receive_response(ref) do
    id = ref.id
    receive do 
      %HTTPoison.AsyncStatus{code: code, id: ^id} when 200 <= code and code < 300 -> 
        {:ok, {:status_code, code}}
      %HTTPoison.AsyncStatus{code: code, id: ^id} ->
        {:error, {:status_code, code}}

      %HTTPoison.AsyncHeaders{headers: headers, id: ^id}->
        {:ok, {:headers, headers}}
      
      %HTTPoison.AsyncChunk{chunk: chunk, id: ^id}->
        {:ok, {:chunk, chunk}}

      %HTTPoison.AsyncEnd{id: ^id}-> :done
    end
  end
end

So in this way you can get a stream from a http response and you can use it like this:

HTTPDownloader.stream!( url_to_my_s3_file )
|> Enum.each(fn chunk-> send_chunk_to_client(client_conn, chunk) end)

So, if you don’t want to deal yourself with AWS HTTP API, you can get the presigned url using the ex_aws_s3 library, and use the url in your backend to get the stream of chunks to send to the client.

LostKobrakai

LostKobrakai

I was about to suggest presigned urls as well, but for the timespan those are valid they’re indeed shareable. I’m not sure if @benonymus wants that or does really need authentication on each request to the resource.

Where Next?

Popular in Questions Top

fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
jaysoifer
Is there a way to rollback a specific migration and only that one ("skipping" all the other ones)? Would mix ecto.rollback -v 2008090...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a &gt; b) do {:ok, "a"} end if (a &lt; b) do {:ok, b} end if (a == b) do {:ok, "eq...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New

Other popular topics Top

TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41454 115
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52238 488
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New

We're in Beta

About us Mission Statement