benonymus

benonymus

How to stream file from aws to client through elixir backend

Hey there,
I want to make my bucket closed, so people can’t get anything form it even with the links,
so I want to authenticate through the app (already done) and if that is passed I want to return the file with a new link, how would one go about this?
I read that chunking can be useful, but I don’t get the whole picture, that how will I have a new link for the client?

Most Liked

alvises

alvises

Are you using ExAws.S3 ?

In the past I did a service like wetransfer, using S3. So, if you want to use your application just for authorisation you can use a presigned url.

client → (your backend generate a presigned url) —> redirect the client to this url —> client downloads

In this way the client downloads directly from S3, without using bandwidth of your servers.

alvises

alvises

I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream.

defmodule HTTPDownload do

  def stream!(url) do
    Stream.resource(
      fn -> start_request(url) end,
      fn ref -> 
        case receive_response(ref) do
          #returning the chunk to the stream
          {:ok, {:chunk, chunk}} -> 
            HTTPoison.stream_next(ref)
            {[chunk], ref}
          {:ok, msg} -> 
            IO.inspect(msg)
            HTTPoison.stream_next(ref)
            {[], ref}
          {:error, error} -> 
            IO.puts("ERROR")
            raise("error #{inspect error}")
          :done -> {:halt, ref}
        end
      end,
      fn ref -> :hackney.stop_async(ref) end
    )
  end


  defp start_request(url) do
    {:ok, ref} = HTTPoison.get(url, %{}, stream_to: self(), async: :once)
    ref
  end

  defp receive_response(ref) do
    id = ref.id
    receive do 
      %HTTPoison.AsyncStatus{code: code, id: ^id} when 200 <= code and code < 300 -> 
        {:ok, {:status_code, code}}
      %HTTPoison.AsyncStatus{code: code, id: ^id} ->
        {:error, {:status_code, code}}

      %HTTPoison.AsyncHeaders{headers: headers, id: ^id}->
        {:ok, {:headers, headers}}
      
      %HTTPoison.AsyncChunk{chunk: chunk, id: ^id}->
        {:ok, {:chunk, chunk}}

      %HTTPoison.AsyncEnd{id: ^id}-> :done
    end
  end
end

So in this way you can get a stream from a http response and you can use it like this:

HTTPDownloader.stream!( url_to_my_s3_file )
|> Enum.each(fn chunk-> send_chunk_to_client(client_conn, chunk) end)

So, if you don’t want to deal yourself with AWS HTTP API, you can get the presigned url using the ex_aws_s3 library, and use the url in your backend to get the stream of chunks to send to the client.

LostKobrakai

LostKobrakai

I was about to suggest presigned urls as well, but for the timespan those are valid they’re indeed shareable. I’m not sure if @benonymus wants that or does really need authentication on each request to the resource.

Last Post!

benonymus

benonymus

ok basically i got it, I will show the solution shortly

Where Next?

Popular in Questions Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36820 110
New
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

We're in Beta

About us Mission Statement