like, now I have the urls to the files, but how will I get the files and then have new urls for the files to return to the client?
benonymus
alvises
For streaming using HTTPoison I wrote an article few weeks ago: Download Large Files with HTTPoison Async Requests, but it would be better to use directly a library like ExAws.S3. I can’t find a function to get an Elixir stream from an S3 object though…
Update
Am I wrong or here in the ExAws.S3 there is a massive overhead: ex_aws_s3/lib/ex_aws/s3/download.ex at main · ex-aws/ex_aws_s3 · GitHub ??
Each chunk seems to be requested with a separate http request..
1
alvises
I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream.
defmodule HTTPDownload do
def stream!(url) do
Stream.resource(
fn -> start_request(url) end,
fn ref ->
case receive_response(ref) do
#returning the chunk to the stream
{:ok, {:chunk, chunk}} ->
HTTPoison.stream_next(ref)
{[chunk], ref}
{:ok, msg} ->
IO.inspect(msg)
HTTPoison.stream_next(ref)
{[], ref}
{:error, error} ->
IO.puts("ERROR")
raise("error #{inspect error}")
:done -> {:halt, ref}
end
end,
fn ref -> :hackney.stop_async(ref) end
)
end
defp start_request(url) do
{:ok, ref} = HTTPoison.get(url, %{}, stream_to: self(), async: :once)
ref
end
defp receive_response(ref) do
id = ref.id
receive do
%HTTPoison.AsyncStatus{code: code, id: ^id} when 200 <= code and code < 300 ->
{:ok, {:status_code, code}}
%HTTPoison.AsyncStatus{code: code, id: ^id} ->
{:error, {:status_code, code}}
%HTTPoison.AsyncHeaders{headers: headers, id: ^id}->
{:ok, {:headers, headers}}
%HTTPoison.AsyncChunk{chunk: chunk, id: ^id}->
{:ok, {:chunk, chunk}}
%HTTPoison.AsyncEnd{id: ^id}-> :done
end
end
end
So in this way you can get a stream from a http response and you can use it like this:
HTTPDownloader.stream!( url_to_my_s3_file )
|> Enum.each(fn chunk-> send_chunk_to_client(client_conn, chunk) end)
So, if you don’t want to deal yourself with AWS HTTP API, you can get the presigned url using the ex_aws_s3 library, and use the url in your backend to get the stream of chunks to send to the client.
3
benonymus
and how can I return a link to the user where they can get the image like this, in the json response?
0
alvises
do you need to return the file or a link?
At some point the client will have to download the file, so the example I wrote above works to download the file in chunks while sending the chunks to the client.
0
benonymus
well now I am returning a json with fields, one of those fields have the link to aws, I need to replace that with a link, so I would need a link
this is what I return now:
%{
attachment_url:
XD.Utils.get_image_aws_url(
"post_attachment_url",
post_attachment.attachment,
post_attachment
),
type: post_attachment.type,
caption: post_attachment.caption,
user: render_one(post_attachment.user, UserView, "user_mini.json")
}
so i would create the aws url earlier do the magic and replace it with the new one pointing to the file
0