Uploading multiple files with Task.async_stream ex_aws

I was uploading multiple files to AWS using ex_aws, as

S3.do_save("#{upload_path}#{compare_id}.gif", gif_content, [content_type: "image/gif", acl: :public_read])
S3.do_save("#{upload_path}#{compare_id}.mp4", mp4_content, [acl: :public_read])
S3.do_save("#{camera_exid}/compares/#{compare_id}/thumb-#{compare_id}.jpg", File.read!("#{root}thumb-#{compare_id}.jpg"), [content_type: "image/jpg", acl: :public_read])

and In S3.do_save I had a simple method as

  def do_save(path, content, opts) do
    ExAws.S3.put_object("evercam-camera-assets", path, content, opts)
    |> ExAws.request!
  end

I am trying to upload multiple files now using Task.async_stream as described in ExAws docs

upload_file = fn {src_path, dest_path} ->
  ExAws.S3.put_object("my-assets", dest_path, File.read!(src_path), [acl: :public_read])
  |> ExAws.request!
end
paths
|> Task.async_stream(upload_file, max_concurrency: 10)
|> Stream.run

and paths supposed to be like that

%{      
  "storage/compa-ycqozxb/compa-ycqozxb.gif" => "crome-tidcz/compares/compa-ycqozxb/compa-ycqozxb.gif",
  "storage/compa-ycqozxb/compa-ycqozxb.mp4" => "crome-tidcz/compares/compa-ycqozxb/compa-ycqozxb.mp4",
  "storage/compa-ycqozxb/thumb-compa-ycqozxb.jpg" => "crome-tidcz/compares/compa-ycqozxb/thumb-compa-ycqozxb.jpg"
}

But each time I am getting this error

** (exit) exited in: Task.Supervised.stream(5000)
    ** (EXIT) time out
    (elixir) lib/task/supervised.ex:280: Task.Supervised.stream_reduce/7
    (elixir) lib/stream.ex:637: Stream.run/1

Filesizes are:
GIF: 3.1 MB
MP4: 1.0 MB
JPG: 97.4 KB

Any help would be thankful,

Are you sure those files can be uploaded in 5 seconds? Otherwise you need to increase the timeout.

How I can increase those 5 seconds? Even if I want to increase it?

|> Task.async_stream(upload_file, max_concurrency: 10, timeout: timeout_in_ms)

1 Like

Okay I understand, Can we speed this up ? for example then what is the difference in

Simple request and this Task.async_stream? Is there any possibility to upload the files faster than normal?

async_stream is about uploading files in parallel instead of one after the other. In case a single file upload does not saturate your connection this will speed up the total time needed to upload all files. What won’t change is how long a single file needs to be uploaded.

1 Like