When I do large file uploading, which process function do you recommend?

Hello~
I am working on code that uploads video file(100MB ~ 300MB) to Amazon S3.
It takes quite some time. But I am not sure which elixir module or function I should use for concurrency.
I read some article about it but still don’t understand clearly.
And in the future, I need to code for uploading progress.
Which one do you recommend?

example

def upload_video_to_s3_async(path, username) do
    s3_filename = create_s3_filename(path, username)
    Task.async(fn ->
        path
        |> S3.Upload.stream_file()
        |> S3.upload(System.get_env("S3-Bucket-Name"), s3_filename)
        |> ExAws.request()
    end)
end

Task, Task.Supervisor, spawn?
I need your help!

Where does the video file come from? If someone has uploaded it to you, I would recommend using presigned S3 urls and having the user upload it directly to S3 instead of having your application as a middleman.

4 Likes

Hi! Thanks for your reply.
(I am actually fan of your book and building backend api using absinthe :slight_smile:)
anyway.

video comes from users. Users record video and upload it to amazon S3

What does this mean? User uploads video to amazon S3 through my backend api.

And that is the problem. Instead make user upload that video directly to the S3 instead, without even touching your backend API.

1 Like

use arc or the (afaik) more recently updated fork https://github.com/elixir-waffle/waffle - to upload to s3 through own server… uploading large files like video through/to own server is something I would generally avoid - especially when/if they are stored/persisted to s3… see this answer https://stackoverflow.com/questions/42211542/elixir-phoenix-client-side-browser-ajax-upload-to-s3 - for the “magic” url whereupon the user can upload directly to s3…

1 Like

Thanks for your link…
so in server side, only make presigned url then upload happens in client side using presigned url. Am I correct?