Liveview Uploads - Uploading large amounts of large files

I’m a bit in a fuzz about this topic. I’ve worked out some sort of Dropbox feature for a specific client. This feature allows him to upload files (zips, pdf’s, ..) up to a limit of 50. As a test, I’ve uploaded 25 pdf’s with a file size of 50mb.

During the tests, I got it working with 10 as it’s temporary limit and it needs some time, as it would just stay still on 25 files and die with a silent error in my terminal. I would like to discuss some ideas how we can make sure we can pick up all 50 files and what the most correct way would be to send them over as secure as possible with a reasonable tempo.

Any ideas?

Some details:

• We are uploading to an Amazon S3 Bucket
• The file uploader has a custom Write script

The uploader assign for our uploader

     |> allow_upload(:dropbox_files,
       accept: @extensions,
       validator: &validate_extension/1,
       max_entries: @max_files,
       auto_upload: true,
       chunk_size: 5 * 1024 * 1024,
       progress: &handle_progress/3,
       max_file_size: 500_000_000,
       writer: fn _name, entry, socket ->
         key =
           ClientDropbox.tmp_key(socket.assigns.client, socket.assigns.user, entry.client_name)

         {DropboxWriter, key: key, content_type: MIME.from_path(key)}
       end
     )}

Our Write file:

defmodule MyCompanyWeb.ClientDashboard.DropboxWriter do
  @behaviour Phoenix.LiveView.UploadWriter

  alias MyCompany.Helpers.FileHelpers

  @impl true
  def init(opts) do
    opts = Keyword.validate!(opts, [:content_type, :key])

    with {:ok, upload} <- FileHelpers.initiate_multipart_upload(opts[:key], opts[:content_type]) do
      {:ok, %{key: opts[:key], chunk: 1, parts: [], upload: upload}}
    end
  end

  @impl true
  def meta(state) do
    %{key: state.key, filename: Path.basename(state.key)}
  end

  @impl true
  def write_chunk(data, state) do
    case FileHelpers.upload_part(state.upload, state.chunk, data) do
      {:ok, part} ->
        {:ok, %{state | chunk: state.chunk + 1, parts: [part | state.parts]}}

      {:error, reason} ->
        {:error, reason, state}
    end
  end

  @impl true
  def close(state, :done) do
    case FileHelpers.complete_multipart_upload(state.upload, state.parts) do
      :ok -> {:ok, state}
      {:error, reason} -> {:error, reason}
    end
  end

  def close(state, _reason) do
    FileHelpers.abort_multipart_upload(state.upload)

    {:ok, state}
  end
end