Here's how to upload to Cloudflare R2 - tweaks from original S3 implementation code

Phoenix Forum Chat / Discussions
, ,
1

After rummaging through ElixirForum, dissecting the HexDocs, I’ve finally been successful at uploading to Cloudflare R2 (which is cheaper than S3).

For those who don’t know - Cloudflare R2 actually has the same API as S3, so most of the library can be used that was developed for use with S3.

So, I decided to share snippets to give back to the community :slight_smile:

In CloudFlare R2 Settings

I’m going to skip this part but you gotta create the R2 instance first.
Then go to Settings → CORS Policy → Edit CORS Policy
Here’s what I added:

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:4000",
      "https://myapp.fly.dev"
    ],
    "AllowedMethods": [
      "GET",
      "PUT",
      "POST"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": []
  }
]

And then save. Obviously for AllowedOrigins, you can add more hosts to suit your need.

That’s it. Keep everything default.

Now that’s done, let’s go to the Phoenix Liveview App.

Liveview

As you all know, there are a couple parts to this. I mostly followed the backbone structure of the Live Upload tutorial on External Uploads.

You can set up the code following that tutorial to the letter, and then start the modifications below.

App.js

...
let Uploaders = {};
Uploaders.S3 = function (entries, onViewError) {
  entries.forEach((entry) => {
    let { url } = entry.meta;
    let xhr = new XMLHttpRequest();

    onViewError(() => xhr.abort());

    xhr.onload = () =>
      xhr.status >= 200 && xhr.status < 300
        ? entry.progress(100)
        : entry.error();

    xhr.onerror = () => entry.error();
    xhr.upload.addEventListener("progress", (event) => {
      debugger;
      if (event.lengthComputable) {
        let percent = Math.round((event.loaded / event.total) * 100);
        if (percent < 100) {
          entry.progress(percent);
        }
      }
    });

    xhr.open("PUT", url, true);
    xhr.setRequestHeader("credentials", "same-origin parameter");
    xhr.send(entry.file);
  });
};

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: { _csrf_token: csrfToken },
});
...

I did also copy some bits and pieces from the live_beats git repo, but they’re not necessary for the upload (more on the visual side).

Form_Component.ex :: render()

It’s literally the default form_component.ex when generated via mix task.
On top of that, there is this snippet to add the section for file upload. My example uploads just one file.

In this example, the variable name for the upload assign is member_photo.

def render(assigns) do
    ~H"""
    <div>
      <.header>
        <%= @title %>
        <:subtitle>Use this form to manage member records in your database.</:subtitle>
      </.header>

      <.simple_form
        for={@form}
        id="member-form"
        phx-target={@myself}
        phx-change="validate"
        phx-submit="save"
      >
        <!-- ....... -->
        <!-- upload -->
        <div class="sm:grid sm:border-t sm:border-gray-200 sm:pt-5">
          <label for="member_member_from" class="block text-sm font-semibold leading-6 text-zinc-800">
            Member Photo
          </label>
          <%!-- render each member_photo entry --%>
          <%= for entry <- @uploads.member_photo.entries do %>
            <article class="upload-entry">
              <figure>
                <.live_img_preview entry={entry} />
                <figcaption><%= entry.client_name %></figcaption>
              </figure>

              <%!-- entry.progress will update automatically for in-flight entries --%>
              <progress value={entry.progress} max="100"><%= entry.progress %>%</progress>

              <%!-- a regular click event whose handler will invoke Phoenix.LiveView.cancel_upload/3 --%>
              <button
                type="button"
                phx-click="cancel-upload"
                phx-value-ref={entry.ref}
                aria-label="cancel"
              >
                &times;
              </button>

              <%!-- Phoenix.Component.upload_errors/2 returns a list of error atoms --%>
              <%= for err <- upload_errors(@uploads.member_photo, entry) do %>
                <.error><%= error_to_string(err) %></.error>
              <% end %>
            </article>
          <% end %>
          <div class="mt-1 sm:mt-0" phx-drop-target={@uploads.member_photo.ref}>
            <%= if Enum.any?(@error_messages) do %>
              <div class="rounded-md bg-red-50 p-4 mb-2">
                <div class="flex">
                  <div class="flex-shrink-0">
                    <.icon name={:x_circle} class="h-5 w-5 text-red-400" />
                  </div>
                  <div class="ml-3">
                    <h3 class="text-sm font-medium text-red-800">
                      Oops!
                    </h3>
                    <div class="mt-2 text-sm text-red-700">
                      <ul role="list" class="list-disc pl-5 space-y-1">
                        <%= for {label, kind} <- @error_messages do %>
                          <li><.file_error label={label} kind={kind} /></li>
                        <% end %>
                      </ul>
                    </div>
                  </div>
                </div>
              </div>
            <% end %>

            <div class="max-w-lg flex justify-center px-6 pt-5 pb-6 border-2 border-gray-300 border-dashed rounded-md">
              <div class="space-y-1 text-center">
                <svg
                  class="mx-auto h-12 w-12 text-gray-400"
                  stroke="currentColor"
                  fill="none"
                  viewBox="0 0 48 48"
                  aria-hidden="true"
                >
                  <path
                    d="M28 8H12a4 4 0 00-4 4v20m32-12v8m0 0v8a4 4 0 01-4 4H12a4 4 0 01-4-4v-4m32-4l-3.172-3.172a4 4 0 00-5.656 0L28 28M8 32l9.172-9.172a4 4 0 015.656 0L28 28m0 0l4 4m4-24h8m-4-4v8m-12 4h.02"
                    stroke-width="2"
                    stroke-linecap="round"
                    stroke-linejoin="round"
                  >
                  </path>
                </svg>
                <div class="flex text-sm text-gray-600">
                  <label
                    for="file-upload"
                    class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500"
                  >
                    <span phx-click={js_exec("##{@uploads.member_photo.ref}", "click", [])}>
                      Upload file
                    </span>
                    <.live_file_input upload={@uploads.member_photo} class="sr-only" tabindex="0" />
                  </label>
                  <p class="pl-1">or drag and drop</p>
                </div>
                <p class="text-xs text-gray-500">
                  images up to 20MB
                </p>
              </div>
            </div>
          </div>
        </div>
        <!-- /upload -->
        <:actions>
          <.button phx-disable-with="Saving...">Save Member</.button>
        </:actions>
      </.simple_form>

      <%!-- <form id="upload-form" phx-submit="save" phx-change="validate"> --%>
      <%!-- <.live_file_input upload={@uploads.member_photo} /> --%>
      <%!-- <button type="submit">Upload</button> --%>
      <%!-- </form> --%>
    </div>
    """
  end

Note that there are some visual aspects of above code that isn’t really necessary. Please see the hex docs for the drop zone implementation / upload triggers.

Form_Component.ex :: update()

def update(%{member: member} = assigns, socket) do
    changeset = Members.change_member(member)

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> assign(changesets: %{}, error_messages: [])
     |> allow_upload(:member_photo,
       accept: ~w(.jpg .jpeg .png),
       max_entries: 1,
       external: &presign_upload/2
     )}
  end

Form_Component.ex :: presign_upload/2

defp presign_upload(entry, socket) do
    uploads = socket.assigns.uploads
    member = socket.assigns.member
    filename = "#{member.id}-#{member.name}#{Path.extname(entry.client_name)}"
    key = "public/#{filename}"

    config = %{
      region: region(),
      access_key_id: access_key_id(),
      secret_access_key: secret_access_key(),
      url: "https://#{bucket()}.#{account_id()}.r2.cloudflarestorage.com"
    }

    {:ok, presigned_url} =
      SimpleS3Upload.presigned_put(config, bucket(),
        key: key,
        content_type: entry.client_type,
        max_file_size: uploads[entry.upload_config].max_file_size
      )

    meta = %{
      uploader: "S3",
      key: key,
      url: presigned_url
    }

    {:ok, meta, socket}
  end

Form_Component.ex :: what about save_resource()?

Nothing upload-specific per se, you can tweak it once the upload is done - such as storing the filename.

Form_Component.ex :: extra helper functions

  defp file_error(%{kind: :dropped} = assigns),
    do: ~H|<%= @label %>: dropped (exceeds limit of 10 files)|

  defp file_error(%{kind: :too_large} = assigns),
    do: ~H|<%= @label %>: larger than 10MB|

  defp file_error(%{kind: :not_accepted} = assigns),
    do: ~H|<%= @label %>: not a valid member_photo file|

  defp file_error(%{kind: :too_many_files} = assigns),
    do: ~H|too many files|

  defp file_error(%{kind: :invalid} = assigns),
    do: ~H|Something went wrong|

  defp file_error(%{kind: %Ecto.Changeset{}} = assigns),
    do: ~H|<%= @label %>: <%= translate_changeset_errors(@kind) %>|

  defp file_error(%{kind: {msg, opts}} = assigns) when is_binary(msg) and is_list(opts),
    do: ~H|<%= @label %>: <%= translate_error(@kind) %>|

  defp error_to_string(:too_large), do: "Too large"
  defp error_to_string(:not_accepted), do: "You have selected an unacceptable file type"
  defp error_to_string(:too_many_files), do: "You have selected too many files"
  defp error_to_string(:external_client_failure), do: "External client has failed upload"

  defp access_key_id(), do: Application.get_env(:my_app, :s3_compatible_access_key_id)
  defp secret_access_key(), do: Application.get_env(:my_app, :s3_compatible_secret_access_key)
  defp account_id(), do: Application.get_env(:my_app, :s3_compatible_application_id)
  defp region(), do: "auto"
  defp bucket(), do: "my_bucket_name"

Obviously the config files should have the :my_app, :s3* values.

SimpleS3Upload

The way how Chris McCord originally wrote it is based on using form-based uploads, which isn’t supported in Cloudflare R2. We have to use the way of generating presigned URL.

defmodule MyApp.Uploading.SimpleS3Upload do
  @moduledoc """
  Below is code from Chris McCord, modified for Cloudflare R2

  https://gist.github.com/chrismccord/37862f1f8b1f5148644b75d20d1cb073

  """
  @one_hour_seconds 3600

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string

  """
  def presigned_put(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "PUT",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string
    ## NOTE - I haven't actually tested this but the gist of the idea is correct.
  """
  def presigned_get(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "GET",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end
end

mix.exs - adding :aws_signature to deps

defp deps do
[
  ...
   {:aws_signature, "~> 0.3.1"},
  ...
]
end

Finished!

Thanks to all who left hints in this forum to get it working.
I still haven’t quite figured out how to do the progress bar working, or how to store something so I can retrieve the stored data back (probably just using some filename generating function).

9 Likes
2

I bumped into this as I wanted to evaluate R2. I don’t think Cloudfare R2 serves the files just like S3 does, from the link. Still discovering but it seems you need to set up a Cloudfare worker. The idea is of course to serve the images from the Cloudfare CDN.

I wanted to pass multpart uploads with a formdata and did not succeed (yet?) to use ExAws.S3 to upload to R2 from the server. It must be a question of passing the correct config I imagine. The trip via the server is to transform the uploaded files.

You can still upload " by hand to R2" - from the server when you use a signed url, and use the “idempotent” version “PUT”, and put the MIME type as well. Some code in case of any interest.

# a Phoenix controller to handle a formdata.
def handle_r2(conn, params) do
%Plug.Upload{path: path, filename: filename} = Map.get(params, "file")

config = Application.get_env(:my_app, :r2)
# "https://<account_id>.r2.cloudflarestorage.com/<bucket_name>/<clean_filename>"
uploaded_file_name = Keyword.get(config, :r2_url) <> "/" <> URI.encoded(filename)

url_sign_v4 =
   :aws_signature.sign_v4_query_params(
       Keyword.get(config, :access_key_id),
       Keyword.get(config, :secret_access_key),
       "auto",
       "s3",
       :calendar.universal_time(),
       "PUT",
       uploaded_file_name,
       body_digest: "UNSIGNED-PAYLOAD",
       ttl: 3600
     )

  with {:ok, %{size: file_length} <- 
           File.stat!(path),
       {:ok, %{mime_type: mime}} <-
           GenMagic.Server.perform(:gen_magic, path) do
       headers = [{"Content-Type", mime}, {"Content-Length", file_length}] 
       f_stream = File.stream!(path, [], 16_384)
       Finch.build("PUT", url_sign_v4, headers, {:stream, f_stream})
       |> Finch.request!(MyFinch)
   else
      ...
end
3

For getting the file, you can just use “GET” as the method when generating the presigned url.
Not sure if multi-part form works with Worker, but as-is R2 doesn’t support multipart: Presigned URLs · Cloudflare R2 docs

Thanks for your tip!! I will definitely be using that too :slight_smile: the frontend based approach would have a nice progress bar showing up, so I guess that is why the implementation example used the way to do the upload at the client side.

4

Yes you can GET by I wanted R2 to serve the files from the bucket. Perhaps if you set up your domain with Cloudfare and cache the bucket on the CDN, it may work.

It seems that R2 allows multipart uploads.

Screenshot 2023-10-13 at 05.48.44

I wanted to use ExAws.S3.Upload since it handles mulitpart nicely, but the R2 endpoint is different: <account_id><dns_suffix>/<bucket>/<filename>, and no region reference.

5

Interesting. So you were trying to use ExAws.S3.Upload and it does the upload by the multipart form upload?

Is there any benefit of multipart versus just presigned URL?

6

AWS Multipart documentation

So you can use ExAws.S3 … once you have the config. Just use the key host with the R2 endpoint, and that’s it, all the heavy work is done for you :slight_smile:

config :ex_aws, :s3,
     access_key_id: System.get_env("..."),
     secret_key_id: System.get_env("..."),
     host: <ACCOUNT_ID>.r2.cloudflarestorage.com",
     bucket: System.get_env("R2_BUCKET")

When I upload a file from a form, I get (in a controller) a %Plug.Upload{path, filename, content_type} struct. Then I copy/paste “standard” code from the docs:

{:ok, %{body: %{location: location}, status_code: 200} =
  path
  |> ExAws.S3.Upload.stream_file()
  |> ExAws.S3.upload(System.get_env("R2_BUCKET"), URI.encode(filename),
    acl: :public_read,
    content_type: content_type,
    content_disposition: "inline"
  )
  |> ExAws.request()

json(conn, %{location: location})

But R2 doesn’t serve this file. You need a domain registered by Cloudfare.

NB. I tested BackBlaze B2, works exactly the same way with ExAws, just change the config host = s3.<REGION>.backblazeb2.com and that’s it. And BackBlaze serves the URL this time.

NB. This trip to the server makes sense if you make some actions on the file, otherwise its a client task. You can maybe push the response URL to the server to save it in a database if needed (eg Instagram feed).

7

Interesting, good to know - nice to have this in the tool chest.
For now in my problem space I’m able to do creation, reading, and deleting using the presigned urls,
so for now I’m probably going to not use the ExAws.S3 feature - but really appreciate your thorough explanation! Thank you!

It’s awesome how much knowledge this forum has :slight_smile: I felt alone trying this out but I’m glad I reached out and posted my experience :+1:

1 Like
8

Here is a sample app that does upload to S3 directly from the LiveView. It uses the latest Phoenix.LivieView.UploadWriter to do the multipart upload.

I have tested this on clouldflare R2 and it works flawlessly.

:slight_smile:

2 Likes