mekusigjinn

mekusigjinn

Here's how to upload to Cloudflare R2 - tweaks from original S3 implementation code

After rummaging through ElixirForum, dissecting the HexDocs, I’ve finally been successful at uploading to Cloudflare R2 (which is cheaper than S3).

For those who don’t know - Cloudflare R2 actually has the same API as S3, so most of the library can be used that was developed for use with S3.

So, I decided to share snippets to give back to the community :slight_smile:

In CloudFlare R2 Settings

I’m going to skip this part but you gotta create the R2 instance first.
Then go to Settings → CORS Policy → Edit CORS Policy
Here’s what I added:

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:4000",
      "https://myapp.fly.dev"
    ],
    "AllowedMethods": [
      "GET",
      "PUT",
      "POST"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": []
  }
]

And then save. Obviously for AllowedOrigins, you can add more hosts to suit your need.

That’s it. Keep everything default.

Now that’s done, let’s go to the Phoenix Liveview App.

Liveview

As you all know, there are a couple parts to this. I mostly followed the backbone structure of the Live Upload tutorial on External Uploads.

You can set up the code following that tutorial to the letter, and then start the modifications below.

App.js

...
let Uploaders = {};
Uploaders.S3 = function (entries, onViewError) {
  entries.forEach((entry) => {
    let { url } = entry.meta;
    let xhr = new XMLHttpRequest();

    onViewError(() => xhr.abort());

    xhr.onload = () =>
      xhr.status >= 200 && xhr.status < 300
        ? entry.progress(100)
        : entry.error();

    xhr.onerror = () => entry.error();
    xhr.upload.addEventListener("progress", (event) => {
      debugger;
      if (event.lengthComputable) {
        let percent = Math.round((event.loaded / event.total) * 100);
        if (percent < 100) {
          entry.progress(percent);
        }
      }
    });

    xhr.open("PUT", url, true);
    xhr.setRequestHeader("credentials", "same-origin parameter");
    xhr.send(entry.file);
  });
};

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: { _csrf_token: csrfToken },
});
...

I did also copy some bits and pieces from the live_beats git repo, but they’re not necessary for the upload (more on the visual side).

Form_Component.ex :: render()

It’s literally the default form_component.ex when generated via mix task.
On top of that, there is this snippet to add the section for file upload. My example uploads just one file.

In this example, the variable name for the upload assign is member_photo.

def render(assigns) do
    ~H"""
    <div>
      <.header>
        <%= @title %>
        <:subtitle>Use this form to manage member records in your database.</:subtitle>
      </.header>

      <.simple_form
        for={@form}
        id="member-form"
        phx-target={@myself}
        phx-change="validate"
        phx-submit="save"
      >
        <!-- ....... -->
        <!-- upload -->
        <div class="sm:grid sm:border-t sm:border-gray-200 sm:pt-5">
          <label for="member_member_from" class="block text-sm font-semibold leading-6 text-zinc-800">
            Member Photo
          </label>
          <%!-- render each member_photo entry --%>
          <%= for entry <- @uploads.member_photo.entries do %>
            <article class="upload-entry">
              <figure>
                <.live_img_preview entry={entry} />
                <figcaption><%= entry.client_name %></figcaption>
              </figure>

              <%!-- entry.progress will update automatically for in-flight entries --%>
              <progress value={entry.progress} max="100"><%= entry.progress %>%</progress>

              <%!-- a regular click event whose handler will invoke Phoenix.LiveView.cancel_upload/3 --%>
              <button
                type="button"
                phx-click="cancel-upload"
                phx-value-ref={entry.ref}
                aria-label="cancel"
              >
                &times;
              </button>

              <%!-- Phoenix.Component.upload_errors/2 returns a list of error atoms --%>
              <%= for err <- upload_errors(@uploads.member_photo, entry) do %>
                <.error><%= error_to_string(err) %></.error>
              <% end %>
            </article>
          <% end %>
          <div class="mt-1 sm:mt-0" phx-drop-target={@uploads.member_photo.ref}>
            <%= if Enum.any?(@error_messages) do %>
              <div class="rounded-md bg-red-50 p-4 mb-2">
                <div class="flex">
                  <div class="flex-shrink-0">
                    <.icon name={:x_circle} class="h-5 w-5 text-red-400" />
                  </div>
                  <div class="ml-3">
                    <h3 class="text-sm font-medium text-red-800">
                      Oops!
                    </h3>
                    <div class="mt-2 text-sm text-red-700">
                      <ul role="list" class="list-disc pl-5 space-y-1">
                        <%= for {label, kind} <- @error_messages do %>
                          <li><.file_error label={label} kind={kind} /></li>
                        <% end %>
                      </ul>
                    </div>
                  </div>
                </div>
              </div>
            <% end %>

            <div class="max-w-lg flex justify-center px-6 pt-5 pb-6 border-2 border-gray-300 border-dashed rounded-md">
              <div class="space-y-1 text-center">
                <svg
                  class="mx-auto h-12 w-12 text-gray-400"
                  stroke="currentColor"
                  fill="none"
                  viewBox="0 0 48 48"
                  aria-hidden="true"
                >
                  <path
                    d="M28 8H12a4 4 0 00-4 4v20m32-12v8m0 0v8a4 4 0 01-4 4H12a4 4 0 01-4-4v-4m32-4l-3.172-3.172a4 4 0 00-5.656 0L28 28M8 32l9.172-9.172a4 4 0 015.656 0L28 28m0 0l4 4m4-24h8m-4-4v8m-12 4h.02"
                    stroke-width="2"
                    stroke-linecap="round"
                    stroke-linejoin="round"
                  >
                  </path>
                </svg>
                <div class="flex text-sm text-gray-600">
                  <label
                    for="file-upload"
                    class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500"
                  >
                    <span phx-click={js_exec("##{@uploads.member_photo.ref}", "click", [])}>
                      Upload file
                    </span>
                    <.live_file_input upload={@uploads.member_photo} class="sr-only" tabindex="0" />
                  </label>
                  <p class="pl-1">or drag and drop</p>
                </div>
                <p class="text-xs text-gray-500">
                  images up to 20MB
                </p>
              </div>
            </div>
          </div>
        </div>
        <!-- /upload -->
        <:actions>
          <.button phx-disable-with="Saving...">Save Member</.button>
        </:actions>
      </.simple_form>

      <%!-- <form id="upload-form" phx-submit="save" phx-change="validate"> --%>
      <%!-- <.live_file_input upload={@uploads.member_photo} /> --%>
      <%!-- <button type="submit">Upload</button> --%>
      <%!-- </form> --%>
    </div>
    """
  end

Note that there are some visual aspects of above code that isn’t really necessary. Please see the hex docs for the drop zone implementation / upload triggers.

Form_Component.ex :: update()

def update(%{member: member} = assigns, socket) do
    changeset = Members.change_member(member)

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> assign(changesets: %{}, error_messages: [])
     |> allow_upload(:member_photo,
       accept: ~w(.jpg .jpeg .png),
       max_entries: 1,
       external: &presign_upload/2
     )}
  end

Form_Component.ex :: presign_upload/2

defp presign_upload(entry, socket) do
    uploads = socket.assigns.uploads
    member = socket.assigns.member
    filename = "#{member.id}-#{member.name}#{Path.extname(entry.client_name)}"
    key = "public/#{filename}"

    config = %{
      region: region(),
      access_key_id: access_key_id(),
      secret_access_key: secret_access_key(),
      url: "https://#{bucket()}.#{account_id()}.r2.cloudflarestorage.com"
    }

    {:ok, presigned_url} =
      SimpleS3Upload.presigned_put(config, bucket(),
        key: key,
        content_type: entry.client_type,
        max_file_size: uploads[entry.upload_config].max_file_size
      )

    meta = %{
      uploader: "S3",
      key: key,
      url: presigned_url
    }

    {:ok, meta, socket}
  end

Form_Component.ex :: what about save_resource()?

Nothing upload-specific per se, you can tweak it once the upload is done - such as storing the filename.

Form_Component.ex :: extra helper functions

  defp file_error(%{kind: :dropped} = assigns),
    do: ~H|<%= @label %>: dropped (exceeds limit of 10 files)|

  defp file_error(%{kind: :too_large} = assigns),
    do: ~H|<%= @label %>: larger than 10MB|

  defp file_error(%{kind: :not_accepted} = assigns),
    do: ~H|<%= @label %>: not a valid member_photo file|

  defp file_error(%{kind: :too_many_files} = assigns),
    do: ~H|too many files|

  defp file_error(%{kind: :invalid} = assigns),
    do: ~H|Something went wrong|

  defp file_error(%{kind: %Ecto.Changeset{}} = assigns),
    do: ~H|<%= @label %>: <%= translate_changeset_errors(@kind) %>|

  defp file_error(%{kind: {msg, opts}} = assigns) when is_binary(msg) and is_list(opts),
    do: ~H|<%= @label %>: <%= translate_error(@kind) %>|

  defp error_to_string(:too_large), do: "Too large"
  defp error_to_string(:not_accepted), do: "You have selected an unacceptable file type"
  defp error_to_string(:too_many_files), do: "You have selected too many files"
  defp error_to_string(:external_client_failure), do: "External client has failed upload"

  defp access_key_id(), do: Application.get_env(:my_app, :s3_compatible_access_key_id)
  defp secret_access_key(), do: Application.get_env(:my_app, :s3_compatible_secret_access_key)
  defp account_id(), do: Application.get_env(:my_app, :s3_compatible_application_id)
  defp region(), do: "auto"
  defp bucket(), do: "my_bucket_name"

Obviously the config files should have the :my_app, :s3* values.

SimpleS3Upload

The way how Chris McCord originally wrote it is based on using form-based uploads, which isn’t supported in Cloudflare R2. We have to use the way of generating presigned URL.

defmodule MyApp.Uploading.SimpleS3Upload do
  @moduledoc """
  Below is code from Chris McCord, modified for Cloudflare R2

  https://gist.github.com/chrismccord/37862f1f8b1f5148644b75d20d1cb073

  """
  @one_hour_seconds 3600

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string

  """
  def presigned_put(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "PUT",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string
    ## NOTE - I haven't actually tested this but the gist of the idea is correct.
  """
  def presigned_get(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "GET",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end
end

mix.exs - adding :aws_signature to deps

defp deps do
[
  ...
   {:aws_signature, "~> 0.3.1"},
  ...
]
end

Finished!

Thanks to all who left hints in this forum to get it working.
I still haven’t quite figured out how to do the progress bar working, or how to store something so I can retrieve the stored data back (probably just using some filename generating function).

Most Liked

TwistingTwists

TwistingTwists

https://github.com/TwistingTwists/learning-concurrent-elixir/blob/master/sample.exs

Here is a sample app that does upload to S3 directly from the LiveView. It uses the latest Phoenix.LivieView.UploadWriter to do the multipart upload.

I have tested this on clouldflare R2 and it works flawlessly.

:slight_smile:

mekusigjinn

mekusigjinn

Interesting, good to know - nice to have this in the tool chest.
For now in my problem space I’m able to do creation, reading, and deleting using the presigned urls,
so for now I’m probably going to not use the ExAws.S3 feature - but really appreciate your thorough explanation! Thank you!

It’s awesome how much knowledge this forum has :slight_smile: I felt alone trying this out but I’m glad I reached out and posted my experience :+1:

Where Next?

Popular in Discussions Top

andre1sk
A big advantage to Elixir is all the distributed goodness but for many applications running on multiple nodes having integrated Etcd, Zoo...
New
WolfDan
After doing a port from a c++ library to my project in phoenix I’ve seen that I need a faster way to run this algorithm and I found this ...
New
jeramyRR
This is an interesting article to read. Elixir’s performance, like usual, is excellent. However, it seems like the high CPU usage is co...
New
laiboonh
Hi all, I am trying to convince my team to use liveview over the current react. What are some of the points where one should consider us...
New
thojanssens1
It would be nice to be able to define a redirect from one route to another from the router.ex file. E.g.: redirect "/", UserController, ...
New
pillaiindu
I want to convert a Phoenix LiveView CRUD website to a CRUD mobile app. What do you think is the easiest way to do so?
New
New
Qqwy
I would like to spark a discussion about the static access operator: .. For whom does not know: it is used in Elixir to access fields of...
New
pdgonzalez872
If this has been asked here before, please point me to where it was asked as I didn’t find it when I searched the forum. Maybe a mailing ...
New
slashdotdash
Phoenix Live View is now publicly available on GitHub. Here’s Chris McCord’s tweet announcing making it public.
New

Other popular topics Top

New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43806 214
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement