mekusigjinn

mekusigjinn

Here's how to upload to Cloudflare R2 - tweaks from original S3 implementation code

After rummaging through ElixirForum, dissecting the HexDocs, I’ve finally been successful at uploading to Cloudflare R2 (which is cheaper than S3).

For those who don’t know - Cloudflare R2 actually has the same API as S3, so most of the library can be used that was developed for use with S3.

So, I decided to share snippets to give back to the community :slight_smile:

In CloudFlare R2 Settings

I’m going to skip this part but you gotta create the R2 instance first.
Then go to Settings → CORS Policy → Edit CORS Policy
Here’s what I added:

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:4000",
      "https://myapp.fly.dev"
    ],
    "AllowedMethods": [
      "GET",
      "PUT",
      "POST"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": []
  }
]

And then save. Obviously for AllowedOrigins, you can add more hosts to suit your need.

That’s it. Keep everything default.

Now that’s done, let’s go to the Phoenix Liveview App.

Liveview

As you all know, there are a couple parts to this. I mostly followed the backbone structure of the Live Upload tutorial on External Uploads.

You can set up the code following that tutorial to the letter, and then start the modifications below.

App.js

...
let Uploaders = {};
Uploaders.S3 = function (entries, onViewError) {
  entries.forEach((entry) => {
    let { url } = entry.meta;
    let xhr = new XMLHttpRequest();

    onViewError(() => xhr.abort());

    xhr.onload = () =>
      xhr.status >= 200 && xhr.status < 300
        ? entry.progress(100)
        : entry.error();

    xhr.onerror = () => entry.error();
    xhr.upload.addEventListener("progress", (event) => {
      debugger;
      if (event.lengthComputable) {
        let percent = Math.round((event.loaded / event.total) * 100);
        if (percent < 100) {
          entry.progress(percent);
        }
      }
    });

    xhr.open("PUT", url, true);
    xhr.setRequestHeader("credentials", "same-origin parameter");
    xhr.send(entry.file);
  });
};

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: { _csrf_token: csrfToken },
});
...

I did also copy some bits and pieces from the live_beats git repo, but they’re not necessary for the upload (more on the visual side).

Form_Component.ex :: render()

It’s literally the default form_component.ex when generated via mix task.
On top of that, there is this snippet to add the section for file upload. My example uploads just one file.

In this example, the variable name for the upload assign is member_photo.

def render(assigns) do
    ~H"""
    <div>
      <.header>
        <%= @title %>
        <:subtitle>Use this form to manage member records in your database.</:subtitle>
      </.header>

      <.simple_form
        for={@form}
        id="member-form"
        phx-target={@myself}
        phx-change="validate"
        phx-submit="save"
      >
        <!-- ....... -->
        <!-- upload -->
        <div class="sm:grid sm:border-t sm:border-gray-200 sm:pt-5">
          <label for="member_member_from" class="block text-sm font-semibold leading-6 text-zinc-800">
            Member Photo
          </label>
          <%!-- render each member_photo entry --%>
          <%= for entry <- @uploads.member_photo.entries do %>
            <article class="upload-entry">
              <figure>
                <.live_img_preview entry={entry} />
                <figcaption><%= entry.client_name %></figcaption>
              </figure>

              <%!-- entry.progress will update automatically for in-flight entries --%>
              <progress value={entry.progress} max="100"><%= entry.progress %>%</progress>

              <%!-- a regular click event whose handler will invoke Phoenix.LiveView.cancel_upload/3 --%>
              <button
                type="button"
                phx-click="cancel-upload"
                phx-value-ref={entry.ref}
                aria-label="cancel"
              >
                &times;
              </button>

              <%!-- Phoenix.Component.upload_errors/2 returns a list of error atoms --%>
              <%= for err <- upload_errors(@uploads.member_photo, entry) do %>
                <.error><%= error_to_string(err) %></.error>
              <% end %>
            </article>
          <% end %>
          <div class="mt-1 sm:mt-0" phx-drop-target={@uploads.member_photo.ref}>
            <%= if Enum.any?(@error_messages) do %>
              <div class="rounded-md bg-red-50 p-4 mb-2">
                <div class="flex">
                  <div class="flex-shrink-0">
                    <.icon name={:x_circle} class="h-5 w-5 text-red-400" />
                  </div>
                  <div class="ml-3">
                    <h3 class="text-sm font-medium text-red-800">
                      Oops!
                    </h3>
                    <div class="mt-2 text-sm text-red-700">
                      <ul role="list" class="list-disc pl-5 space-y-1">
                        <%= for {label, kind} <- @error_messages do %>
                          <li><.file_error label={label} kind={kind} /></li>
                        <% end %>
                      </ul>
                    </div>
                  </div>
                </div>
              </div>
            <% end %>

            <div class="max-w-lg flex justify-center px-6 pt-5 pb-6 border-2 border-gray-300 border-dashed rounded-md">
              <div class="space-y-1 text-center">
                <svg
                  class="mx-auto h-12 w-12 text-gray-400"
                  stroke="currentColor"
                  fill="none"
                  viewBox="0 0 48 48"
                  aria-hidden="true"
                >
                  <path
                    d="M28 8H12a4 4 0 00-4 4v20m32-12v8m0 0v8a4 4 0 01-4 4H12a4 4 0 01-4-4v-4m32-4l-3.172-3.172a4 4 0 00-5.656 0L28 28M8 32l9.172-9.172a4 4 0 015.656 0L28 28m0 0l4 4m4-24h8m-4-4v8m-12 4h.02"
                    stroke-width="2"
                    stroke-linecap="round"
                    stroke-linejoin="round"
                  >
                  </path>
                </svg>
                <div class="flex text-sm text-gray-600">
                  <label
                    for="file-upload"
                    class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500"
                  >
                    <span phx-click={js_exec("##{@uploads.member_photo.ref}", "click", [])}>
                      Upload file
                    </span>
                    <.live_file_input upload={@uploads.member_photo} class="sr-only" tabindex="0" />
                  </label>
                  <p class="pl-1">or drag and drop</p>
                </div>
                <p class="text-xs text-gray-500">
                  images up to 20MB
                </p>
              </div>
            </div>
          </div>
        </div>
        <!-- /upload -->
        <:actions>
          <.button phx-disable-with="Saving...">Save Member</.button>
        </:actions>
      </.simple_form>

      <%!-- <form id="upload-form" phx-submit="save" phx-change="validate"> --%>
      <%!-- <.live_file_input upload={@uploads.member_photo} /> --%>
      <%!-- <button type="submit">Upload</button> --%>
      <%!-- </form> --%>
    </div>
    """
  end

Note that there are some visual aspects of above code that isn’t really necessary. Please see the hex docs for the drop zone implementation / upload triggers.

Form_Component.ex :: update()

def update(%{member: member} = assigns, socket) do
    changeset = Members.change_member(member)

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> assign(changesets: %{}, error_messages: [])
     |> allow_upload(:member_photo,
       accept: ~w(.jpg .jpeg .png),
       max_entries: 1,
       external: &presign_upload/2
     )}
  end

Form_Component.ex :: presign_upload/2

defp presign_upload(entry, socket) do
    uploads = socket.assigns.uploads
    member = socket.assigns.member
    filename = "#{member.id}-#{member.name}#{Path.extname(entry.client_name)}"
    key = "public/#{filename}"

    config = %{
      region: region(),
      access_key_id: access_key_id(),
      secret_access_key: secret_access_key(),
      url: "https://#{bucket()}.#{account_id()}.r2.cloudflarestorage.com"
    }

    {:ok, presigned_url} =
      SimpleS3Upload.presigned_put(config, bucket(),
        key: key,
        content_type: entry.client_type,
        max_file_size: uploads[entry.upload_config].max_file_size
      )

    meta = %{
      uploader: "S3",
      key: key,
      url: presigned_url
    }

    {:ok, meta, socket}
  end

Form_Component.ex :: what about save_resource()?

Nothing upload-specific per se, you can tweak it once the upload is done - such as storing the filename.

Form_Component.ex :: extra helper functions

  defp file_error(%{kind: :dropped} = assigns),
    do: ~H|<%= @label %>: dropped (exceeds limit of 10 files)|

  defp file_error(%{kind: :too_large} = assigns),
    do: ~H|<%= @label %>: larger than 10MB|

  defp file_error(%{kind: :not_accepted} = assigns),
    do: ~H|<%= @label %>: not a valid member_photo file|

  defp file_error(%{kind: :too_many_files} = assigns),
    do: ~H|too many files|

  defp file_error(%{kind: :invalid} = assigns),
    do: ~H|Something went wrong|

  defp file_error(%{kind: %Ecto.Changeset{}} = assigns),
    do: ~H|<%= @label %>: <%= translate_changeset_errors(@kind) %>|

  defp file_error(%{kind: {msg, opts}} = assigns) when is_binary(msg) and is_list(opts),
    do: ~H|<%= @label %>: <%= translate_error(@kind) %>|

  defp error_to_string(:too_large), do: "Too large"
  defp error_to_string(:not_accepted), do: "You have selected an unacceptable file type"
  defp error_to_string(:too_many_files), do: "You have selected too many files"
  defp error_to_string(:external_client_failure), do: "External client has failed upload"

  defp access_key_id(), do: Application.get_env(:my_app, :s3_compatible_access_key_id)
  defp secret_access_key(), do: Application.get_env(:my_app, :s3_compatible_secret_access_key)
  defp account_id(), do: Application.get_env(:my_app, :s3_compatible_application_id)
  defp region(), do: "auto"
  defp bucket(), do: "my_bucket_name"

Obviously the config files should have the :my_app, :s3* values.

SimpleS3Upload

The way how Chris McCord originally wrote it is based on using form-based uploads, which isn’t supported in Cloudflare R2. We have to use the way of generating presigned URL.

defmodule MyApp.Uploading.SimpleS3Upload do
  @moduledoc """
  Below is code from Chris McCord, modified for Cloudflare R2

  https://gist.github.com/chrismccord/37862f1f8b1f5148644b75d20d1cb073

  """
  @one_hour_seconds 3600

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string

  """
  def presigned_put(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "PUT",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string
    ## NOTE - I haven't actually tested this but the gist of the idea is correct.
  """
  def presigned_get(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "GET",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end
end

mix.exs - adding :aws_signature to deps

defp deps do
[
  ...
   {:aws_signature, "~> 0.3.1"},
  ...
]
end

Finished!

Thanks to all who left hints in this forum to get it working.
I still haven’t quite figured out how to do the progress bar working, or how to store something so I can retrieve the stored data back (probably just using some filename generating function).

Most Liked

TwistingTwists

TwistingTwists

https://github.com/TwistingTwists/learning-concurrent-elixir/blob/master/sample.exs

Here is a sample app that does upload to S3 directly from the LiveView. It uses the latest Phoenix.LivieView.UploadWriter to do the multipart upload.

I have tested this on clouldflare R2 and it works flawlessly.

:slight_smile:

mekusigjinn

mekusigjinn

Interesting, good to know - nice to have this in the tool chest.
For now in my problem space I’m able to do creation, reading, and deleting using the presigned urls,
so for now I’m probably going to not use the ExAws.S3 feature - but really appreciate your thorough explanation! Thank you!

It’s awesome how much knowledge this forum has :slight_smile: I felt alone trying this out but I’m glad I reached out and posted my experience :+1:

Where Next?

Popular in Discussions Top

matthias_toepp
I’d love to hear what people think about Wisp, the new Gleam web framework started by Gleam’s primary creator Louis Pilfold. Gleam, alon...
New
sashaafm
Piggy backing a bit on @dvcrn topic BEAM optimization for functions with static return type?, I’ve been trying to understand in a deeper ...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
pillaiindu
In django there is a cache framework backed by memcached. Rails also puts a lot of emphasis on caching, and even the idea of russian-doll...
New
arpan
Hello everyone :wave: Today I am very excited to announce a project that I have been working on for almost 3 months now. The project is...
New
pillaiindu
I want to convert a Phoenix LiveView CRUD website to a CRUD mobile app. What do you think is the easiest way to do so?
New
nburkley
AWS re:Invent is on at the moment with some interesting announcements. One new feature in particular is the Lambda Runtime API for AWS La...
New
shishini
I think this twitter post and youtube video didn’t get as much attention as I hoped I am still new to Elixir, so can’t really judge ...
New
PragTob
Hey everyone, this has been on my mind for some time and I’d love your input on it! TLDR: I feel like maps are superioer for storing and...
New
chulkilee
Here are the list of HTTP client libraries/wrappers, and some thoughts on HTTP client in general. I’d like to hear from others how they w...
New

Other popular topics Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement