mekusigjinn

mekusigjinn

Here's how to upload to Cloudflare R2 - tweaks from original S3 implementation code

After rummaging through ElixirForum, dissecting the HexDocs, I’ve finally been successful at uploading to Cloudflare R2 (which is cheaper than S3).

For those who don’t know - Cloudflare R2 actually has the same API as S3, so most of the library can be used that was developed for use with S3.

So, I decided to share snippets to give back to the community :slight_smile:

In CloudFlare R2 Settings

I’m going to skip this part but you gotta create the R2 instance first.
Then go to Settings → CORS Policy → Edit CORS Policy
Here’s what I added:

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:4000",
      "https://myapp.fly.dev"
    ],
    "AllowedMethods": [
      "GET",
      "PUT",
      "POST"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": []
  }
]

And then save. Obviously for AllowedOrigins, you can add more hosts to suit your need.

That’s it. Keep everything default.

Now that’s done, let’s go to the Phoenix Liveview App.

Liveview

As you all know, there are a couple parts to this. I mostly followed the backbone structure of the Live Upload tutorial on External Uploads.

You can set up the code following that tutorial to the letter, and then start the modifications below.

App.js

...
let Uploaders = {};
Uploaders.S3 = function (entries, onViewError) {
  entries.forEach((entry) => {
    let { url } = entry.meta;
    let xhr = new XMLHttpRequest();

    onViewError(() => xhr.abort());

    xhr.onload = () =>
      xhr.status >= 200 && xhr.status < 300
        ? entry.progress(100)
        : entry.error();

    xhr.onerror = () => entry.error();
    xhr.upload.addEventListener("progress", (event) => {
      debugger;
      if (event.lengthComputable) {
        let percent = Math.round((event.loaded / event.total) * 100);
        if (percent < 100) {
          entry.progress(percent);
        }
      }
    });

    xhr.open("PUT", url, true);
    xhr.setRequestHeader("credentials", "same-origin parameter");
    xhr.send(entry.file);
  });
};

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: { _csrf_token: csrfToken },
});
...

I did also copy some bits and pieces from the live_beats git repo, but they’re not necessary for the upload (more on the visual side).

Form_Component.ex :: render()

It’s literally the default form_component.ex when generated via mix task.
On top of that, there is this snippet to add the section for file upload. My example uploads just one file.

In this example, the variable name for the upload assign is member_photo.

def render(assigns) do
    ~H"""
    <div>
      <.header>
        <%= @title %>
        <:subtitle>Use this form to manage member records in your database.</:subtitle>
      </.header>

      <.simple_form
        for={@form}
        id="member-form"
        phx-target={@myself}
        phx-change="validate"
        phx-submit="save"
      >
        <!-- ....... -->
        <!-- upload -->
        <div class="sm:grid sm:border-t sm:border-gray-200 sm:pt-5">
          <label for="member_member_from" class="block text-sm font-semibold leading-6 text-zinc-800">
            Member Photo
          </label>
          <%!-- render each member_photo entry --%>
          <%= for entry <- @uploads.member_photo.entries do %>
            <article class="upload-entry">
              <figure>
                <.live_img_preview entry={entry} />
                <figcaption><%= entry.client_name %></figcaption>
              </figure>

              <%!-- entry.progress will update automatically for in-flight entries --%>
              <progress value={entry.progress} max="100"><%= entry.progress %>%</progress>

              <%!-- a regular click event whose handler will invoke Phoenix.LiveView.cancel_upload/3 --%>
              <button
                type="button"
                phx-click="cancel-upload"
                phx-value-ref={entry.ref}
                aria-label="cancel"
              >
                &times;
              </button>

              <%!-- Phoenix.Component.upload_errors/2 returns a list of error atoms --%>
              <%= for err <- upload_errors(@uploads.member_photo, entry) do %>
                <.error><%= error_to_string(err) %></.error>
              <% end %>
            </article>
          <% end %>
          <div class="mt-1 sm:mt-0" phx-drop-target={@uploads.member_photo.ref}>
            <%= if Enum.any?(@error_messages) do %>
              <div class="rounded-md bg-red-50 p-4 mb-2">
                <div class="flex">
                  <div class="flex-shrink-0">
                    <.icon name={:x_circle} class="h-5 w-5 text-red-400" />
                  </div>
                  <div class="ml-3">
                    <h3 class="text-sm font-medium text-red-800">
                      Oops!
                    </h3>
                    <div class="mt-2 text-sm text-red-700">
                      <ul role="list" class="list-disc pl-5 space-y-1">
                        <%= for {label, kind} <- @error_messages do %>
                          <li><.file_error label={label} kind={kind} /></li>
                        <% end %>
                      </ul>
                    </div>
                  </div>
                </div>
              </div>
            <% end %>

            <div class="max-w-lg flex justify-center px-6 pt-5 pb-6 border-2 border-gray-300 border-dashed rounded-md">
              <div class="space-y-1 text-center">
                <svg
                  class="mx-auto h-12 w-12 text-gray-400"
                  stroke="currentColor"
                  fill="none"
                  viewBox="0 0 48 48"
                  aria-hidden="true"
                >
                  <path
                    d="M28 8H12a4 4 0 00-4 4v20m32-12v8m0 0v8a4 4 0 01-4 4H12a4 4 0 01-4-4v-4m32-4l-3.172-3.172a4 4 0 00-5.656 0L28 28M8 32l9.172-9.172a4 4 0 015.656 0L28 28m0 0l4 4m4-24h8m-4-4v8m-12 4h.02"
                    stroke-width="2"
                    stroke-linecap="round"
                    stroke-linejoin="round"
                  >
                  </path>
                </svg>
                <div class="flex text-sm text-gray-600">
                  <label
                    for="file-upload"
                    class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500"
                  >
                    <span phx-click={js_exec("##{@uploads.member_photo.ref}", "click", [])}>
                      Upload file
                    </span>
                    <.live_file_input upload={@uploads.member_photo} class="sr-only" tabindex="0" />
                  </label>
                  <p class="pl-1">or drag and drop</p>
                </div>
                <p class="text-xs text-gray-500">
                  images up to 20MB
                </p>
              </div>
            </div>
          </div>
        </div>
        <!-- /upload -->
        <:actions>
          <.button phx-disable-with="Saving...">Save Member</.button>
        </:actions>
      </.simple_form>

      <%!-- <form id="upload-form" phx-submit="save" phx-change="validate"> --%>
      <%!-- <.live_file_input upload={@uploads.member_photo} /> --%>
      <%!-- <button type="submit">Upload</button> --%>
      <%!-- </form> --%>
    </div>
    """
  end

Note that there are some visual aspects of above code that isn’t really necessary. Please see the hex docs for the drop zone implementation / upload triggers.

Form_Component.ex :: update()

def update(%{member: member} = assigns, socket) do
    changeset = Members.change_member(member)

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> assign(changesets: %{}, error_messages: [])
     |> allow_upload(:member_photo,
       accept: ~w(.jpg .jpeg .png),
       max_entries: 1,
       external: &presign_upload/2
     )}
  end

Form_Component.ex :: presign_upload/2

defp presign_upload(entry, socket) do
    uploads = socket.assigns.uploads
    member = socket.assigns.member
    filename = "#{member.id}-#{member.name}#{Path.extname(entry.client_name)}"
    key = "public/#{filename}"

    config = %{
      region: region(),
      access_key_id: access_key_id(),
      secret_access_key: secret_access_key(),
      url: "https://#{bucket()}.#{account_id()}.r2.cloudflarestorage.com"
    }

    {:ok, presigned_url} =
      SimpleS3Upload.presigned_put(config, bucket(),
        key: key,
        content_type: entry.client_type,
        max_file_size: uploads[entry.upload_config].max_file_size
      )

    meta = %{
      uploader: "S3",
      key: key,
      url: presigned_url
    }

    {:ok, meta, socket}
  end

Form_Component.ex :: what about save_resource()?

Nothing upload-specific per se, you can tweak it once the upload is done - such as storing the filename.

Form_Component.ex :: extra helper functions

  defp file_error(%{kind: :dropped} = assigns),
    do: ~H|<%= @label %>: dropped (exceeds limit of 10 files)|

  defp file_error(%{kind: :too_large} = assigns),
    do: ~H|<%= @label %>: larger than 10MB|

  defp file_error(%{kind: :not_accepted} = assigns),
    do: ~H|<%= @label %>: not a valid member_photo file|

  defp file_error(%{kind: :too_many_files} = assigns),
    do: ~H|too many files|

  defp file_error(%{kind: :invalid} = assigns),
    do: ~H|Something went wrong|

  defp file_error(%{kind: %Ecto.Changeset{}} = assigns),
    do: ~H|<%= @label %>: <%= translate_changeset_errors(@kind) %>|

  defp file_error(%{kind: {msg, opts}} = assigns) when is_binary(msg) and is_list(opts),
    do: ~H|<%= @label %>: <%= translate_error(@kind) %>|

  defp error_to_string(:too_large), do: "Too large"
  defp error_to_string(:not_accepted), do: "You have selected an unacceptable file type"
  defp error_to_string(:too_many_files), do: "You have selected too many files"
  defp error_to_string(:external_client_failure), do: "External client has failed upload"

  defp access_key_id(), do: Application.get_env(:my_app, :s3_compatible_access_key_id)
  defp secret_access_key(), do: Application.get_env(:my_app, :s3_compatible_secret_access_key)
  defp account_id(), do: Application.get_env(:my_app, :s3_compatible_application_id)
  defp region(), do: "auto"
  defp bucket(), do: "my_bucket_name"

Obviously the config files should have the :my_app, :s3* values.

SimpleS3Upload

The way how Chris McCord originally wrote it is based on using form-based uploads, which isn’t supported in Cloudflare R2. We have to use the way of generating presigned URL.

defmodule MyApp.Uploading.SimpleS3Upload do
  @moduledoc """
  Below is code from Chris McCord, modified for Cloudflare R2

  https://gist.github.com/chrismccord/37862f1f8b1f5148644b75d20d1cb073

  """
  @one_hour_seconds 3600

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string

  """
  def presigned_put(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "PUT",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end

  @doc """
    Returns `{:ok, presigned_url}` where `presigned_url` is a url string
    ## NOTE - I haven't actually tested this but the gist of the idea is correct.
  """
  def presigned_get(config, opts) do
    key = Keyword.fetch!(opts, :key)
    expires_in = Keyword.get(opts, :expires_in, @one_hour_seconds)
    uri = "#{config.url}/#{URI.encode(key)}"

    url =
      :aws_signature.sign_v4_query_params(
        config.access_key_id,
        config.secret_access_key,
        config.region,
        "s3",
        :calendar.universal_time(),
        "GET",
        uri,
        ttl: expires_in,
        uri_encode_path: false,
        body_digest: "UNSIGNED-PAYLOAD"
      )

    {:ok, url}
  end
end

mix.exs - adding :aws_signature to deps

defp deps do
[
  ...
   {:aws_signature, "~> 0.3.1"},
  ...
]
end

Finished!

Thanks to all who left hints in this forum to get it working.
I still haven’t quite figured out how to do the progress bar working, or how to store something so I can retrieve the stored data back (probably just using some filename generating function).

Most Liked

TwistingTwists

TwistingTwists

https://github.com/TwistingTwists/learning-concurrent-elixir/blob/master/sample.exs

Here is a sample app that does upload to S3 directly from the LiveView. It uses the latest Phoenix.LivieView.UploadWriter to do the multipart upload.

I have tested this on clouldflare R2 and it works flawlessly.

:slight_smile:

mekusigjinn

mekusigjinn

Interesting, good to know - nice to have this in the tool chest.
For now in my problem space I’m able to do creation, reading, and deleting using the presigned urls,
so for now I’m probably going to not use the ExAws.S3 feature - but really appreciate your thorough explanation! Thank you!

It’s awesome how much knowledge this forum has :slight_smile: I felt alone trying this out but I’m glad I reached out and posted my experience :+1:

Where Next?

Popular in Discussions Top

mikl
I wanted to capitalize a string, and tried using String.capitalize(). That generally works well, until you try to capitalize a word like...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
nunobernardes99
Hi there Elixir friends :vulcan_salute: In a recent task I was on, I needed to check in two dates which of them is the maximum and which...
New
AlexMcConnell
The reason that Rails is as popular as it is is because it’s very easy for relatively inexperienced developers to get a lot of work done....
588 19675 166
New
mbenatti
Following https://github.com/tbrand/which_is_the_fastest |&gt; https://raw.githubusercontent.com/tbrand/which_is_the_fastest/master/imgs...
New
jer
I’ve been using umbrellas for a while, and generally started off (on greenfield projects at least) by isolating subapps based on clearly ...
New
hazardfn
I suppose this question is effectively hackney vs. ibrowse but we are at a point in our project where we have to make a choice between th...
New
acrolink
How does the two languages compare when it comes to server side application development? Any experiences or ideas? Thank you.
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

Other popular topics Top

ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52774 488
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement