Trouble configuring ARC for S3 image uploads

Hello, I’m having a problem configuring ARC to work in my app. First, when I use the configuration as described below, the file is loaded into local storage. If I add the line

def __storage, do: Arc.Storage.Local

to uploader.ex, I get an error – not too informative: “something went wrong.” Also, when I try a direct upload, here is what happens:

iex(2)> Avatar.store("/Users/carlson/Downloads/jupiter.jpg")                
{:ok, "jupiter.jpg"}

However, when I check my S3 bucket, I find only a few old items. And nothing is added to local storage

  1. CONFIG
# files: config.dev, config.prod
config :arc,
 storage: Arc.Storage.S3,
 bucket: "noteimages"

config :ex_aws,
 access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
 secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY")
  1. MIX.EXS
# file: mix.exs:
 defp deps do
    [{:phoenix, "~> 1.2.1"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_ecto, "~> 3.0"},
     {:postgrex, ">= 0.0.0"},
     {:phoenix_html, "~> 2.6"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"},
     {:comeonin, "~> 2.0"},
     {:mnemonix, "~> 0.2.0"},
     {:timex, "~>3.0"},
     {:timex_ecto, "~> 3.0"},
     {:benchfella, "~> 0.3.2"},
     {:json, "~> 1.0"},
     {:joken, "~> 1.1"},
     {:arc, "~> 0.8.0"},
     {:arc_ecto, "~> 0.4", override: true},
     {:shared, in_umbrella: true},
     {:ex_aws, "~> 1.1"},
     {:poison, "~> 2.0"},
     {:httpoison, "~> 0.11"},
     {:hackney, "~> 1.6"}]
  end
  1. S3 Bucket policy
{
    "Version": "2012-10-17",
    "Id": "Policy1493266128611",
    "Statement": [
        {
            "Sid": "Stmt1493266101437",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::noteimages/*"
        }
    ]
}

what region is your s3 bucket in?

It is in US East; I am getting these errors:

[warn] ExAws: HTTP ERROR: :nxdomain

check your ex_aws config it seems wrong (uses a list & not System.get_env in their docs) https://github.com/CargoSense/ex_aws

then see if you can list buckets or something ExAws.S3.list_buckets |> ExAws.request

1 Like

also - unless you are on us-east-1 then setup the region:

config :ex_aws,
  region: "eu-central-1",
  s3: [
    scheme: "https://",
    host: "s3.eu-central-1.amazonaws.com",
    region: "eu-central-1"
  ]

and

config :arc,
  virtual_host: true

That was a great suggestion. Ran the code and got this reply, confirming that the app knows about the bucket:

iex(2)> ExAws.S3.list_buckets |> ExAws.request
{:ok,
 %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<ListAllMyBucketsResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Owner><ID>a09476c61701fa4229ec1e38b08341df146b41d8b8b949d4039a69b2716f0354</ID><DisplayName>carlson618</DisplayName></Owner><Buckets><Bucket><Name>13850ffd-33f5-49fd-8fc6-a80c3d348e7b</Name><CreationDate>2015-12-15T15:05:52.000Z</CreationDate></Bucket><Bucket><Name>954a1a28-4bd6-488e-a0d2-c3e44235e4d8</Name><CreationDate>2015-12-15T15:06:05.000Z</CreationDate></Bucket><Bucket><Name>b7f82009-ea20-4a26-a249-1cbdb77def55</Name><CreationDate>2015-12-15T15:05:58.000Z</CreationDate></Bucket><Bucket><Name>c57faebd-5013-46ee-97a2-03a2589fcc13</Name><CreationDate>2015-12-15T15:05:51.000Z</CreationDate></Bucket><Bucket><Name>ey-backup-de6c03151f3d</Name><CreationDate>2015-12-16T02:37:50.000Z</CreationDate></Bucket><Bucket><Name>ey-backup-f33c93c356f9</Name><CreationDate>2015-12-16T02:37:50.000Z</CreationDate></Bucket><Bucket><Name>ey-keys-26919-7df27d7a-5904-41aa-9b06-a05e6716943b</Name><CreationDate>2016-10-28T10:17:05.000Z</CreationDate></Bucket><Bucket><Name>jcng.memories</Name><CreationDate>2016-12-26T20:31:33.000Z</CreationDate></Bucket><Bucket><Name>notefile</Name><CreationDate>2017-04-27T03:37:53.000Z</CreationDate></Bucket><Bucket><Name>noteimages</Name><CreationDate>2017-04-27T03:40:04.000Z</CreationDate></Bucket><Bucket><Name>nsx-test</Name><CreationDate>2015-04-03T04:14:23.000Z</CreationDate></Bucket><Bucket><Name>psurl</Name><CreationDate>2016-07-28T06:24:16.000Z</CreationDate></Bucket><Bucket><Name>s3hub-a09476c61701fa4229ec1e38b08341df146b41d8b8b949d4039a69b27</Name><CreationDate>2013-09-07T23:15:32.000Z</CreationDate></Bucket><Bucket><Name>vschool</Name><CreationDate>2013-09-07T23:12:54.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>",
   headers: [{"x-amz-id-2",
     "fvPhszH9DdILoKdrbS949cewwR4L3nqympzbxcwvrNT/K7U67COVmwuiRLm9Sg6Z8NpQ/2sZbf4="},
    {"x-amz-request-id", "9153316FD5ECBF75"},
    {"Date", "Wed, 03 May 2017 21:28:17 GMT"},
    {"Content-Type", "application/xml"}, {"Transfer-Encoding", "chunked"},
    {"Server", "AmazonS3"}], status_code: 200}}

However, note the below. I tried to directly upload an image and get an error about (KeyError) key :src not found in: %{body: "<?xml version=\"1.0\" .... (I’ve reworked my arc and ex_aws configs)

iex(2)> Avatar.store("/Users/carlson/Downloads/jupiter.jpg")
** (EXIT from #PID<0.562.0>) an exception was raised:
    ** (KeyError) key :src not found in: %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<InitiateMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Bucket>noteimages</Bucket><Key>uploads/jupiter.jpg</Key><UploadId>nvlNvGedIJA3LMyvatb62D4Ht1SrHBXyn4udTEzHne0eyAVk6cXUs5u3Aipb1xRaWSPoybm93TjnldZ9.H8zhnE7eUn6tptBLSnEN126.OMMHM3F3VVuBS6R_NAMAREK</UploadId></InitiateMultipartUploadResult>", headers: [{"x-amz-id-2", "QEdfearXN1qavEPKRud2UAWHtElW4tLVE/pNGnJeS5e0qBtj+wxtPl5kT1bGE3Yd"}, {"x-amz-request-id", "6532A850BA09FA62"}, {"Date", "Wed, 03 May 2017 21:24:20 GMT"}, {"Transfer-Encoding", "chunked"}, {"Server", "AmazonS3"}], status_code: 200}
        (ex_aws) lib/ex_aws/s3/upload.ex:83: ExAws.Operation.ExAws.S3.Upload.perform/2
        (arc) lib/arc/storage/s3.ex:57: Arc.Storage.S3.do_put/3
        (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
        (elixir) lib/task/supervised.ex:36: Task.Supervised.reply/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

isn’t sweet_xml missing from deps? poison should probably also be bumped to 3.1…

see https://github.com/stavro/arc#installation

defp deps do
  [
    arc: "~> 0.8.0",

    # If using Amazon S3:
    ex_aws: "~> 1.1",
    hackney: "~> 1.6",
    poison: "~> 3.1",
    sweet_xml: "~> 0.6"
  ]
end
1 Like

That was the final step! Thank you very much!!

1 Like

awesome…

btw just bookmarked this today https://medium.com/coletiv-stories/phoenix-with-image-upload-to-s3-in-an-api-implementation-and-testing-6ab5187175b0 might be useful, for testing etc.

1 Like

Looking at it now … thanks!

1 Like

The article you recommended is just what I needed … a way to store each user’s image files in a separate “folder”

great, remember to add security if you are storing private user files/images… non-public bucket and presigned urls is usually the way to do it.

Thx, so far, no worries – but I may need to consult with you later. Project going well – I love Elixir &Phoenix! (burned out dev from Rails project that went wild)

2 Likes