Upload to any S3 object service using :ex_aws library - error: path in URI must be empty

Phoenix Forum Questions / Help
, ,
1

I am trying to upload to Storj S3 buckets.

What I tried?
using ex_aws library to upload. BUt I am getting this error.

:path in URI must be empty or an absolute path if URL has a :host, got: %URI{authority: nil, fragment: nil, host: "gateway.ap1.storjshare.io", 
path: "https:/gateway.ap1.storjshare.io/73bc549931824937b15209d8940ca1ff-voter-id-card-500x500.jpg", port: nil, query: "", scheme: "https", userinfo: nil}

relevant code

    storj_base = "https://gateway.ap1.storjshare.io/"
    file_uuid = UUID.uuid4(:hex)
    image_filename = URI.encode(image_params.filename)

    unique_filename = "#{file_uuid}-#{image_filename}"
    upload_url = "#{storj_base}#{unique_filename}" |> IO.inspect()

    {:ok, image_binary} = File.read(image_params.path)

    bucket_name = System.get_env("BUCKET_NAME")

    image =
      ExAws.S3.put_object(bucket_name, upload_url, image_binary)
      |> ExAws.request!()
2

Second argument of the ExAws.S3.put_object/4 function is the name of the object to be uploaded.

In your code, you’re passing whole URL there. You probably want to pass just the unique_filename as a 2nd argument to the put_object function.

file_uuid = UUID.uuid4(:hex)
image_filename = URI.encode(image_params.filename)

unique_filename = "#{file_uuid}-#{image_filename}"

bucket_name = System.get_env("BUCKET_NAME")

image =
  ExAws.S3.put_object(bucket_name, unique_filename, image_binary)
  |> ExAws.request!()
3 Likes
3

You can customize the scheme, host, and port for ex_aws_s3 in the application config - this section of the docs briefly mentions it: GitHub - ex-aws/ex_aws_s3

I use a temporary Minio instance in my tests, configured like so:

config :ex_aws, :s3,
  scheme: "http://",
  host: "localhost",
  port: 9000,
  access_key_id: "minio",
  secret_access_key: "minio123"
2 Likes
4

I did try that. but could not succeed.

I realised I had to configure it as proposed by @kreiling.io .

Thanks! :slight_smile:

5

strangely enough, can’t see a file in my s3 bucket.

and there is no error.

How can I debug that?

6

Does this approach work if the urls of the OTHER s3 service are constructed differently ?

amazon aws has demo-bucket.amazonaws.com/uuid-filename.mp4 scheme

but storj has http://gateway.ap1.storjshare.io/demo-bucket/uuid/filename.mp4

strangely enough, the calls are NOT creating the object but creating api key

image
image971×404 21.8 KB

I need to dig more.

7

Here is what I tried to debug so far

iex(11)> {:ok, image_binary} = File.read(path)
{:ok,
 <<255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 1, 1, 44, 1,
   44, 0, 0, 255, 219, 0, 67, 0, 16, 11, 12, 14, 12, 10, 16, 14, 13,
   14, 18, 17, 16, 19, 24, 40, 26, 24, 22, 22, 24, 49, 35, ...>>}

iex(13)> unique_filename = "random-#{path}"
"random-voter-id-card-500x500.jpg"

iex(14) ExAws.S3.put_object(bucket_name, unique_filename, image_binary) |> ExAws.request!()
%{
  body: "",
  headers: [
    {"Accept-Ranges", "bytes"},
    {"Content-Length", "0"},
    {"Content-Security-Policy", "block-all-mixed-content"},
    {"Date", "Tue, 22 Feb 2022 03:22:27 GMT"},
    {"Etag", "\"a93e51c8fbcb62ced96a301bdbff1272\""},
    {"Server", "MinIO"},
    {"Vary", "Origin"},
    {"X-Amz-Request-Id", "16D5FDA9DBFA72BB"},
    {"X-Xss-Protection", "1; mode=block"}
  ],
  status_code: 200
}

In iex(15) why is the content length 0 ?
and {"Content-Security-Policy", "block-all-mixed-content"}, maybe why it doesn’t work?

Any leads will be helpful.

8 
 Key: "my-object",

as stated here is needed to decrypt the bucket. Because s3 in storj is server side encrypted.

How I can use that with ex_aws?

9

You’re looking at the content-length of the response which is 0 (hence the body being an empty string the equivalent of <<>>)

Are you able to get the object from the bucket at the key you stored it in?

10

Yes. I am able to download using key .

S3.download can save the file locally. I can’t see the file in my service dashboard :sweat_smile: but i can upload, download and space is being consumed on s3 server.

So, i guess things are working.