ExAws always get maximum 1000 records from S3

I am using ExAws library in my project to upload and download files from S3. Everything is fine and working.

But It always give maximum 1000 records.

Here is code which i am using to get list of objects:

ExAws.S3.list_objects("camera-assets", max_keys: 1500, prefix: "snapshots/", delimiter: "/") |> ExAws.request!

I want to get maximum 1500 records and I added max_keys param as described in ExAws docs.

I also debugged code and it shows max_keys passed in request but it returns 1000 only.

https://s3-eu-west-1.amazonaws.com/camera-assets/?delimiter=%2F&encoding-type=url&max-keys=1500&prefix=everc-fhlcr%2Fsnapshots%2F2017%2F10%2F16%2F

How to solve this issue?

believe there is a 1000 limit in the api from S3 - http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

the first response should have a NextContinuationToken that you can send in another request as the continuation-token - I haven’t tested with exaws, so ymmv

As @outlog notes, this is an AWS limit. I would consider using ExAws.stream! which can make additional requests behind the scenes:

ExAws.S3.list_objects("camera-assets", [
  prefix: "snapshots/",
  delimiter: "/"
]) |> ExAws.stream! |> Enum.take(1500)
4 Likes

Hi @azharmalik3,
I get the response like below text. How can u get the key value?
Can give me a sample code?

{:ok,
%{
body: %{
common_prefixes: [],
contents: [
%{
e_tag: ““d41d8cd98f00b203ksm00998ecf8427e””,
key: “img/xxx/”,
last_modified: “2018-11-15T04:15:20.000Z”,
owner: %{
display_name: “test”,
id: “e9d42832cdasdasdbab7sdaeacfdee6b5d3csw369c6dc339a13e87cs0”
},
size: “0”,
storage_class: “STANDARD”
},
%{
e_tag: "“e8daeb48d4d1028snso9eaf6753b3833"”,
key: “img/xxx/zzz.svg”,
last_modified: “2018-11-15T04:15:40.000Z”,
owner: %{
display_name: “test”,
id: “e9d42832cddfe3bf5lw1nsoqe6b5d3c64f4528e369c6dc339a13e87sdq1”
},
size: “1149”,
storage_class: “STANDARD”
}
],
is_truncated: “false”,
marker: “”,
max_keys: “1000”,
name: "s3-test”,
next_marker: “”,
prefix: “img/xxx/”
},
headers: [
…….
],
status_code: 200
}}