Req stream response body upload to s3 using ExAWS

Questions / Help
, ,
1

I would like to upload file to s3 pulled from a URL without having to store it in a temp file. If I understand it correctly it is doable as ex_aws can take a stream to upload and Req can read the response in stream . I am looking at this for inspiration Upload to S3 from Google Drive by downloading the file in chunks and uploading it to S3 at the same time - #12 by cjbottaro
As of now I am on the 1st part to read the data from URL in a stream trying out the example in Req but it throws this wierd error. I might be missing something obvious
Req.get!("<url>", auth: {:bearer, "<token>"}, into: IO.stream())

** (ArgumentError) argument error
    (stdlib 5.2) io.erl:103: :io.put_chars(:standard_io, <<31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 220, 189, 89, 111, 35, 205, 178, 32, 246, 62, 191, 130, 183, 206, 253, 142, 91, 243, 81, 226, 190, 169, 91, 250, 64, 113, 145, 168, 133, 164, 184, 104, 107, 245, 45, 100, 85, 37, 89, 41, ...>>)
    (req 0.4.8) lib/req/steps.ex:760: anonymous fn/3 in Req.Steps.finch_stream_into_collectable/5
    (finch 0.17.0) lib/finch.ex:365: anonymous fn/3 in Finch.stream/5
    (finch 0.17.0) lib/finch/http1/conn.ex:344: Finch.HTTP1.Conn.receive_response/8
    (finch 0.17.0) lib/finch/http1/conn.ex:129: Finch.HTTP1.Conn.request/7
    (finch 0.17.0) lib/finch/http1/pool.ex:60: anonymous fn/9 in Finch.HTTP1.Pool.request/5
    (nimble_pool 1.0.0) lib/nimble_pool.ex:349: NimblePool.checkout!/4
    (finch 0.17.0) lib/finch/http1/pool.ex:52: Finch.HTTP1.Pool.request/5
    (finch 0.17.0) lib/finch.ex:424: anonymous fn/6 in Finch.stream_while/5
    (telemetry 1.2.1) /home/dar/code/gitlab/topgun/maverick/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
    (req 0.4.8) lib/req/steps.ex:769: Req.Steps.finch_stream_into_collectable/5
    (req 0.4.8) lib/req/request.ex:992: Req.Request.run_request/1
    (req 0.4.8) lib/req/request.ex:937: Req.Request.run/1
    (req 0.4.8) lib/req.ex:1027: Req.request!/2

I can read the file fine without the into: IO.Stream

Req.get!("<url>", auth: {:bearer, "<token>"})

%Req.Response{
  status: 200,
  headers: %{
    "accept-ranges" => ["bytes"],
    "cache-control" => ["private, no-cache, no-store, must-revalidate"],
    "connection" => ["keep-alive"],
    "content-digest" => ["adler32=3941013662"],
    "content-disposition" => ["inline;filename=File.jpg"],
    "content-type" => ["image/jpeg"],
    "cross-origin-resource-policy" => ["cross-origin"],
    "date" => ["Fri, 02 Feb 2024 03:08:48 GMT"],
    "etag" => ["-6143874524592237114"],
    "x-manifold-obj-blobsizebytes" => ["13382"],
    "x-manifold-obj-canonicalpath" => ["flat/752350453469807"],
    "x-manifold-obj-ctime" => ["1706798945"],
    "x-manifold-obj-entrytype" => ["1"],
    "x-manifold-obj-expiresat" => ["0"],
    "x-manifold-obj-isuserdata" => ["0"],
    "x-manifold-obj-propertiesjson" => ["{\"blob_state\":\"created\",\"__ctime\":\"1706798944\"}"],
    "x-manifold-obj-symlinktarget" => [""]
  },
  body: <<255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 1, 0, 0, 0, 0, 0,
    0, 255, 219, 0, 67, 0, 11, 9, 9, 7, 9, 9, 7, 9, 9, 9, 9, 11, 9, 9, 9, 9, 9,
    9, 11, 9, 11, 11, ...>>,
  trailers: %{},
  private: %{}
}
2

so there is an issue on the project’s github that answers and get past this error

3

when content type is "content-type" => ["text/html; charset=\"utf-8\""], it works but for "content-type" => ["image/jpeg"] it fails , which brings me to the question what stream to use to support all types if possible

4

I assume its not possible with Req at the moment Streaming from one place to another · Issue #268 · wojtekmach/req · GitHub