darnahsan
I would like to upload file to s3 pulled from a URL without having to store it in a temp file. If I understand it correctly it is doable as ex_aws can take a stream to upload and Req can read the response in stream . I am looking at this for inspiration Upload to S3 from Google Drive by downloading the file in chunks and uploading it to S3 at the same time - #12 by cjbottaro
As of now I am on the 1st part to read the data from URL in a stream trying out the example in Req but it throws this wierd error. I might be missing something obvious
Req.get!("<url>", auth: {:bearer, "<token>"}, into: IO.stream())
** (ArgumentError) argument error
(stdlib 5.2) io.erl:103: :io.put_chars(:standard_io, <<31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 220, 189, 89, 111, 35, 205, 178, 32, 246, 62, 191, 130, 183, 206, 253, 142, 91, 243, 81, 226, 190, 169, 91, 250, 64, 113, 145, 168, 133, 164, 184, 104, 107, 245, 45, 100, 85, 37, 89, 41, ...>>)
(req 0.4.8) lib/req/steps.ex:760: anonymous fn/3 in Req.Steps.finch_stream_into_collectable/5
(finch 0.17.0) lib/finch.ex:365: anonymous fn/3 in Finch.stream/5
(finch 0.17.0) lib/finch/http1/conn.ex:344: Finch.HTTP1.Conn.receive_response/8
(finch 0.17.0) lib/finch/http1/conn.ex:129: Finch.HTTP1.Conn.request/7
(finch 0.17.0) lib/finch/http1/pool.ex:60: anonymous fn/9 in Finch.HTTP1.Pool.request/5
(nimble_pool 1.0.0) lib/nimble_pool.ex:349: NimblePool.checkout!/4
(finch 0.17.0) lib/finch/http1/pool.ex:52: Finch.HTTP1.Pool.request/5
(finch 0.17.0) lib/finch.ex:424: anonymous fn/6 in Finch.stream_while/5
(telemetry 1.2.1) /home/dar/code/gitlab/topgun/maverick/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
(req 0.4.8) lib/req/steps.ex:769: Req.Steps.finch_stream_into_collectable/5
(req 0.4.8) lib/req/request.ex:992: Req.Request.run_request/1
(req 0.4.8) lib/req/request.ex:937: Req.Request.run/1
(req 0.4.8) lib/req.ex:1027: Req.request!/2
I can read the file fine without the into: IO.Stream
Req.get!("<url>", auth: {:bearer, "<token>"})
%Req.Response{
status: 200,
headers: %{
"accept-ranges" => ["bytes"],
"cache-control" => ["private, no-cache, no-store, must-revalidate"],
"connection" => ["keep-alive"],
"content-digest" => ["adler32=3941013662"],
"content-disposition" => ["inline;filename=File.jpg"],
"content-type" => ["image/jpeg"],
"cross-origin-resource-policy" => ["cross-origin"],
"date" => ["Fri, 02 Feb 2024 03:08:48 GMT"],
"etag" => ["-6143874524592237114"],
"x-manifold-obj-blobsizebytes" => ["13382"],
"x-manifold-obj-canonicalpath" => ["flat/752350453469807"],
"x-manifold-obj-ctime" => ["1706798945"],
"x-manifold-obj-entrytype" => ["1"],
"x-manifold-obj-expiresat" => ["0"],
"x-manifold-obj-isuserdata" => ["0"],
"x-manifold-obj-propertiesjson" => ["{\"blob_state\":\"created\",\"__ctime\":\"1706798944\"}"],
"x-manifold-obj-symlinktarget" => [""]
},
body: <<255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 1, 0, 0, 0, 0, 0,
0, 255, 219, 0, 67, 0, 11, 9, 9, 7, 9, 9, 7, 9, 9, 9, 9, 11, 9, 9, 9, 9, 9,
9, 11, 9, 11, 11, ...>>,
trailers: %{},
private: %{}
}
Trending in Questions
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
Hello,
I’m trying to build a basic Phoenix web-app, and I’d like to use Tailwind.
However, when I launch mix phx.server, I get an error...
New
I really like the adapter patterns that ecto, nebulex, waffle, etc. use and would love find something similar for a key management servic...
New
Hello folks!
So at work, we are seeing some situations where we have to define some “fixed” strings that are used across the codebase in...
New
I’m working on a small exercise involving update_in/3, and I came up with this solution:
data = %{
name: "Periodic Table",
category:...
New
I’ve got trouble wrapping my head around the order in which functions are called in this snippet (from Phoenix’s authentication):
toke...
New
Is there any way to avoid the Hologram compiler running when using iex? It seems like the front-end code could potentially be disregarded...
New
Other Trending Topics
Edit: 2026 May 15 - This post is archived.
Mob is alive!!
Main docs: mob v0.7.11 — Documentation
A bit of explanation for the slightly c...
New
Hobbes is a low-level distributed database for the Elixir programming language.
Hobbes provides a simple, safe, and scalable storage lay...
New
A little off-topic, but I feel like people here have a good head on their shoulders.
I used to be quite good at making software. Was luc...
New
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #ai
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #metaprogramming











Showing Posts 1 to 3- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
darnahsan
so there is an issue on the project’s github that answers and get past this error
darnahsan
when content type is
"content-type" => ["text/html; charset=\"utf-8\""],it works but for"content-type" => ["image/jpeg"]it fails , which brings me to the question what stream to use to support all types if possibledarnahsan
I assume its not possible with Req at the moment Streaming from one place to another · Issue #268 · wojtekmach/req · GitHub