AlchemistCamp
Upload external files from URLs (automated, no front-end)
I’m looking for a good way to fetch files from URLs and upload them to an S3 bucket. I’ve seen a lot of posts here about uploads, but they’re generally focused on images uploaded by users via a form. The scenario I’m working on is an automated script fetching images from a file server and putting them onto S3.
I have a list of [img_url1, img_url2, img_url3, | _the_rest], and I want to fetch and upload all of those files to my S3 bucket. What’s a good way to go about it?
I’m using Phoenix, but AFIK, that’s probably not very relevant to this task.
Most Liked
achempion
Hey, Waffle author here.
You can do this via the Waffle library like this:
Avatar.store("http://example.com/file.png") #=> {:ok, "file.png"}
You only need to configure access for your S3 storage and create an uploader.
Please, feel free to ask if you’ll have any troubles.
LostKobrakai
I’ll essentially be using your prefered http client library to fetch the files and using your prefered aws library to push the data to S3. From there you could layer optimizations like streaming or parallelism on top if needed.
andreaseriksson
I have used Waffle for this.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Real-time error tracking, performance insights, and observability for devs.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"