thbar
Hi!
A part of my ETL/data-pipelining work involves dealing with input files whose size can reach 1 to 2 GB in size (compressed).
When working with Ruby, I usually shell out to tools like axel to handle the download in a way that can be resumed, should a disconnection occur, or even accelerated, if the source server is doing some throttling in a way or another.
To your knowledge, is there an already built solution to achieve that in Elixir or Erlang?
Solutions I’m considering:
- Shelling out to Axel using Porcelain (but porcelain has issues apparently)
- Implementing a resumable download manager myself on top of an existing Elixir HTTP library
- Wrapping a Rust download manager such as zou with e.g. rustler
If anyone dived into this topic, your feedback is most welcome!
– Thibaut
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixir-ls
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
idi527
For resumable downloads, I’d think it could be done with most http clients with a bit of work. For example, with httpoison, you’d write the stream of bytes (
stream_to: self()option) to a temporary file and only “save” it (move the temporary file to a “permanent” location similarly to howPlug.Uploadworks) once the download is complete. If there is already a temporary file when a download process starts, it would read it, calculate the “offset” needed for the range header, and only download what it needs to.For accelerated downloads, you’d need to split the file into chunks of some configurable size and download each with the “resumable” downloader described above, then
catthem into a single file at the final destination …This is what I’d try before reaching out to rust or porcelain.
Zesky665
You can just use a Task to run a cli app(using System.cmd\3) to do the same thing.