wmnnd
Belt - A Flexible File-Storage Library
Hi there,
for my project DBLSQD, I needed a file storage solution that is a bit more flexible than Arc. Because I thought others might find it useful as well, I decided to share it with everyone and release it as free software. You can install version 0.1.1 from hex.pm.
What can it do?
Belt allows you to store files on remote or local systems. Currently, SFTP, S3 and the local filesystem are supported as targets. Belt is especially useful if you want to target multiple storage systems that are configured at runtime (e. g. because they are user-provided).
Belt is built on top of GenStage and also supports asynchronous uploads.
Belt also offers some convenience functions such as retrieving hashes of a file.
How does it work?
Here is a little example on how to use Belt:
#Simple file upload
{:ok, config} = Belt.Provider.SFTP.new(host: "example.com", directory: "/var/files",
user: "…", password: "…")
Belt.store(config, "/path/to/local/file.ext")
#=> {:ok, %Belt.FileInfo{…}}
#Asynchronous file upload
{:ok, config} = Belt.Provider.S3.new(access_key_id: "…", secret_access_key: "…",
bucket: "belt-file-bucket")
{:ok, job} = Belt.store_async(config, "/path/to/local/file.ext")
#Do other things while Belt is uploading in the background
Belt.await(job)
#=> {:ok, %Belt.FileInfo{…}}
You can read more on how to install and configure Belt in the Getting Started guide.
Roadmap
Belt is definitely usable the way it is right now but I have planned several additional features such as an Ecto integration.
Feedback welcome!
Suggestions for useful new features are more than welcome.
Also, if you discover bugs or find the documentation lacking in any way, please file a bug report.
Most Liked
wmnnd
Upload in-memory data with Belt 0.4.0!
Thanks to @jayjun’s fantastic work, a brand-new feature is now available in Belt: Uploading iodata directly to all Providers without the need to go through a file first.
Belt.store_data/3 works just like Belt.store/3, except that it takes iodata instead of a file path as the second argument and the :key option is mandatory.
Of course, it’s also accompanied by Belt.store_data_async/3 
I’ve also updated the ex_aws dependency to the 2.x branch.
I originally wasn’t sure if this is the right thing to do, since it’s a major version update for this (optional) dependency and just a minor version update for Belt.
But since there is a bug in ExAws 1.x which breaks it on Elixir 16/OTP 21 and @benwilson512 pointed out that ExAws 2.x is “plug and play compatible” with 1.x, I decided this was the way to go 
As always, Feedback is very welcome.
wmnnd
#New version 0.1.8
Since DBLSQD is finally up and running, there is at least one instance of Belt out in the wild now 
More features will come as DBLSQD and Belt evolve. For now, these are the changes of the most recent release:
Changelog
- Improvement: When storing files on SFTP, hashes are now calculated locally. Retrieving hashes when none were requested is now a noop.
- Improvement: Better compatibility with SFTP servers with small maximum package sizes. Thanks to the obscure and undocumented (yet public) functions
:ssh_sftp.send_window/2and:ssh_sftp.recv_window/2, package size is now determined dynamically. - Fixed bug causing infinite loops when listing files on some SFTP servers
- Fixed connecting to SFTP servers with
verify_hostset tofalse
##roadmap
- Make jobs more flexible
- Explore the possibility of using ETS directly or ConCache as a backend for jobs
Feedback and suggestions are welcome as always!
wmnnd
Small SFTP update 0.2.0
Version 0.2.0 of Belt brings improved SFTP transfer performance (as much as 10 times faster transfer speed) thanks to asynchronous writing as well as better support for different SFTP servers.
What’s next?
Belt 0.x will continue to receive updates and soon I’ll add a feature to test configurations without the need to upload something or use one of the other commands first.
A rewrite is coming for version 1.x which will bring new Ecto-style configuration and direct uploads - both for S3 and by proxy for SFTP as well.
Popular in Announcing
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"