@wmnnd: nice!

I have some questions:

1. supports asynchronous uploads - is it support add upload file to queue, so we can send only x files at one time and y files on one storage type at one time
2. Do you support/plan to upload done callback/event?
3. Do you support/plan to create mirrors between storages?
4. Do you support/plan to sync files/folders automatically?
5. Do you support/plan to priorities? For example. You have a 1GB folder, but none of your project storage providers support that big space, so you want to split it automatically for x providers and then download/sync that folder to another machine automatically.

@Eiji: Thank you for your questions!

1. This is not currently implemented but I will add it in the next release! You will be able to limit uploads both overall and per storage Provider.
2. Belt has a concept of Jobs that work similar to Tasks in Elixir. You can easily use that mechanism for creating callbacks already. But I suppose it’s a feature worth exploring. How would you like to see it implemented? Would you want to be able to execute arbitrary code when a Job has been completed? Is this considered good practice in Elixir? So far I haven’t seen many examples of this kind of callback being used.
3. Queries to multiple targets are not there yet but they are on my to-do list already.
4. I think this would exceed the scope of a pure file storage library, but you easily implement something like this on top of Belt.
5. This is an interesting idea but Belt does not currently keep track of the available storage size a provider has available.

Currently, SFTP, S3 and the local filesystem are supported as targets.

Cool, I’m working a lot with AWS right now, so this response may be
colored by spending a lot of time with the AWS CLI, but

1. It seems wrong to have to specify a bucket as an option to config;
   aws s3 ls alone will give me a list of buckets, which is useful for
   further processing

2. I may have missed it, but most aws-related tools support pulling
   secret keys from the standard ~/.aws/credentials file (and reading
   an AWS_PROFILE env var if present). Is that supported? And if
   not would you entertain a PR?

Regardless, thanks for publishing this.

@hassan: Thank you for your feedback!

1. One of the ideas behind Belt is to abstract different storage providers behind a unified API. From how I see S3, only a set of credentials with a bucket is an actual valid storage destination. But maybe it’d be worth adding a wrapper for ExAws.S3.list_buckets/1 to Belt.Provider.S3. Do you think that would be worth it in order to avoid having to work directly with ExAws?
2. Right now, there is no mechanism in Belt that pulls default configuration for any provider. However, I would like to add support for default destinations so you can also use Belt if you don’t need to configure providers at runtime. ExAws which is used by Belt already has support for reading AWS CLI files. How do you imagine using Belt with defaults read from the CLI file? My current approach would be something like this:

{:ok, config} = Belt.Provider.S3.default_config(override_options \\ [])

You could then configure defaults in your config file. A provider could also automatically try to pull common environment variables. What do you think about this idea?

@wmnnd:
1, 3 - nice
2. I think that GenStage will be really good here, so I can listen on what you will produce, so in async mode background job could catch errors and apply callbacks on specified producer events.
4. If we are talking about Producer -> Consumers scenario then it could be good to have a stand alone process that checks differences between shared files/folders, so after example app will got event that something was changed it could call callback to (ask) sync data.

1. One of the ideas behind Belt is to abstract different storage providers behind a unified API. From how I see S3, only a set of credentials with a bucket is an actual valid storage destination. But maybe it’d be worth adding a wrapper for ExAws.S3.list_buckets/1 to Belt.Provider.S3. Do you think that would be worth it in order to avoid having to work directly with ExAws?

Put that way, probably not a high priority

2. Right now, there is no mechanism in Belt that pulls default configuration for any provider. However, I would like to add support for default destinations so you can also use Belt if you don’t need to configure providers at runtime. ExAws which is used by Belt already has support for reading AWS CLI files. How do you imagine using Belt with defaults read from the CLI file? My current approach would be something like this:

{:ok, config} = Belt.Provider.S3.default_config(override_options \ )

You could then configure defaults in your config file. A provider could also automatically try to pull common environment variables. What do you think about this idea?

Good point, let me look at how ExAws handles config first before
offering any more useless suggestions

I’m sure you did your research, but I want to share it anyways: there’s https://flysystem.thephpleague.com/, which does what you created in php. This might be interesting in terms of kick-starting other cloud providers.

@LostKobrakai: Cool, thank you for pointing this out!
I’ll see if I can’t get some good ideas from there Is there something in particular that you like about Flysystem?

New version 0.1.2

I have just released a small update, version 0.1.2.
It comes with the following changes:

Changelog

• Added Belt.delete_all/2 and Belt.delete_scope/3
• Added default configurations with Belt.Provider.default/1
  You can now configure your own defaults with Mix.Config. Belt.Provider.S3 can also make use of the AWS environment variables and AWS CLI config (read more). Thanks to @hassan for suggesting this.

Roadmap

Next up will be:

• Improved configuration of the maximum number concurrent jobs
• Caching of completed job replies.
• Ecto integration.

Feedback and suggestions regarding functionality, code quality and/or documentation are always welcome!

New version 0.1.4

I have just released a little update, version 0.1.4.

Changelog

• Added Belt.Ecto.Config for storing Provider config structs with Ecto
  You can now directly store config structs in your database and Belt will take care of serializing and deserializing those structs so that no manual conversion is necessary. This is what it looks like:

#in migrations

create table(:belt_providers) do
  add :config, :map #Belt.Ecto.Config uses Ecto primitive :map
end

#in schemas

schema "belt_providers" do
  field :config, Belt.Ecto.Config
end

Roadmap

Next up will be:

• Improved configuration of the maximum number concurrent jobs
• Caching of completed job replies.
• Ecto Type for storing FileInfo structs

I’m looking forward to your feedback