abitdodgy

abitdodgy

@wmnnd I’m not using Belt yet. But in Refile it works like this:

When you add files to the form, Refile makes a request to the app to get a signed token, then using this signed token the app uploads the files directly to AWS and writes the results to a JSON field to store the meta data. This is great because I can have multiple uploads running concurrently, and submit the form and save the record without having much overhead or IO blocking.

Where Refile fails, however, is that it does on the fly processing, which means requested images are processed on the fly from AWS and served through a rack app. This is a major headache on PaaS services like Heroku, even with a CDN, where a page load with lots of avatars causes timeouts as the rackapp struggles to process and serve each image.

The ideal way would be to be able to directly upload to AWS and request the files directly from AWS or via CDN without having the app get involved. Any processing can happen in the background.

In Refile, much of this direct uploading is powered by circa 100 lines of javascript.

0
Post #21
wmnnd

wmnnd OP

Getting the signed upload URL isn’t much of a problem but the JS and form-building part is what worries me here. Right now, Belt is completely independent from Phoenix and/or Plug even if they play together very nicely.

A belt_phoenix or belt_plug module that offers a provider-agnostic way of getting direct or indirect upload links would probably be nice but it’s not at the top of my to-do list right now.
I’d imagine the API to be relatively low-level and something along the lines of Belt.upload_link(config, opts). What do you think about this idea?
Installation for a more high-level module would also be a little more complicated than for Refile since Phoenix doesn’t have a global asset pipeline that allows for the adding of global helper functions and/or JavaScript.

For now, I suggest simply trying if using Belt works out for you even without direct S3 uploads (which are also not supported by Arc, by the way). If your application isn’t, say, a file hoster, this should not pose a major problem :slight_smile: Otherwise you’ll have to build something yourself with ExAWS.

1
Post #22
abitdodgy

abitdodgy

I’d imagine the API to be relatively low-level and something along the lines of Belt.upload_link(config, opts). What do you think about this idea?

I think that would be great. We could then build our own JS implementations around the lib.

0
Post #23
kwando

kwando

Do I have to do something special when using Belt.Provider.S3 in a test case? It justs blow up with this error all the time :confused:

(I haveex_aws and sweet_xml in my mix.exs)

16:59:18.658 [error] GenServer Belt terminating
** (RuntimeError) provider Belt.Provider.S3 not registered with Belt
    (belt) lib/belt.ex:284: anonymous fn/1 in Belt.init/1
    (gen_stage) lib/gen_stage/partition_dispatcher.ex:214: anonymous fn/3 in GenStage.PartitionDispatcher.dispatch/3
    (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
    (gen_stage) lib/gen_stage/partition_dispatcher.ex:213: GenStage.PartitionDispatcher.dispatch/3
    (gen_stage) lib/gen_stage.ex:2309: GenStage.dispatch_events/3
    (gen_stage) lib/gen_stage.ex:1960: GenStage.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.519.0>): {:store, [%Belt.Provider.S3.Config{access_key_id: "<redacted>", base_url: "https://<redacted>", bucket: "demo-data", host
: "static.bemt.eu", https: true, port: 443, provider: Belt.Provider.S3, region: "us-west-2", secret_access_key: "<redacted>"}, "/Users/kwando/projects/

work/SMS/setup.sh", [key: "840277a243d9f9769a71d56285d2569489fb21747f0d32b4db302c77a561b10c3842c579ffc20c28"]]}
16:59:18.662 [error] GenServer Belt.Provider.Filesystem.Supervisor terminating
** (RuntimeError) provider Belt.Provider.S3 not registered with Belt
    (belt) lib/belt.ex:284: anonymous fn/1 in Belt.init/1
    (gen_stage) lib/gen_stage/partition_dispatcher.ex:214: anonymous fn/3 in GenStage.PartitionDispatcher.dispatch/3
    (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
    (gen_stage) lib/gen_stage/partition_dispatcher.ex:213: GenStage.PartitionDispatcher.dispatch/3
    (gen_stage) lib/gen_stage.ex:2309: GenStage.dispatch_events/3
    (gen_stage) lib/gen_stage.ex:1960: GenStage.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:DOWN, #Reference<0.611478655.3527409665.27844>, :process, #PID<0.311.0>, {%RuntimeError{message: "provider Belt.Provider.S3 not registered with Belt"}, [{Belt, :"-in
it/1-fun-0-", 1, [file: 'lib/belt.ex', line: 284]}, {GenStage.PartitionDispatcher, :"-dispatch/3-fun-0-", 3, [file: 'lib/gen_stage/partition_dispatcher.ex', line: 214]}, {Enum, :"-r
educe/3-lists^foldl/2-0-", 3, [file: 'lib/enum.ex', line: 1811]}, {GenStage.PartitionDispatcher, :dispatch, 3, [file: 'lib/gen_stage/partition_dispatcher.ex', line: 213]}, {GenStage
, :dispatch_events, 3, [file: 'lib/gen_stage.ex', line: 2309]}, {GenStage, :handle_call, 3, [file: 'lib/gen_stage.ex', line: 1960]}, {:gen_server, :try_handle_call, 4, [file: 'gen_s
erver.erl', line: 636]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 665]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}
0
Post #24
wmnnd

wmnnd OP

@kwando Hey there, thanks for giving Belt a try! You need to register the providers that you want to use in your mix configuration.
Here is the example from the Getting Started guide:

config :belt,
  providers: [Belt.Provider.Filesystem, Belt.Provider.SFTP, Belt.Provider.S3]
0
Post #25
wmnnd

wmnnd OP

Small SFTP update 0.2.0

Version 0.2.0 of Belt brings improved SFTP transfer performance (as much as 10 times faster transfer speed) thanks to asynchronous writing as well as better support for different SFTP servers.

What’s next?

Belt 0.x will continue to receive updates and soon I’ll add a feature to test configurations without the need to upload something or use one of the other commands first.
A rewrite is coming for version 1.x which will bring new Ecto-style configuration and direct uploads - both for S3 and by proxy for SFTP as well.

4
Post #26
aislan

aislan

Would be awesome if this bring delete feature to S3. That’s is the very problem I’m having right now with Arc and Arc_ecto.

0
Post #27
wmnnd

wmnnd OP

Which delete feature are you referring to? Belt.delete/3?

0
Post #28
jaimeiniesta

jaimeiniesta

Nice library, thanks!

Is there a way to store a file without reading from disk? I mean, in my app I want to store a file in SFTP, but its contents don’t come from an existing file, they’re generated in memory. Is there a way to upload directly those or should I write to disk first?

1
Post #29
NobbZ

NobbZ

This depends on the SFTP lib you use.

0
Post #30