(Preview) Capsule: modular file upload and storage

Today, I’m releasing what I’m calling a “preview” version of a new file upload utility for Elixir (with optional Ecto integration). If you want to get straight to the code, the source README should outline the basics. If you’d like to hear more about why, rationale is below.

Last week I needed to add a pretty simple upload feature to an app I’m working on. Since this isn’t something I’ve done before in Elixir/Phoenix, I posted here asking what others were doing. I didn’t get much of a response, which leads me to believe people were either using Arc, Waffle (the fork of Arc) or just creating their own solution. I wasn’t super excited about either lib, and my use case was pretty simple, so I decided to implement my own custom solution. But after a few hours of playing with it I decided it might be worth separating out and offering my own library.

On my last big Rails project, which involved a good amount of file upload and processing, we used Shrine. Despite the usual gripes with dealing with opaque behavior and callback mazes one has with a lot of Ruby projects that tightly couple with ActiveRecord (and Shrine isn’t even that coupled), I was really impressed with its power and simple API, and the plugin system that gave you access to a kind of “middleware” layer between the upload and the storage phases (although in practice they ended up somewhat entangled in sometimes unexpected and confusing ways). Almost immediately I could see that Elixir offered some really exciting possibilities to build a similar design, but with even more clarity and simplicity.

From the beginning I knew I wanted the following:

  1. Modular system so I could use just what I needed and nothing more
  2. Flexible interface so I could build custom use cases on top of that with minimal hassle
  3. Clean sensible DSL so I could avoid the cognitive load of FS details when I just want to move file from A to B

What excited me was not only how easy it was to accomplish (I think) these goals in a relatively short period of time, but the increasing conviction that the stuff I wasn’t planning to bother to do, ostensibly because other libraries already did it fairly well, I didn’t need to do and in fact shouldn’t do. This is probably the most opinionated aspect of this library and the reason I’m sharing it here. Versions, “convert” commands, backgrounding, these are things are certainly possible to do with Elixir. But once you are adding those kind of features to file upload in an app, two things are true: 1) there are some details about what you need that no library will be able to predict, and so can’t support well, and if you have the three things above, the added work to add these on top is vanishingly small.

As I worked on this, I’ve been thinking back to one of my first impressions of (what I take to be) the “Elixir Way” (every language has its Way, right?) when I started learning Ecto syntax a few years ago, one I’ve since seen repeated many times by other new learners. Where is my Model.last? Do I really have to type MyApp.Context.Model |> Ecto.Query.last |> MyApp.Repo.one? The point I started really getting into Elixir is when I realized this wasn’t a defect in the design, or even an oversight, but an intentional decision built deep into Ecto, Elixir itself, and I assume at least in part, the underlying VM. And that was another impression–how much the people I could tell really understood Elixir talked about Erlang! I remember thinking, “isn’t this an Elixir forum??” How often do you catch a Rubyist waxing eloquent about C? Not often…

So, as usual, I’d welcome any feedback the community has about my approach here. Specifically I’d love to have a conversation around the following questions:

  1. Does the model make sense? Are there common (or even uncommon) file upload patterns that would be difficult to implement with the current design?
  2. Is the DSL sensible? Should I choose different terminology for any of the API?
  3. (And because I’m arguably somewhat out of my depth as a web dev here) Are there especially serious security/performance issues I should consider before moving forward?

Thanks!

15 Likes

I was asked a question via DM about handling file cleanup that prompted me to add a section to the capsule_ecto README (it applies equally well without Ecto, although the need might not be quite as obvious):

4 Likes

Thank you!

1 Like

Kicking off Hacktoberfest with a new release.

Aside from various bugfixes, it contains 2 significant, non-backwards-compatible changes:

  • The Storage.move callback has been removed in favor of copy. If you were using this API you will need to replace it with a combination of copy and delete
  • All built in Storage and Upload implementations have been relocated to a new, separate repo: https://github.com/elixir-capsule/supplement

The latter comes with a new Storage and Upload implementation as well: S3 and Plug.Upload respectively.

I’ll be looking to add more, so if you want to get your t-shirt (or tree) feel free to pitch in :wink:

2 Likes

0.7 is a small release that extends options support to all storage callbacks, which is used in the latest version of the S3 and Disk storages to allow overriding configs (bucket and root directory) in specific calls.

1 Like

Another small maintenance release!

0.8 contains a breaking change to the API. Capsule.Storage.open is now Capsule.Storage.read.

Supplement also now includes a RAM storage for putting files in memory (via StringIO). After running into a confusing dependency conflict issue, I also published a versioned release on hex: https://hex.pm/packages/capsule_supplement

I haven’t been running into many issues thus far, so the first stable release should be coming soon. So, if you have been trying it out and have pain points, or want to try it out, speak your feedback now or forever hold your peace…or at least until the next major release :grinning_face_with_smiling_eyes:

1 Like