Wondering if anyone has good recommendations for associating file storage with Ecto models?
We’re working on a document conversion service that accepts either uploaded documents or remote URLs. Documents are downloaded into local S3-compatible storage and converted.
I’ve been trying to use Arc, which almost does what I need, but am running into some issues. The most recent is this one, where we’re told not to cast the field containing the attachment, but the field doesn’t reach my Repo.update call at all unless I do. This works for folks in Ecto 3.1, but I’m using 3.2, and am wondering if something has changed.
I’m also looking at Belt, but am also considering just rolling my own. Essentially, we’re only storing files in S3-compatible storage. Conversions are handled by our own service. I don’t know if there’s some amazing file storage library Google just isn’t showing me, or if these don’t add much beyond what I can assemble on my own. I do like Arc’s extension whitelisting features, as well as how it transparently handles downloading remote URLs or handling files. Something similar for Belt might work, as it doesn’t appear to fetch files remotely.
And any thoughts on the previously-linked arc_ecto issue would be great. If I can stick with it then I’d like to, but I’m not sure what I’m missing. Thanks for any help.
My general rule - if storing files isn’t your main application raison d’être then do not handle it at all. Just send user the signed URL where they are supposed to send the file they want (S3-like) and make user browser handle everything without your application being involved in the traffic at all. In that way you do not need any library at all (maybe except for S3 signing as this can be troublesome sometimes, but it is much smaller than whole Arc).
That’s great! I’ll give it a shot when I continue working on this
project on Monday.
Just to sanity-check an assumption I got wrong earlier this week and am
working to correct. If I have an Ecto field called :original, to which
I’ve assigned my uploader type, I associate files with that field by
assigning either a URL or a file upload to that field? I was calling store({url, scope} before, and that downloaded my file, but it didn’t
assign anything to my :original Ecto field. I’m now tweaking my
controller to assign my URL directly to that field, and that’s when
everything went to hell and I started hitting Arc issues.
Anyhow, just wanted to make sure I was on the right track. I’ve never
seen a formal specification of what I’m permitted to assign to the Ecto
model field when Arc/Waffle is used with a scope, or of what the model
field will contain once the file is associated. I’m particularly
interested in this because I’m migrating an app to Elixir, so we have a
bunch of existing data, and I’m trying to ensure that my legacy data
eventually lines up with whatever my library of choice expects to find
in the field.
I’m maintainer of the Waffle library. We’ve discussed this issue with @ndarilek at the github. In case if someone would have the same issue I’ve updated the docs and described how to use :id in filepath on creation stage here at hexdocs.
I would be glad to answer if you have any additional questions.
Oh nice!
That fixes another issue I’m facing. When I use unique changeset for the resource to be inserted and its attachment, if the insert fails due to database constraint for example the attachment is stored anyway. The tricky solution I came to was removing myself the uploaded file. But with your solution I won’t have to bother with that because the attachment will be treated only when everything is really ok with the insertion.
