benonymus
Delete file from aws(arc) fom postgres trigger
Hey there,
I am using arc for file processing with aws, and I am trying to delete images from aws once an item is deleted
from my db.
I am using postgres triggers(https://blog.lelonek.me/listen-and-notify-postgresql-commands-in-elixir-187c49597851), so I am getting the item form the database, arc for the files only saves a string to the database such as:
test_document_fEfRY-pR2JX3jIF9CrXPC.pdf?63754075895
My problem is, since my trigger is upon deletion I cannot get the item by id from Repo to have the file taken care of by arc.
The attachment inside my item instead of a map is just a string.
I am usually using arc’s delete from the avatar with the image and entity but without having the item going through ecto it just won’t delete by that string that I mentioned above.
Any thoughts how could I get the attachment deleted with the string or how to load that attachment manually in order to be able to delete it?
First Post!
dimitarvp
Most Liked
benwilson512
I would probably invert this whole process. Don’t delete the record from the database first. Mark the database record as deleted_at pending_deletion or something, and then enqueue a job in something like Oban which can try to hit AWS to delete the record. Only after that has succeeded do you delete the record in the DB. The flag lets you filter it out from UI lists immediately, but you want the record around to reason about the file until it’s actually gone.
The job part here is critical. AWS S3 has good uptime, but not perfect up time. Requests to delete files will fail periodically and will need to be retried.
baldwindavid
If you like Arc, you might take a look at Waffle (I've forked Arc to Waffle and merged all open PRs). Arc seems to have gone unmaintained and Waffle is a fork with fixes. I prefer the transparency and control of working directly with ex_aws_s3 and orchestrating uploads and deletions via Ecto.Multi, but understand the allure of an opinionated package.
dimitarvp
Hm, I see. In this case I recommend you switch approaches. You’re better off putting a “please delete attachmend with ID 123” command on a message queue (RabbitMQ, Kafka, you can also use your DB as well) and then have a background worker that periodically works through those and just have that delete the files from AWS.
Last Post!
benonymus
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"