Delete file from aws(arc) fom postgres trigger

benonymus

Hey there,

I am using arc for file processing with aws, and I am trying to delete images from aws once an item is deleted
from my db.

I am using postgres triggers(https://blog.lelonek.me/listen-and-notify-postgresql-commands-in-elixir-187c49597851), so I am getting the item form the database, arc for the files only saves a string to the database such as:

test_document_fEfRY-pR2JX3jIF9CrXPC.pdf?63754075895

My problem is, since my trigger is upon deletion I cannot get the item by id from Repo to have the file taken care of by arc.
The attachment inside my item instead of a map is just a string.
I am usually using arc’s delete from the avatar with the image and entity but without having the item going through ecto it just won’t delete by that string that I mentioned above.

Any thoughts how could I get the attachment deleted with the string or how to load that attachment manually in order to be able to delete it?

11 comments

#ecto #postgresql #arc

0 2045 11

2020-04-15 01:09:26 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

dimitarvp

What days is your Elixir callback function actually receiving? Give us an example.

Post #1

benonymus

example:

%{
  operation: "DELETE",
  record: %{
    attachment: "test_docume_nt_fEfRY-pR2JX3jIF9CrXPC.pdf?63754075895"
  }
}

I am getting similar to this, I removed the clutter, the attachment in the schema has the arc avatar to the field.
and I want to delete record.attachment from aws

Post #2

dimitarvp

Hm, I see. In this case I recommend you switch approaches. You’re better off putting a “please delete attachmend with ID 123” command on a message queue (RabbitMQ, Kafka, you can also use your DB as well) and then have a background worker that periodically works through those and just have that delete the files from AWS.

Post #3

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I would probably invert this whole process. Don’t delete the record from the database first. Mark the database record as deleted_at pending_deletion or something, and then enqueue a job in something like Oban which can try to hit AWS to delete the record. Only after that has succeeded do you delete the record in the DB. The flag lets you filter it out from UI lists immediately, but you want the record around to reason about the file until it’s actually gone.

The job part here is critical. AWS S3 has good uptime, but not perfect up time. Requests to delete files will fail periodically and will need to be retried.

Post #4

baldwindavid

If you like Arc, you might take a look at Waffle (I've forked Arc to Waffle and merged all open PRs). Arc seems to have gone unmaintained and Waffle is a fork with fixes. I prefer the transparency and control of working directly with ex_aws_s3 and orchestrating uploads and deletions via Ecto.Multi, but understand the allure of an opinionated package.

Post #5

achempion

Waffle maintainer here.

Deleting by triggers approach is completely new to me. Generally it’s recommended to keep all logic inside the application in order to have all in one place.

If you want to have async deletion operation, you could utilise queue libraries for this.

As you described, you’re performing record deletion by handling prostgres events with triggers. All you need to do is to call elixir callback after record deletion with record attributes which will call file deletion function.

In other words, file deletion will work as any other callback action. I guess your question is not about file deletion but about how to invoke a callback in elixir after prostgres trigger. Am I right?

Post #6

benonymus

Yes, the question is about that in order to delete I would need the attachment that comes from the repo with the through the arc avatar, but as you can see here I only have the string straight from the dB.

Thank you for all the responses, seems like a different approach will work better!

Post #7

achempion

If you have a string it should be enough to delete a file. You don’t need to have any Repo interface to call a delete function.

You can pass the filepath directly to the delete function. Record only need to generate that path automatically for you but you can do it manually as well.

Post #8

benonymus

So if I pass the string that is stored in the files that the arc avatar matches to in my case attachment with the value of test_docume_nt_fEfRY-pR2JX3jIF9CrXPC.pdf?63754075895

If I pass this to the delete function this should do the delete?

Post #9