How to make sure a cleanup task will always run

Maybe You could use Briefly[ briefly | Hex ], the file will live only for the time of the process.

Definitely Briefly. Never had a problem with it, used it at least in 3 commercial projects. It just does what it says and gets out of the way.

Thanks for the suggestion, I’ll give it a try.

By the way, I love this forum, always nice people giving good advice !

I never used Briefly / was unaware of its existence. If you truly care about the cleaning up eventually (and timely) happening, I’d consider having an out-of-band recurring “task” that cleans up orphaned files, e.g. as an Oban cron job (optional, but gives you visibility into past executions).

From reading the Briefly docs, my takeaway is that it aligns the lifetime of a temp dir with the one of a Process. You can likely achieve the same thing without the external dependency if you care about that. It also gives you a way to “clean up all”, which is probably not what you want in the system you described, but you could build your own “clean up all stale temp dirs” code.

I switched to Briefly. Even though I’m not sure what is the extent of a process in my context (is it a request handling ie. the whole plug pipeline, is it a function end?), it seems to work super well.
After 3 days in pre-production, (handling 3 reqs/s), not one temporary directory is left behind. All cleaned up nicely, my problem is solved

Thanks for the idea of Oban, I might use it if I need other jobs scheduling for this app.

Indeed in normal circumstances you should not see any problem. The same would be true with explicitly creating and deleting files (i.e. no errors/process crashes in between).

You can simulate a hard crash (which could be caused by a deployment, NIF or other conditions) with kill -9 BEAM_PID. Then, depending on your deployment strategy, your app comes back (e.g. restarted by systemd or Docker) and you have a handful of directories left over (because it is impossible to guarantee a certain line of code will always execute, think a hard powerloss, no programming language/environment can give that guarantee).

Maybe you’ll be fine ignoring that, maybe you account for it. Other than a scheduled job, you can also clean up on startup.

The idea of having visibility on when and how often those things are happening would also help you operate the system in the long-run. Again, may not be relevant in your particular case.

Those decisions are on you depending on your system and goals.

Wish you success!

Just for future reference, I usually use an extra process which monitors my “main” process, the one which does the work. In your case it would be the request/response pipeline. You can spawn a process and when it receives a down message, it will clean the temporary resources.

Some people are scared of spawning processes but processes are great exactly for managing lifecycle and concurrency.

A quick feedback.

The reason why Briefly was so efficient was because of me calling Briefly.cleanup/0 as a final step of my plug pipeline. I thoughts there was some kind of scope for Briefly to only remove the directories related to the current plug, but it was naive of me Briefly.cleanup removes every temporary directories. Period.
So, this was not that great for my service
The solution I now deployed in pre-production is based on a GenServer running a periodic cleanup. Oban seemed oversized for this.

This was the first GenServer I wrote, and it seems to work great.