Is there a GenResourceManager?

Questions / Help
1

I have a use case where I process a bunch of zip archives and was looking for a way to:

  1. Unpack the archive to temporary directory
  2. Process it with workers
  3. Clean it up after use

The first approach was to just wrap all the processing in try do...after and clean it there. But then I looked at this thread on how File.open works and how the file using process is monitored and when it crashes, a “file manager” will close the resource.

I thought this is the OTP way to approach the problem and started to write a GenServer to manage these archives and remove their temporary directories after no processes are using them anymore - this way I can even run many workers in parallel, and the temp dir will be removed after the last exits - great!

But then I thought - this is such a generic case, there is (or should be) a gen_resource_manager to just use and define open(resource_id) and close(resource_id).

Googled a bit but did not find anything, so decided to ask here: do you know about something like this?

2

Some time ago I went down the burrow trying to figure out how live upload files automatically disappeared. If you take a look at upload.ex in the plug library I think you will see some of the patterns you are asking about. The handle_info({:DOWN... does clean up when the GenServer dies. The module docs for Plug.Upload state:

A server (a GenServer specifically) that manages uploaded files.

Uploaded files are stored in a temporary directory
and removed from that directory after the process that
requested the file dies.

During the request, files are represented with
a Plug.Upload struct that contains three fields:

* `:path` - the path to the uploaded file on the filesystem
* `:content_type` - the content type of the uploaded file
* `:filename` - the filename of the uploaded file given in the request

Note: as mentioned in the documentation for Plug.Parsers, the :plug
application has to be started in order to upload files and use the
Plug.Upload module.

Security

The :content_type and :filename fields in the Plug.Upload struct are
client-controlled. These values should be validated, via file content
inspection or similar, before being trusted.

Edit: PS - if the zip files aren’t very big you could do the whole thing in memory. The zstream library, for example, looks like it would help with this.

1 Like
3

I’d checkout briefly v0.5.0 — Documentation (disclosure, I helped write it). Basically you do:

{:ok, dir_path} = Briefly.create(directory: true)

and it creates a path in the system temporary files area bound to your current process. If you give it the directory: true it makes a directory, if you do a simple Briefly.create you get a file path. When your process dies, Briefly cleans up after you!

As @mindok notes Plug.Upload basically does this for temporary plug uploads and in fact Briefly borrows heavily from Plug internals.

4 Likes
4

Nice!

Wouldn’t it be cool if Briefly could:

  1. Handle any kind of resource. eg.:
defmodule BucketManager do 
   use Briefly

   @impl true
   def acquire(meta)  do 
     AWS.S3.create_bucket meta[:name]
   end

   @impl true
   def release(bucket) do 
     AWS.S3.delete_bucket bucket.name
   end
end

and be able to do something like:

{:ok, bucket} = BucketManager.open(%{name: "abcdef"})
  1. Allow reference counting so a resource can have many users-processes, when a resource is already created, it will just be returned, and released only after the last user-process exits!