LostKobrakai

Depot - Filesystem Abstraction

Depot is a filesystem abstraction inspired by the great flysystem PHP package.

This allows interacting with many different filesystems (cloud based or not) via a common sensible and in some parts simplified API. Change e.g. cloud providers just by switching out the filesystem setup, leaving any business logic as is. It supports both “module based” systems as well as systems setup purely at runtime (thanks to @michalmuskala presisting on it ages ago) .

defmodule LocalFileSystem do
  use Depot.Filesystem,
    adapter: Depot.Adapter.Local,
    prefix: "/some/path/to/storage"
end

LocalFileSystem.write("test.txt", "Hello World")
{:ok, "Hello World"} = LocalFileSystem.read("test.txt")

# or

filesystem = Depot.Adapter.Local.configure(prefix: "/some/other/path/to/storage")
Depot.write(filesystem, "test.txt", "Hello World")
{:ok, "Hello World"} = Depot.read(filesystem, "test.txt")

The following parts are done.

Basic API
Stream based API (with the help of @mcrumm)
Portable visibility (:public vs. :private only)
Adapters can support more complex visibility handling
Copy between different filesystems with
- Copy directly on the filesystem (e.g. S3 bucket_01 → bucket_02) if supported.
  Must be same adapter for both filesystems.
- Copy using streaming via the current machine.
- Drop to synchronous read/write when streaming is not supported.

Outstanding additions:

Further common metadata: Size, Type, …

GitHub:
https://github.com/elixir-depot/depot

The abstraction for the different underlying filesystems are done using adapters:

InMemory (Included)
Local (Included)
S3 (Currently not up to date with latest depot behaviour)
Google (Currently not up to date with latest depot behaviour; by @jayjun)
…

9 comments

#library #file #filesystem #depot

21 1843 9

2024-06-06 09:00:11 UTC

First Post!

dimitarvp

Does this help you with your problem with backslashed paths that you shared about some time ago?

Post #1

Most Liked

LostKobrakai

I’ve picked up work on this again and am currently trying to get the API and the S3 adapter finalized. Today I bumped into the reality that minio (what I’m using in dev/test) doesn’t support object level ACLs like AWS S3 does, the docs seem to suggest using bucket policies. I’ve not much experience around those policies, so I’m wondering if anyone here does and would like to help me out or collaborate on the visibility handling part of the adapter. I’d really like for the adapter to support not just AWS.

Post #8

LostKobrakai

It was the reason for me bringing it up. One would think using Path.type to enforce only relative paths would work, but it’s os dependent. I ended up just copying the code and modifying it to fit the needs of the module.

Post #2

hauleth

Out of interest, why use Agent in the InMemory adapter instead of ETS? Recently I have used ETS for such behaviour (I should publish that before you published Depot) with 2 tables:

“paths” index in form of {path(), hash()}
file contents indexed by hash {hash(), content()}

In that way file copies are “cheap” and I do not need to store them twice (in my case I couldn’t rely on Erlang immutability, as I was reading file content from tarballs, so each binary would be “different” from the viewpoint of VM).

I am also preparing the PR with “balckhole” adapter which I often find useful for testing.

BTW

I have reviewed the code and I think it would be useful to make some callbacks optional, as not all adapters will support them, and that would make it easier to implement them for situations when some features aren’t available.

Post #3

Last Post!

pehbehbeh

We also have a PHP/Laravel background and also missed something like Flysystem. We are currently considering building something like this ourselves and came across this library during our research. Are there any current plans to further develop and maintain the package? Is it still alive?

Post #9