The erlang ftp module does not allow you to set the chunksize (or at least I couldn’t find it in their docs), so I assume it’s sftp
Awlexus
linusdm
Just out of curiosity… I’m building a feature where a Phoenix application is creating files that need to end up on a remote filesystem with low latency, through sFTP. I want to retain flexibility, and avoid coupling the application to the specifics of the transfer protocol used. rclone seems like a really nice tool for that job, offloading the responsibility to copy files to whatever destination. I’m not sure how you’d use rclone in conjunction with a regular Phoenix application. Do you run it in response to some event that triggers the file generation? Or does it run in the background, managed by an elixir process and supervisor, or maybe even outside the Phoenix application? One of my concerns would be what happens when multiple application nodes are pointing to the same destination (but maybe there is no problem whatsoever).
Thanks for referencing rclone as an alternative!
0
dimitarvp
No clear-and-cut answer but I would very likely opt for having N background rclone processes waiting (a pool for Oban workers for example) and then enforce uniqueness via Oban’s mechanism so e.g. you can’t write to the same file from 2 or more workers. That’s trivially achievable.
The tricky part is of course if somebody wants the file upload to be synchronous but IIRC Oban offers that as well? (I think you can send a job and wait on it?)
BTW when it comes to sFTP in particular I think Erlang is slightly better equipped but don’t quote me on this, I only used a library for it once and it was a few years ago. I’d still try work with a normal library and only reach for an external tool if things don’t work stably – as it happened with OP.
But you also said you don’t want to tie your application’s logic to the transport protocol. Hm, in that case I’d probably choose ex_aws_s3 because you can still use it not only for Amazon S3 like Cloudflare’s B2 but also for self-hosted software like MinIO.
1
D4no0
Yeah sftp erlang client works pretty much flawlessly, the standard is sound and almost all servers use pretty much the same version (v3 sftp, v2 ssh if I remember well), so I would recommend to use that instead of calling an external process, because there is always the risk in a long-running system that you will have some kind process leaks at some point.
2