Any interest in a library that wraps rsync?

Your Libraries & OS Mentoring Pre-release Feedback/Help
, , , ,
1

I’ve written a crude module for retrieving files from a remote server using rsync, since I didn’t see any existing tools already. This could be useful anywhere one needs to efficiently move files across filesystem or network boundaries, and could even be used for better control and monitoring of deployments. Does it sound like something I should package into a library?

The inner logic takes advantage of the --info=progress2 reporting mode for rsync, reading status lines to monitor progress:

port = Port.open(
  {:spawn_executable, System.find_executable("rsync")},
  [args: ["-a", "--info=progress2"] ++ remote_urls ++ [local_path],
    :binary, :exit_status, :hide, :use_stdio, :stderr_to_stdout]
)

receive do
  {^port, {:data, data}} ->
    ....

  {^port, {:exit_status, status}} ->
    ....
end
6 Likes
2

I am not a reliable source but IMO that would be more interesting as a blog post showing how did you wrap rsync.

4 Likes
3

A blog post is a good idea, and I ended up publishing as a library so there’ll be something to write about:

3 Likes
4

Thanks. Had a quick look. IMO you should include a complete example with the error messages that your GenServer can also emit (not only the happy path). I’d be interested in just copy-pasting such an entire block of code in a project of mine and then tuning the separate case branches for my goals. Better peace of mind.

5

I was also curious about error cases, so I added some tests that show reasonable behavior when rsync emits random junk (forward to Logger at :warning level) and when the process crashes (forward error to linked parent).
Deleted: random claims that the tests are a full example.

Feel free to share the case statements you add for your own application goals.

6

I struggle with basic OTP principles. Is the normal way to handle a start_link failure to run it under a Supervisor? Anyone feel like sharing a snippet showing how to run a brittle command like this and catch the error?

In my own app I’m letting Oban handle the retry logic and in tests I’ve trapped the exit but these are side-stepping typical usage I suspect.

7

Thanks for the suggestion, here are my observations as a blog post: Elixir/Ports and external process wiring - ludd

1 Like
8

Very informative post, Adam. I am using ports to communicate with long-running OS processes and struggle to understand the niceties about how to signal them from the BEAM. I did not know about erl_child_setup but I see that it is used by two other OS process-oriented libraries: alcove and prx.

9

I’d be curious to hear more about the obstacles that you’ve run into. If you mean “signal” in the generic sense of sending data back and forth, there are certainly some complexities but it should be possible to accomplish whatever you need, just set up a gen_server to write and read to and from the process.

But if you mean “signal” in the POSIX kill sense I think it’s impossible to do with the built-in functions, since the OS PID mapping is strictly internal and can’t be accessed from code. If this is the intention, you could write a wrapper in C or shell, which listens for messages from BEAM and translates those into signals to send to child. But I wait to hear more!

1 Like
10

since the OS PID mapping is strictly internal and can’t be accessed from code.

BEAM does return OS PID: erlang — erts v16.1.1

iex> port = Port.open({:spawn_executable, "/usr/bin/cat"}, [])
#Port<0.11>

iex> {:os_pid, pid} = Port.info(port, :os_pid)
{:os_pid, 367095}

iex> {output, 0} = System.cmd("ps", [to_string(pid)], into: IO.stream())
    PID TTY      STAT   TIME COMMAND
 367095 ?        Ss     0:00 /usr/bin/cat
2 Likes
11

@akash-akya thanks for pointing this out! I updated the linked blog with your correction but unfortunately can’t edit my post above, so the misinformation will linger…

The other shortcoming of the “kill” approach which I should mention here is that it’s hard to include as an automatic clean-up, for example if the controlling BEAM process tree is shut down. That’s what lead me to push the signaling down into erl_child_setup or the analogous C shim program included in the rsync library.