ijunaidfarooq
The task is to:
- SSH to a Linux server.
- Stop a service.
- Using tar create a tar.gz file.
from a directory (30GB size Max). - Upload that file to an FTP server.
- Start the stopped service.
- Whole above process is going to be done on Each Sunday
I have covered pretty much all steps and failures in the code, assuming that there is going to be a service running and also directory is present as well on which it’s going to do tar operation.
I am using GitHub - rubencaro/sshex: Simple SSH helpers for Elixir. SSH is useful, but we all love SSHEx ! · GitHub for SSH to the server and then do all the commands.
Questions:
- How can I make the pipe to wait until the tar process completed and then run the next command? as on 30GB file, It almost takes more than 10 minutes to tar that size of the directory. Task 3 and Task 4 are the most time consuming, And I want to make pipe wait until these commands are done then proceed to next.
- Is this the best way to do
taron a big directory (32GB max). - Is it the best way to do FTP? in linux?
- Is it the best solution to do that? I mean Elixir is great but is doing all these steps through an Elixir Task is good?
- Can you suggest me some notes and changes to the code, I have written to make it better?
UPDATE:
While doing tar I get this error from ssh connection
{:error, "Timeout. Did not receive data for 5000ms."}
All Module:
defmodule EvercamMedia.BackUpSeaweedfsFiler do
require Logger
@zip_date Calendar.Strftime.strftime(Calendar.DateTime.now_utc, "%Y%m%d") |> elem(1)
@ftp_domain Application.get_env(:evercam_media, :ftp_domain)
@ftp_username Application.get_env(:evercam_media, :ftp_username)
@ftp_password Application.get_env(:evercam_media, :ftp_password)
@filer_server Application.get_env(:evercam_media, :filer_server)
@filer_username Application.get_env(:evercam_media, :filer_username)
@filer_password Application.get_env(:evercam_media, :filer_password)
@stop_seaweedfs_command "systemctl stop seaweedfs.service"
@start_seaweedfs_command "systemctl start seaweedfs.service"
@zip_filer_directory "tar -czf #{@zip_date}.tar.gz /storage/filer"
@copy_to_ftp "curl -T #{@zip_date}.tar.gz #{@ftp_domain} --user #{@ftp_username}:#{@ftp_password}"
def start do
connect_to_server(@filer_server, @filer_username, @filer_password)
|> connected?
|> grant_for_backup()
end
defp grant_for_backup({:exit, true}), do: :noop
defp grant_for_backup(connected) do
connected
|> run_command_on_server(@stop_seaweedfs_command)
|> response_is(connected)
|> run_command_on_server(@zip_filer_directory)
|> response_is(connected)
|> run_command_on_server(@copy_to_ftp)
|> response_is(connected)
|> run_command_on_server(@start_seaweedfs_command)
|> completed?
end
defp connected?({:ok, connected}), do: connected
defp connected?({:halt, true, message}) do
Logger.info("Process stopped due to: " <> message)
send_email_for_backup("Process stopped due to: " <> message)
{:exit, true}
end
defp completed?({:halt, reason}) do
Logger.info("Process stopped due to: " <> reason)
send_email_for_backup("Process stopped due to: " <> reason)
end
defp completed?(_conn) do
Logger.info "Process has been completed."
send_email_for_backup("Process has been completed.")
end
defp run_command_on_server({:halt, reason}, _command) do
Logger.info("Process stopped due to: " <> reason)
send_email_for_backup("Command Couldn't run due to: " <> reason)
:noop
end
defp run_command_on_server(connected, command) do
connected
|> SSHEx.run(command)
end
defp response_is({:ok, _res, _}, connected), do: connected
defp response_is({:error, reason}, _connected), do: {:halt, reason}
defp connect_to_server(ip, username, password, tries \\ 1)
defp connect_to_server(_ip, _username, _password, 3), do: {:halt, true, "Not possible to make connection with server."}
defp connect_to_server(ip, username, password, tries) do
SSHEx.connect(ip: ip, user: username, password: password)
|> case do
{:error, :nxdomain} -> {:halt, true, "IP does'nt seem correct."}
{:error, :timeout} -> connect_to_server(ip, username, password, tries + 1)
{:error, reason} -> {:halt, true, reason}
{:ok, connected} -> {:ok, connected}
end
end
defp send_email_for_backup(message) do
EvercamMedia.UserMailer.seaweedfs_filer_backup(message)
end
end
Thanks in Advance.
Trending in Questions
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
Hi everyone.
My team and I have been working on a fairly modest app based around video streaming and chat, but we’ve landed a customer t...
New
Hello,
I’m trying to build a basic Phoenix web-app, and I’d like to use Tailwind.
However, when I launch mix phx.server, I get an error...
New
I really like the adapter patterns that ecto, nebulex, waffle, etc. use and would love find something similar for a key management servic...
New
Hello folks!
So at work, we are seeing some situations where we have to define some “fixed” strings that are used across the codebase in...
New
How Can I Optimise Compile Time Dependencies
I have been building an elixir application for about 2 years now. Many modules and files ha...
New
Other Trending Topics
A little off-topic, but I feel like people here have a good head on their shoulders.
I used to be quite good at making software. Was luc...
New
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
I love Elixir. It’s one of 2 programming languages I’ve ever fallen in love with.
But I don’t use it anymore.
Serverless was the promis...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #ai
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #metaprogramming










Showing Posts 1 to 3- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
axelson
For that specific error I think you may want to look into SSH keepalive. That will ensure that something is always being sent down the pipe so that the connection doesn’t die.
Although for me personally I would probably just run a bash script on the remote server to do the heavy lifting.
NobbZ
I’d even do this per Cron on the remote. Without any tools connecting from another computer.
OvermindDL1
I’d probably use a shell script too, it’s just too simple to do it, but if you want to use the BEAM VM then a few things:
Uhh, if you are uploading to the same server that you are SSH’d to, then why not use the SSH file transfer service to send the file over that instead?
Ping it every once in a while, or set a keepalive or so, and make sure to set the timeout to way high enough considering you are dealing with a giant file.
Not really, rsync would be better, though if the files are at all compressable then just zipping them or tar/gzing them would be far better.
FTP has SO many issues everywhere as well as no real security whatsoever that I would never ever ever recommend it, plus if you already have an SSH connection then just use the file transfer service on it.
Elixir is fine, but honestly I’d just use a shell script as it would be just a couple of lines long is all (and much easier rsync work).