SSH Server to Server File Transfer uses the Path in Red
Many people do site to site file transfers the hard way--copying and replicating data from one host to another. The Secure Shell works very well with pipes and creating a pipeline over the network is a nice way to go. Using software pipelines will make server to server file replication much easier. We will examine copying from two servers tied together with a jump server.
You already know you can pipe output in *NIX systems:
ls -lsa | sort -n # sort by block size
You can also start a command on one server and pipe the output to a remote server. You can use this feature to tie two remote servers together using a common jump server.
Login and execute from the jump server:
ssh EastCoastServer "(cd /home/azcrumpty; tar -zcf - .)" | ssh WestCoastServer "(cd /home/azcrumpty; tar -zxvf -)"
This will copy my home directory specified as the dot from the east coast to the west coast.
The secret to success is creating a process pipeline across the network using SSH. The command above tells the EastCoastServer to create a subprocess within the parentheses. That process changes to my home directory and makes a compressed tarball of my home directory to standard out (STDOUT). The output is piped to another remote SSH session that changes to the home directory and untars the compressed archive. If you have an older *NIX where compression isn't built in to tar, add your favorite compressor (gzip/gunzip or compress/uncompress) to the pipeline.
You already know you can pipe output in *NIX systems:
ls -lsa | sort -n # sort by block size
You can also start a command on one server and pipe the output to a remote server. You can use this feature to tie two remote servers together using a common jump server.
Login and execute from the jump server:
ssh EastCoastServer "(cd /home/azcrumpty; tar -zcf - .)" | ssh WestCoastServer "(cd /home/azcrumpty; tar -zxvf -)"
This will copy my home directory specified as the dot from the east coast to the west coast.
The secret to success is creating a process pipeline across the network using SSH. The command above tells the EastCoastServer to create a subprocess within the parentheses. That process changes to my home directory and makes a compressed tarball of my home directory to standard out (STDOUT). The output is piped to another remote SSH session that changes to the home directory and untars the compressed archive. If you have an older *NIX where compression isn't built in to tar, add your favorite compressor (gzip/gunzip or compress/uncompress) to the pipeline.
SSH Site To Site Home Directory Replication
Execute Copy Command:
ssh EastCoastServer "(cd /home/azcrumpty; tar -cf - . | gzip -.)" | ssh WestCoastServer "(cd /home/azcrumpty; gunzip - | tar -xvf -)"
You can do a lot with pipes as well. I could run ssh to sort output from east to west coast.
Execute List and Sort Command:
ssh EastCoastServer "ls -lsa" | ssh WestCoastServer "sort -n > east.manifest"
The diagram above shows the data is transferred along the red lines. This diagram assumes you have firewalls between the servers and a common jump point. If not use the secure shell copy tool to move the data.
The benefits of this site to site copy method is the removal of the need to replicate data on jump server disk risking out of space issues, because the pipe eliminates the additional step of using another disk for the data.
If you transfer large amounts of data that could take several hours, your copy operation could abort due to session termination so use a tool with persistence such as VNC or GNU Screen.
Enter your passwords as prompted. This is much easier when you have installed trust relationships from the jump server to at least one node as you would only be prompted for one password instead of two.
SSH Server Execution Output Piped to Remote Host
RSS Feed
