azcrumpty's Site
  • Home
  • Journal
  • Search

SSH Remote Server to Server File Transfer

5/13/2011

 
Picture
SSH Server to Server File Transfer uses the Path in Red
Many people do site to site file transfers the hard way--copying and replicating data from one host to another.  The Secure Shell works very well with pipes and creating a pipeline over the network is a nice way to go. Using software pipelines will make server to server file replication much easier. We will examine copying from two servers tied together with a jump server.

You already know you can pipe output in *NIX systems:

ls -lsa | sort -n        # sort by block size

You can also start a command on one server and pipe the output to a remote server.  You can use this feature to tie two remote servers together using a common jump server.

Login and execute from the jump server:

ssh EastCoastServer "(cd /home/azcrumpty; tar -zcf - .)" | ssh WestCoastServer "(cd /home/azcrumpty; tar -zxvf -)"

This will copy my home directory specified as the dot from the east coast to the west coast.

The secret to success is creating a process pipeline across the network using SSH. The command above tells the EastCoastServer to create a subprocess within the parentheses. That process changes to my home directory and makes a compressed tarball of my home directory to standard out (STDOUT).  The output is piped to another remote SSH session that changes to the home directory and untars the compressed archive.  If you have an older *NIX where compression isn't built in to tar, add your favorite compressor (gzip/gunzip or compress/uncompress) to the pipeline.

Picture
SSH Site To Site Home Directory Replication

Execute Copy Command:
ssh EastCoastServer "(cd /home/azcrumpty; tar -cf - . | gzip -.)" | ssh WestCoastServer "(cd /home/azcrumpty; gunzip - | tar -xvf -)"

You can do a lot with pipes as well.  I could run ssh to sort output from east to west coast.



Execute List and Sort Command:
ssh EastCoastServer "ls -lsa" | ssh WestCoastServer "sort -n > east.manifest"

The diagram above shows the data is transferred along the red lines.  This diagram assumes you have firewalls between the servers and a common jump point. If not use the secure shell copy tool to move the data.

The benefits of this site to site copy method is the removal of the need to  replicate data on jump server disk risking out of space issues, because the pipe eliminates the additional step of using another disk for the data.

If you transfer large amounts of data that could take several hours, your copy operation could abort due to session termination so use a tool with persistence such as VNC or GNU Screen.

Enter your passwords as prompted.  This is much easier when you have installed trust relationships from the jump server to at least one node as you would only be prompted for one password instead of two.

Picture
SSH Server Execution Output Piped to Remote Host

    Journal

    This is the place for notes and updates.

    azcrumptyon

    Archives

    March 2013
    December 2012
    October 2012
    September 2012
    August 2012
    July 2012
    April 2012
    March 2012
    November 2011
    October 2011
    July 2011
    June 2011
    May 2011
    April 2011

    Categories

    All
    6to4
    Academic Degree
    Anonymous
    Apache
    Apache Http Server
    Automobile
    Blue
    Bridge
    Car
    Cars
    Chrome
    Chrome Os
    Chromium
    Common Gateway Interface
    Compress
    Configure
    Copy
    Dd Wrt
    DD-WRT
    Diesel
    Diesel Engine
    Directories
    Disklabel
    Disk Management
    Disks
    Distance
    Distance Education
    Distance Learning
    Dmz
    Dns
    Driving
    Eco
    Education
    Efficiency
    E-learning
    File
    Filtering
    Firewall
    Freebsd
    Fuel
    Fuel Economy In Automobiles
    Fuel Efficiency
    Green
    Gzip
    Hardware
    Hardware Virtualization
    Hitwise
    Home
    Hosting
    Howto
    Hub
    Hurricane Electric
    Hypertext Transfer Protocol
    Ibm
    Ibm Websphere
    Iis
    Importer
    Internet Service Provider
    Ip
    Ipad
    Ip Address
    Ipv4
    Ipv6
    Lan
    Lastpass
    Learning
    Linux
    Local Area Network
    Master Of Business Administration
    Microsoft
    Microsoft Windows
    Movabletype
    Movable Type
    Multitier Architecture
    Mysql
    Nat
    Network
    Network Interface Controller
    Network Time Protocol
    New York
    New York Times
    Onion
    Online
    Openbsd
    Openvpn
    Operating Systems
    Opera Unite
    Partitions
    Passenger
    Password Management
    Passwordsafe
    Paywall
    Pfsense
    Protocols
    Proxy
    Proxying And Filtering
    Publishing
    Rdns
    Reading
    Red
    Remote
    Replicate
    Reverse
    Reverse Proxy
    Router
    Samba
    Secure Shell
    Security
    Server
    Server Message Block
    Ssh
    Storage
    Tdi
    The Onion Router
    Theory And Research
    Tor
    Torque
    Transfer
    Turbo
    Turbodiesel
    Twitter
    Ubuntu
    Uniform Resource Locator
    United States
    Universal Serial Bus
    Unix
    Usb Flash Drive
    User (computing)
    Virtualbox
    Virtual Hosting
    Virtualized Dmz
    Virtual Machine
    Virtual Private Network
    Vpn
    Wan
    Web
    Web Server
    Windows
    Windows Xp
    Wordpress
    Zone

    RSS Feed

Powered by Create your own unique website with customizable templates.