Rsync is a quite known utility if we want synchronize data between two nodes or even between two folders in the same system, and of course also used for backup purposes. Its main peculiarity is that it sends only incremental changes; for instance, if you get a connection drop when using classic FTP you typically have to re-send the entire folder or file, but with this tool you re-run it again and it will start from where it left.
However Rsync is not so good if you want speed and it doesn`t have any multithreaded capability; so if you have GBs or even 1 TB of data to transfer it can be quite slow in the end.
To overcome these drawbacks you can use BBCP, which is a point-to-point network utility written by Andy Hanushevsky at SLAC with the goal to get close to the line speed in a LAN or WAN connection.
You don’t need any server listening or any SSH daemon, but you have to make sure bbcp is installed and in the PATH in BOTH systems.
Assuming GIT is installed, below are the steps to install it:
Linux
$ make
The pre-requisites are:
GNU C++ compiler
Zlib library
Pthreads library
Some examples of usage are:
-V verbose output
-P 2 display progress every two seconds
-s 16 create 16 parallel network streams (or threads)
-w sets to 8 MB the size of the disk input/output (I/O) buffers
To transfer a directory just use the -r option (it stands for recursive of course).
To resume files in case of a lost connection add the -a and -k switch.
If a firewall is blocking the communication between source and destination, use the -z option
I have tested BBCP only on Linux, but it should be possible to compile it and install it in all major UNIX systems. The Windows platform is not supported.
Resources:
http://www.slac.stanford.edu/~abh/bbcp/
http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm
https://www.olcf.ornl.gov/kb_articles/transferring-data-with-bbcp/