Migrating data from one storage platform to another can be slow and tedious, if you just plug it into the network and hope for the best.
For example, my customer’s network runs at 10/100 Mbps. So, when we plugged in the new DS1812+ to the network we achieved an underwhelming 7MB per second on average. This would have taken over 4 days to transfer 4TB.
To see what is going on, first look at the network speed. A 100Mbps network has about 10% overhead just to send packet headers. After converting BITS to BYTES, you can achieve about 11MB per second. The files we are transferring vary in size, and every start and stop of a file transfer creates a pause in data being transferred, which reduces the resulting transfer rates.
Additionally, the tool you use to transfer files has overhead. In the default cause of using rsync, it uses SSH to transfer files, which means every file has to be encrypted and decrypted. This also reduces the transfer rate because now the processor on both ends needs to do additional work.
To speed things up, we performed a few tricks. First we directly connected the server from a secondary NIC to the DS1812+ secondary NIC, which allowed both devices to connect at 1Gbps speed, which translates in a theoretical max of 100+MB per second. Then we configured both the Synology and DS1812+ to use an Maximum Transmission Unity (MTU) of 9000 BYTES. Then we told rsync to use the rsync daemon (rsyncd) instead of SSH so we could avoid the encryption overhead. Since these devices are directly connected in our datacenter there is no real risk of exposure of data.
Here is how to set the MTU on an OSX server:
sudo networksetup -setMTU en1 9000
The resulting transfer rates now peak out at 50+MB per second. Again, there is overhead for rsync to start transferring each file, and if the file is less than 50MB, the whole file will transfer in less than a second so you will see a lot of transfers being 10MB per second, or 30MB per second, etc, depending on file size.
Here is the command we used:
rsync –recursive –copy-links –times –verbose –progress /Volumes/source rsync://firstname.lastname@example.org/target