Big Data Transfer Using Globus Online
One of the major problems in dealing with large data is to transfer files via network. Mass spec files files are fairly large, often exceeds gigabytes. Computation of mass spec results may take some time, but file transfer can take longer than computation in some cases. If a mass spec file is transferred simultaneously, or as soon as mass spec run is over, total amount of transfer time is similar to the mass spec run time (I am assuming that your network connection is faster than generating mass spec raw data). For a few file transfers, SCP/FTP is ok, but large file transfer is better with Globus online.
What this site does is to transfer files (especially large ones) across the internet fast, and reliably. You can also share files with multiple users. Operation is as simple as Dropbox.
First, you create an account for yourself at Globus online by clicking “sign up now”. After verifying your account with email, you can immediately start using their service. There are things you can do it for free such as:
1) File transfer and synchronization to/from servers
2) Create private and public endpoints
3) Access to shared endpoints created by others
Things that are NOT free ($7/mo or $70/yr):
1) Peer-to-peer transfer and share files
2) Create and manage shared endpoints
As far as I understand, if you have university server which has already signed up for Globus, you can create an endpoint on your computer and start transferring files between the registered server and your PC without any charge.
These are the steps to use Globus online for file transfer.
1) Sign up for a new account
2) Log on to Globus online, and go to “Manage Data” on the top of the page, and select “manage endpoints”.
3) Click “add Globus Connect”.
4) Type endpoint name (e.g. myPC) and click “generate setup key”.
5) Select the operating system you are using (Mac, Linux or Windows).
6) Install globus online on your computer (automatic).
7) Open Globus Connect application.
8) Copy the setup key and paste into the box in Globus Connect app. Hit OK.
9) Click “start transfer” under “Manage Data” tab on the top.
10) Type your endopoint name (account_name#computer_name) in the Endopoint box (either left or right). Then click “Go”.
11) Type the server endopoint name in the other Endopoint box. Then click “Go”.
Once you see file structures for both sides, you can start transferring files. Select multiple files by pressing CTRL key, then hit the arrow head to send the files.
Once you started, you can see the job in the transfer activity page. Real advantage of transferring files using Globus online is
Not only you have faster transfer over the internet, you don’t have to log in to the computer you want to transfer files from. This means, you have three computers (A, B & C), and you can direct transferring files from A to B or vice versa using Computer C.
You can also quickly synchronize the directory by clicking an option and check “only transfer new or changed files”. You can further select an option how you want to define new or changed files.
I also put a link to Globus online manual here.
Note: I had a trouble with sending files from my computer initially and found that my computer was in the part of University Hospital network which restricts ports used by Globus online. Since it was not possible to ask network admin to open a port for Globus online, I had to use VPN to mitigate the issue.
Overall, I am pretty satisfied with the speed of transfer. Globus online is ~5x faster than SCP.