Guide to transferring files from Deep Blue Data with Globus
What is Globus and why do I need it to access files on Deep Blue Data?
Globus is a nonprofit platform created by the University of Chicago and Argonne National Laboratory that enables the simple transfer of digital files as large as petabytes (a petabyte is 1,000 terabytes or 1,000,000 gigabytes) from established endpoints, one of which can be your work or personal computer. Globus also offers additional services related to sharing data with other researchers or parties directly.
The datasets in Deep Blue Data's repository hold files of varying sizes. Many can be downloaded directly from the Work, or dataset, that contains them, but some files are so large that timeouts and errors can occur when trying to retrieve them through a browser.
To enable users to access these larger files, the University of Michigan Library and the Deep Blue Data team have set up an alternative taking advantage of Globus's powerful data transferring capabilities. For those who use Deep Blue Data frequently, Globus also may end up serving as a more convenient way to access files of all types and sizes.
How do I set up Globus so I can transfer files to my computer?
Here is a short guide for setting up Globus on your computer. Globus also has detailed "How To" walkthroughs for basic and more complicated setup processes, which are linked to below.
- Set up an account on Globus
- Install Globus Connect Personal
The first step is to set up an account with the Globus Web App, which can be connected to other personal and institutional accounts, including those at University of Michigan. To begin setting up an account, navigate to https://globus.org/login.
You can find Globus's walkthrough for setting up an account at the following link: https://docs.globus.org/how-to/get-started/.
Once you have an account set up, you will want to establish a personal endpoint on your computer so that you can transfer data to it. To do this, you will need to download and install the program Globus Connect Personal, which will connect to the Globus Web App.
To begin, navigate to https://www.globus.org/app/endpoints/create-gcp. First, provide a descriptive name for the endpoint that will help you identify it in the future (e.g. Sam's Personal Laptop). Next, click "Generate Setup Key," copy the result, and keep it in your clipboard or paste it in another document. Then, select your operating system from those listed in "Step 2," and complete the installation setup. Most default settings are satisfactory, though later you may want to change the folder, or directory, to which Globus sends transferred files. During the installation process or when you first open the program (depending on your operating system), you will be asked to provide the Setup Key you copied earlier.
You can find specific and detailed installation instructions for Mac, Windows and Linux operating systems at https://www.globus.org/globus-connect-personal. On the instructions for Mac and Windows, the Configuration section will tell you how to change the folder that Globus can access.
Now I have Globus set up. How do I get the files I want from Deep Blue Data?
Once you have an account and Global Connect Personal is installed on your computer, you have everything in place to begin transferring files from Deep Blue Data. Follow the steps below to complete the transfer process. In many cases, you will need to tell Deep Blue Data to copy the files you would like to transfer to its endpoint. However, it is possible that the files will already be available for transfer because they were previously requested by another user. If in Step 2 you do not see the "Download All for Globus" button, skip to Step 3.
- Return to the Work in Deep Blue Data from which you would like to transfer files to your computer.
- Just beneath the list of files and the "Download All Files" button, there will be another button labeled "Download All for Globus." Click the button to begin the copying process. How long this will take depends on the size of the files.
- On the right side of the screen, click on the empty "Endpoint" field. In the pop-up window, type the name that you assigned to the personal endpoint when setting up Globus Connect Personal and click on it when it appears, or find the endpoint under the "Administered by Me" tab
- When both endpoints are set, you will see the arrow button in the middle of the screen that is pointing toward your personal endpoint turn blue. To transfer the files, click on the arrow. A message should appear indicating that the task has been submitted to Globus.
If you are not a University of Michigan affiliate and are not logged in to Deep Blue Data, a page will appear asking if you would like to provide your email address so that you can be notified when the files are available for transfer. That window will look like this:
If you would rather not provide your email, you can simply return to the page on Deep Blue Data later to see if the files are available. If you are logged in to Deep Blue Data, you will automatically receive an email to inform you when the files are ready for transfer.
On the left side of your screen, you should see the list of files from the Work on Deep Blue Data. Click on the "select all" button at the top-right of the box if you would like to transfer all files, or use your cursor while holding down the Control button on a PC or the Command button on a Mac to select only those that you want.
Because Globus is expected to handle large files that take some time to download, the platform is set up to pause whenever your computer is no longer connected to the Internet. It will resume automatically when your computer is reconnected. Once the files have been completely downloaded, they will appear in the destination folder on your computer. You can always check the status of tasks by clicking "Activity" next to "Transfer Files" on the page where you set up your transfer. You will also receive an email when the transfer has been completed.
Where can I go if I need further help with Globus or Deep Blue Data?
Globus has numerous How To guides on its Globus Docs website, including videos, which can be found at https://docs.globus.org/how-to/. You can also contact the Research Data Services team by emailing firstname.lastname@example.org.
A Note about Globus
A Note about Globus: By default, Globus Personal Connect will automatically turn on in the background when you start your computer. When you you are not using Globus, to avoid receiving error messages and other notifications, simply find the icon in the bottom right-hand corner of your screen, (or in the menu/status bar at the top for Mac) right-click (or ctrl-click for Mac) and click "Quit Globus Personal Connect."