Data Sharing Workflow for Large Datasets (>1TB) with Globus

Abstract

In the Deep Blue Data (DBD) repository, built on Fedora/Samvera, at the University of Michigan, we have datasets ranging in size from KBs to TBs. Like many institutions, we are struggling with an uptick in file sizes (total data stored in DBD will increase from 2.94TB in October to over 16TB by the end of December). Until recently we were unable to download files larger than 20GB from DBD, which significantly hampered our ability to serve the researchers in our university. However, as of early November, we have started testing the integration of Globus (https://www.globus.org/) with the Deep Blue Data interface as a means of downloading these larger files. Our poster will illustrate the workflow we have created to ingest these large datasets as well as the process of integrating Globus with our instance of Fedora/Samvera. We will be using an actual case study of the workflow from researcher network storage to final Globus download endpoint. The code for this Globus integration is available on GitHub.