Cloud Storage Management¶
Backing up data from Monsoon to Google Drive is easy. This approach can be used with rclone for Dropbox, and OneDrive (untested).
While we can help you get started with this approach, we cannot support any issues that you have with your data in the cloud storage. You are responsible for the management and retrieval of data from your cloud storage.
- We will utilize the rclone utility which is installed on Monsoon as a module:
module load rclone - In this doc we'll be focusing on using Google Drive, which has up to 100GB of storage available.
Pre steps¶
First, sign up for unlimited Google Drive (one time setup) by selecting Faculty/Staff Google Account Request:
Establish a directory structure in your Google Drive via the web interface. For this doc we will use backup as the name of the directory that we will push our data to, from Monsoon. The backup folder is in the root of your Google Drive (one time setup).
Initialize your Google Drive for use with rclone (one time setup):
module load rclone
rclone config
# press `n`, for new
# enter name (you will use this name to refer to your drive all the time)
# enter `6` (for Google Drive) `13` (for Google Drive)
# leave client id blank
# leave password blank
# press `n` for advanced config
# press `n` for no, working without a gui
# copy the url, paste into your client browser
# copy and paste auth code from browser into Monsoon
# `y` (yes this is ok) `n` (for a person Drive) `y` (for a Shared/Team Drive)
# `q` (to quit)
- Create the backup directory in your Google Drive via web interface, or rclone as so:
Google Drive is now ready to use from Monsoon with the rclone utility. Read the rclone documentation, or follow along for a few common scenarios.
Data sets¶
Upload large data set, or many small files¶
Break the directory you would like to backup into manageable pieces. This also makes transferring to "the cloud" a "breeze" as small files won't inhibit the speed of the transfer.
screen # (optional, but will save you if you get disconnected)
tar cf – /projects/comp_genomics | split -d ––bytes=200GB – comp_genomics.tar.
comp_genomics.tar.00, comp_genomics.tar.01 … comp_genomics.tar.NN
Use rclone to copy a tar file to your drive's backup folder, here the drive is named nau_drive
screen # keep a bash session alive when you disconnect
rclone
module load rclone
rclone copy comp_genomics.tar.00 nau_drive:backup`
# or:
for i in $(ls comp_genomics.tar.*); do
rclone copy $i nau_drive:backup
done
Upload small data set, or large data set with few small files¶
Use rclone to sync your data set to Google Drive.
Warning
This will overwrite/change the destination to be identical to the source directory
screen # keep a bash session alive when you disconnect
module load rclone
rclone -c sync ––transfers 32 ––checkers 16 mydatadir/ nau_drive:backup/mydatadir/
rclone will start 32 file transfers in parallel. Only two files per second will actually be accepted by Google however due to their rate limiting. Howver, there can be many more files in transit. This method will prove to be the best as you will achieve the highest combined throughput.
rclone will indicate success or failure for the various copy and sync command. To see what has been uploaded to your remote drive, you can easily check what files and directories are there by running an rclone list command:
Pull down a data set from Google Drive¶
Use rclone to pull down a data set from your Google Drive to storage on Monsoon:
screen # keep a bash session alive when you disconnect
rclone copy nau_drive:cluster/Meetings ./Meetings
In this example, rclone will put the Meetings folder in your current directory into a folder called Meetings
Optionally encrypt your files before sending to Google Drive¶
Warning
This will duplicate the data locally, so only do this if you have to.