Transfer data between AWS and CyVerse

Page under construction

'The most basic way of moving data from Amazon Simple Storage Service (S3) to the CyVerse Data Store is to download the data to your local computer then upload to CyVerse. However, if you have many files or large files to move, this is not practical. For large transfers, you have two options: Cyberduck (easier) and scripted transfer using Atmosphere (more flexible).

Transfer using CyberDuck

1. Open CyberDuck on your computer or on an Atmosphere instance.
2. Establish a connection to the CyVerse Data Store
3. Establish a connection to S3
   1. Click on the Open new connection button (globe with plus sign) or use File>New Connection in the menu
   2. Select Amazon S3 from the drop-down menu
   3. Enter your Amazon credentials. See this page for more details.
   4. Click Connect
4. With both window (connections) open, you should be able to drag and drop files and folders between your CyVerse and S3 accounts.
5. Beware that large transfers (large files or many files) can take a long time.
   1. It is best to do this on a wired, high speed connection
   2. If you have to transfer thousands of files, consider breaking the transfer into batches.

Scripted transfer using Atmosphere

1. Creating a volume for bring the data into CyVerse
2. Launch a new instance from any of the Atmosphere base images and attach your volume to it.
3. Connect to your instance via web GUI or ssh
4. Install the AWS CLI (command line interface) on your instance
   1. see the instructions for installing on Linux
   2. you may need to add the CLI to your path
   3. Configure the CLI so it has your AWS security credentials
5. Run script to move data to instance 
   1. add directions
6. Transfer the data to your volume
   1. add directions
7. Unmount the volume and suspend the instance
   1. add details