AWS CLI Upload Large Files Amazon S3
Amazon web services provides command line interface to interact will all parts of AWS, including Amazon EC2, Amazon S3 and other services.
In this post we discuss about installing AWS cLI on windows environment and using it to list, copy and delete Amazon S3 buckets and objects through command line interface. More information about the AWS CLI can be found in the AWS CLI User Guide.
AWS CLI makes data transfer very efficient. The most important functionality built into the AWS CLI for Amazon S3 tool is to help moving/copying large files very quickly.
This is done by two functionality built into AWS CLI. First, if we are trying to upload large files to Amazon S3, if the file size is over a certain size (5GB), AWS CLI to increase the performace and fastness, automatically breakes the file into smaller peices and upload them in parrllel. when all the peices are uploaded successfully, Amazon S3 will convert all peices into a single file like the orginal one.
AWS CLI sppeds up the uploading by using multi threads to upload files or peices/parts of files to Amazon S3.
Installing AWS CLI on Windows:
The AWS CLI is supported on Microsoft Windows XP or later.
1) Download the appropriate MSI installer.
Download the AWS CLI MSI installer for Windows (64-bit)
Download the AWS CLI MSI installer for Windows (32-bit)
Note: The 64-bit version of the AWS CLI does not currently work with Windows Server 2008 (version 6.0.6002). Please use the 32-bit installer with this version of Windows.
2) Run the downloaded MSI installer.
3) Follow the instructions that appear.
For 64-bit windows, AWS CLI will found in C:\Program Files\Amazon\AWSCLI
For 32-bit windows, AWS CLI will found in C:\Program Files (x86)\Amazon\AWSCLI
To confirm the successfull installation, use aws –version command at a command prompt.
For 62-bit C:\Program Files\Amazon\AWSCLI>aws --version For 32-bit C:\Program Files (x86)\Amazon\AWSCLI>aws --version You can use aws command from any any where by adding the installation directory to your PATH environment variable manually.
To modify your PATH variable (Windows)
1) Press the Windows key and type environment variables.
2) Choose Edit environment variables for your account.
3) Choose PATH and then choose Edit.
4) Add paths to the Variable value field, separated by semicolons.
C:\Program Files\Amazon\AWSCLI or C:\Program Files (x86)\Amazon\AWSCLI based on installation
5) Choose OK twice to apply the new settings.
6) Close any running command prompts and re-open.
Configuring AWS CLI
To configure AWS CLI, open the command line and run the below command
C:\Program Files\Amazon\AWSCLI>aws configure AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [None]: Default output format [None]:
The AWS Access Key ID and AWS Secret Access Key are requied and they are your account credentials.
To get your access key ID and secret access key
1) Open the IAM console.
2) In the navigation pane, choose Users.
3) Choose your IAM user name (not the check box).
4) Choose the Security Credentials tab and then choose Create Access Key.
5) To see your access key, choose Show User Security Credentials. Your credentials will look something like this:
Access Key ID: AKIAIOSFODNN7HJKJEXAMPLE
Secret Access Key: wJalrXUtnFEMI/K7MDEJLKKNG/bPxRfiCYEXAMPLEKEY
To update any of your settings, simply run aws configure again and enter new values as appropriate
Using command with AWS CLI
To list all buckets: C:\Program Files\Amazon\AWSCLI>aws s3 ls
To list all objects.folders in a bucket C:\Program Files\Amazon\AWSCLI>aws s3 ls s3://bucket-name
To upload LARGE files: C:\Program Files\Amazon\AWSCLI>aws s3 cp path_to_large_file s3://bucket_name/folder_name/
After it starts to upload the file, it will print the progress message like
Completed 1 part(s) with … file(s) remaining
at the beginning, and the progress message as follows when it is reaching the end.
Completed 9896 of 9896 part(s) with 1 file(s) remaining
After it successfully uploads the file, it will print a message like upload: path_to_large_file to s3://bucket_name/folder_name//file_name
You find high-level S3 commands here http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
AWS CLI Upload Large Files Amazon S3