Installing AWS CLI on windows to upload large files to Amazon S3

AWS CLI Upload Large Files Amazon S3

Amazon web services provides command line interface to interact will all parts of AWS, including Amazon EC2, Amazon S3 and other services.

In this post we discuss about installing AWS cLI on windows environment and using it to list, copy and delete Amazon S3 buckets and objects through command line interface. More information about the AWS CLI can be found in the AWS CLI User Guide.

AWS CLI makes data transfer very efficient. The most important functionality built into the AWS CLI for Amazon S3 tool is to help moving/copying large files very quickly.

This is done by two functionality built into AWS CLI. First, if we are trying to upload large files to Amazon S3, if the file size is over a certain size (5GB), AWS CLI to increase the performace and fastness, automatically breakes the file into smaller peices and upload them in parrllel. when all the peices are uploaded successfully, Amazon S3 will convert all peices into a single file like the orginal one.

AWS CLI sppeds up the uploading by using multi threads to upload files or peices/parts of files to Amazon S3.

Installing AWS CLI on Windows:

The AWS CLI is supported on Microsoft Windows XP or later.

1) Download the appropriate MSI installer.
Download the AWS CLI MSI installer for Windows (64-bit)
Download the AWS CLI MSI installer for Windows (32-bit)
Note: The 64-bit version of the AWS CLI does not currently work with Windows Server 2008 (version 6.0.6002). Please use the 32-bit installer with this version of Windows.

2) Run the downloaded MSI installer.
3) Follow the instructions that appear.

 
For 64-bit windows, AWS CLI will found in C:\Program Files\Amazon\AWSCLI
For 32-bit windows, AWS CLI will found in C:\Program Files (x86)\Amazon\AWSCLI

To confirm the successfull installation, use aws –version command at a command prompt.

For 62-bit
 C:\Program Files\Amazon\AWSCLI>aws --version

 For 32-bit
 C:\Program Files (x86)\Amazon\AWSCLI>aws --version

 You can use aws command from any any where by adding the installation directory to your PATH environment variable manually.

To modify your PATH variable (Windows)

1) Press the Windows key and type environment variables.

2) Choose Edit environment variables for your account.

3) Choose PATH and then choose Edit.

4) Add paths to the Variable value field, separated by semicolons.

C:\Program Files\Amazon\AWSCLI or C:\Program Files (x86)\Amazon\AWSCLI based on installation

5) Choose OK twice to apply the new settings.

6) Close any running command prompts and re-open.

Configuring AWS CLI

To configure AWS CLI, open the command line and run the below command

C:\Program Files\Amazon\AWSCLI>aws configure
 AWS Access Key ID [None]:
 AWS Secret Access Key [None]:
 Default region name [None]:
 Default output format [None]:

The AWS Access Key ID and AWS Secret Access Key are requied and they are your account credentials.

To get your access key ID and secret access key

1) Open the IAM console.

2) In the navigation pane, choose Users.

3) Choose your IAM user name (not the check box).

4) Choose the Security Credentials tab and then choose Create Access Key.

5) To see your access key, choose Show User Security Credentials. Your credentials will look something like this:

Access Key ID: AKIAIOSFODNN7HJKJEXAMPLE
Secret Access Key: wJalrXUtnFEMI/K7MDEJLKKNG/bPxRfiCYEXAMPLEKEY

More infomation: http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html

To update any of your settings, simply run aws configure again and enter new values as appropriate

Using command with AWS CLI

To list all buckets: 
C:\Program Files\Amazon\AWSCLI>aws s3 ls
To list all objects.folders in a bucket
C:\Program Files\Amazon\AWSCLI>aws s3 ls s3://bucket-name
To upload LARGE files:
C:\Program Files\Amazon\AWSCLI>aws s3 cp path_to_large_file s3://bucket_name/folder_name/

After it starts to upload the file, it will print the progress message like

Completed 1 part(s) with … file(s) remaining
at the beginning, and the progress message as follows when it is reaching the end.

Completed 9896 of 9896 part(s) with 1 file(s) remaining
After it successfully uploads the file, it will print a message like upload: path_to_large_file to s3://bucket_name/folder_name//file_name

You find high-level S3 commands here http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

 

AWS CLI Upload Large Files Amazon S3

Migrate MsSQL Database to Amazon RDS

Migrate MsSQL Database to Amazon RDS

Migrate MsSQL Database

Amazon Relational Database Service (Amazon RDS) supports native backup and restore for Microsoft SQL Server databases using full backup files (.bak files)
But in this article we will migrate database from mssql server to Amazon RDS with out using any .bak files. The approch is very simple. First we generate a database structure schema from local MsSQL database server and execute this schema on Amazon RDS. Then we export data from local MsSQL database to Amazon RDS.

Export Schema from current MsSQL Server

1) Open SQL Server Management Studio.
2) Connect to the local database server.
3) Select database which you want to export to RDS, right click on it and go to Task >> Generate Script

Migrate MsSQL Database 1

4) Select the database objects which you want to export.

Migrate MsSQL Database 2
There are two options here. Based on your requirement select one. For my requirement I selected second option.
By database have only Tables object, othere objects can be Views, Stored Procedures, User-defined Functions, Users etc…
Don’t select the Users. Every new database will already have Users object.

5) Save generated script which we will use on RDS database.

Migrate MsSQL Database 3
Based on your requirement choose the option. For my requirement I opted to save it to the file so that I can use it later.

6) Click Next and Next.
7) Wait until the script is created.

Execute database schema on amazon RDS

  1. Open SQL Server Management Studio.
  2. Connect to the Amazon RDS server.
  3. Create a new empty database by right clicking on Databases.
  4. Copy generated script from management studio or from the saved file and run in command prompt.
  5. Now if you check the database, it contains all the tables.

Migrate MsSQL Database to Amazon RDS (Data)

Now Migrate MsSQL Database data from the source database (local MsSQL database server) to the new database (Amazon RDS database). SQL Server Import and Export Wizard is used to accomplish this. You open this by right clicking on the source database and click Tasks > Export…

  1. First enter the connection details for the source database.
  2. Then enter the connection details for the target database.
  3. Select all the tables whose data has to be copied to destination database.
  4. Click on Edit Mappings…and select Enable Identity Insert check box.
  5. Click Next. Choose Run immediately,
  6. Click Next, Next and Finish.