Installing AWS CLI on windows to upload large files to Amazon S3

AWS CLI Upload Large Files Amazon S3

Amazon web services provides command line interface to interact will all parts of AWS, including Amazon EC2, Amazon S3 and other services.

In this post we discuss about installing AWS cLI on windows environment and using it to list, copy and delete Amazon S3 buckets and objects through command line interface. More information about the AWS CLI can be found in the AWS CLI User Guide.

AWS CLI makes data transfer very efficient. The most important functionality built into the AWS CLI for Amazon S3 tool is to help moving/copying large files very quickly.

This is done by two functionality built into AWS CLI. First, if we are trying to upload large files to Amazon S3, if the file size is over a certain size (5GB), AWS CLI to increase the performace and fastness, automatically breakes the file into smaller peices and upload them in parrllel. when all the peices are uploaded successfully, Amazon S3 will convert all peices into a single file like the orginal one.

AWS CLI sppeds up the uploading by using multi threads to upload files or peices/parts of files to Amazon S3.

Installing AWS CLI on Windows:

The AWS CLI is supported on Microsoft Windows XP or later.

1) Download the appropriate MSI installer.
Download the AWS CLI MSI installer for Windows (64-bit)
Download the AWS CLI MSI installer for Windows (32-bit)
Note: The 64-bit version of the AWS CLI does not currently work with Windows Server 2008 (version 6.0.6002). Please use the 32-bit installer with this version of Windows.

2) Run the downloaded MSI installer.
3) Follow the instructions that appear.

 
For 64-bit windows, AWS CLI will found in C:\Program Files\Amazon\AWSCLI
For 32-bit windows, AWS CLI will found in C:\Program Files (x86)\Amazon\AWSCLI

To confirm the successfull installation, use aws –version command at a command prompt.

For 62-bit
 C:\Program Files\Amazon\AWSCLI>aws --version

 For 32-bit
 C:\Program Files (x86)\Amazon\AWSCLI>aws --version

 You can use aws command from any any where by adding the installation directory to your PATH environment variable manually.

To modify your PATH variable (Windows)

1) Press the Windows key and type environment variables.

2) Choose Edit environment variables for your account.

3) Choose PATH and then choose Edit.

4) Add paths to the Variable value field, separated by semicolons.

C:\Program Files\Amazon\AWSCLI or C:\Program Files (x86)\Amazon\AWSCLI based on installation

5) Choose OK twice to apply the new settings.

6) Close any running command prompts and re-open.

Configuring AWS CLI

To configure AWS CLI, open the command line and run the below command

C:\Program Files\Amazon\AWSCLI>aws configure
 AWS Access Key ID [None]:
 AWS Secret Access Key [None]:
 Default region name [None]:
 Default output format [None]:

The AWS Access Key ID and AWS Secret Access Key are requied and they are your account credentials.

To get your access key ID and secret access key

1) Open the IAM console.

2) In the navigation pane, choose Users.

3) Choose your IAM user name (not the check box).

4) Choose the Security Credentials tab and then choose Create Access Key.

5) To see your access key, choose Show User Security Credentials. Your credentials will look something like this:

Access Key ID: AKIAIOSFODNN7HJKJEXAMPLE
Secret Access Key: wJalrXUtnFEMI/K7MDEJLKKNG/bPxRfiCYEXAMPLEKEY

More infomation: http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html

To update any of your settings, simply run aws configure again and enter new values as appropriate

Using command with AWS CLI

To list all buckets: 
C:\Program Files\Amazon\AWSCLI>aws s3 ls
To list all objects.folders in a bucket
C:\Program Files\Amazon\AWSCLI>aws s3 ls s3://bucket-name
To upload LARGE files:
C:\Program Files\Amazon\AWSCLI>aws s3 cp path_to_large_file s3://bucket_name/folder_name/

After it starts to upload the file, it will print the progress message like

Completed 1 part(s) with … file(s) remaining
at the beginning, and the progress message as follows when it is reaching the end.

Completed 9896 of 9896 part(s) with 1 file(s) remaining
After it successfully uploads the file, it will print a message like upload: path_to_large_file to s3://bucket_name/folder_name//file_name

You find high-level S3 commands here http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

 

AWS CLI Upload Large Files Amazon S3

Drupal Installing composer drush through command line

Drupal Installing composer and drush through command line.

This article explains drupal installing composer and drush through command line on windows environment. Composer is dependency Management for PHP. Drush is a command-line tool for building and maintaining sites.

Installing composer and drush for drupal

Composer helps you declare, manage and install dependencies of PHP projects. Use Composer to manage dependencies in your php applications.

Composer requires PHP 5.3.2+ to run. A few sensitive php settings and compile flags are also required, but when using the installer you will be warned about any incompatibilities.

Installing Composer

Download the installer to the current directory
php -r “copy(‘https://getcomposer.org/installer’, ‘composer-setup.php’);”

Verify the installer SHA-384
php -r “if (hash_file(‘SHA384’, ‘composer-setup.php’) === ‘669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410’) { echo ‘Installer verified’; } else { echo ‘Installer corrupt’; unlink(‘composer-setup.php’); } echo PHP_EOL;”

Run the installer
php composer-setup.php

To install it to specific directory
php composer-setup.php –install-dir=bin

To install a specific version
php composer-setup.php –version=1.0.0-alpha8

Remove the installer
php -r “unlink(‘composer-setup.php’);”

Test the Installation
composer -v

 

composer and drush

 

Drush is a command-line tool for building and maintaining drupal sites. Drush provides additional functionality not available via the administrative interface. It is used install he base software, modules, and themes; updating software; performing database operations, including copying and querying; resetting passwords; and clearing the cache. Doing administrative tasks using Drush is typically faster and less tedious than performing the same tasks through the administrative interface.

 

Installing Drush

Installing
composer require drush/drush

To run Drush
vendor/bin/drush help

 

Use full links:

https://getcomposer.org/download/

http://docs.drush.org/en/master/install/

 

Migrate MsSQL Database to Amazon RDS

Migrate MsSQL Database to Amazon RDS

Migrate MsSQL Database

Amazon Relational Database Service (Amazon RDS) supports native backup and restore for Microsoft SQL Server databases using full backup files (.bak files)
But in this article we will migrate database from mssql server to Amazon RDS with out using any .bak files. The approch is very simple. First we generate a database structure schema from local MsSQL database server and execute this schema on Amazon RDS. Then we export data from local MsSQL database to Amazon RDS.

Export Schema from current MsSQL Server

1) Open SQL Server Management Studio.
2) Connect to the local database server.
3) Select database which you want to export to RDS, right click on it and go to Task >> Generate Script

Migrate MsSQL Database 1

4) Select the database objects which you want to export.

Migrate MsSQL Database 2
There are two options here. Based on your requirement select one. For my requirement I selected second option.
By database have only Tables object, othere objects can be Views, Stored Procedures, User-defined Functions, Users etc…
Don’t select the Users. Every new database will already have Users object.

5) Save generated script which we will use on RDS database.

Migrate MsSQL Database 3
Based on your requirement choose the option. For my requirement I opted to save it to the file so that I can use it later.

6) Click Next and Next.
7) Wait until the script is created.

Execute database schema on amazon RDS

  1. Open SQL Server Management Studio.
  2. Connect to the Amazon RDS server.
  3. Create a new empty database by right clicking on Databases.
  4. Copy generated script from management studio or from the saved file and run in command prompt.
  5. Now if you check the database, it contains all the tables.

Migrate MsSQL Database to Amazon RDS (Data)

Now Migrate MsSQL Database data from the source database (local MsSQL database server) to the new database (Amazon RDS database). SQL Server Import and Export Wizard is used to accomplish this. You open this by right clicking on the source database and click Tasks > Export…

  1. First enter the connection details for the source database.
  2. Then enter the connection details for the target database.
  3. Select all the tables whose data has to be copied to destination database.
  4. Click on Edit Mappings…and select Enable Identity Insert check box.
  5. Click Next. Choose Run immediately,
  6. Click Next, Next and Finish.

php sort multi dimensional array by value

This post explains how to php sort multi dimensional array by value.

$books = array(
array(“type”=>”maths”, “price”=>13.50),
array(“type”=>”science”, “price”=>21.99),
array(“type”=>”social”, “price”=>15.75),
);

$books = sortMultiArrayByValue($inventory, ‘price’,’desc’);
echo “<pre>”;
print_r($books );

function sortMultiArrayByValue($array, $value, $sort_order = ‘asc’){
$sortarray = array();
if(is_array($array)){
foreach ($array as $key => $row){
$sortarray[$key] = $row[$value];
}
if(strtolower(trim($sort_order)) == ‘asc’){
array_multisort($sortarray, SORT_ASC, $array);
}else{
array_multisort($sortarray, SORT_DESC, $array);
}
}
return $array;
}

 

Output:

Array
(
[0] => Array
(
[type] => science
[price] => 21.99
)

[1] => Array
(
[type] => social
[price] => 15.75
)

[2] => Array
(
[type] => maths
[price] => 13.5
)

)

 

I2C using GPIO C code

Understanding the I2C Bus: I2C using GPIO

The I2C bus is a very popular and powerful bus used for communication between a master (or multiple masters) and a single or multiple slave devices. Figure 1 illustrates how many different peripherals may share a bus which is connected to a processor through only 2 wires, which is one of the largest benefits that the I 2C bus can give when compared to other interfaces.

Continue reading

Storage Classes in c language

Storage classes define the scope (visibility) and life time of variables which help us to trace the existence of a particular variable during the run time of a program.

C Language supports 4 types of storage class they are:

Storage Classes

  1. Auto
  2. Register
  3. Static
  4. Extern

Syntax:   <storage class> <data type> <variable name>;

ex:

auto int month; register int month; extern int month; static int month;

Auto:  A variable declared inside a function without any storage class specification, is by default an automatic variable. They are created when a function is called and are destroyed automatically when the function exits. Automatic variables can also be called local variables because they are local to a function. By default they are assigned garbage value by the compiler.

Example

{
   int mount;
   auto int month;
}

Register: The register variable functionality is same as auto storage class except that storage of variable is in register instead of memory. Register variable has faster access than normal variable. Frequently used variables are kept in register. Only few variables can be placed inside register depends on the register of processors. The Register variable does not guarantee the storage of variable in register it depends on the availability of register.

NOTE : We can never get the address of such variables.

Example

{
   register int i;
      for(i=0;i<100;i++) {
   printf("register storage class"); }
}

In the above example the variable i is accessed in loop. using register declaration the loop will be executed fast compared to normal storage class i.e auto.

Static : This storage class is used to declare static variables which are popularly used while writing programs in C language. Static variables have a property of preserving their value even after they are out of their scope! Hence, static variables preserve the value of their last use in their scope. So we can say that they are initialized only once and exist till the termination of the program. Thus, no new memory is allocated because they are not re-declared. Their scope is local to the function to which they were defined. Global static variables can be accessed anywhere in the program. By default, they are assigned the value 0 by the compiler.

Example

#include <stdio.h>
void fun()
{
 static number;
 printf("%d\t", number); 
 number++;
}
void main()
{
 fun();
 fun();
 fun(); 
}

output: 0 1 2

In the above example the static variable number is declared inside the fun function but still the value persist after multiple call the  fun function from main.

Extern: The extern keyword is used before a variable to inform the compiler that this variable is declared somewhere else. The extern declaration does not allocate storage for variables.

When you have multiple files and you define a global variable or function, which will also be used in other files, then extern will be used in another file to provide the reference of defined variable or function. Just for understanding, extern is used to declare a global variable or function in another file.

The extern modifier is most commonly used when there are two or more files sharing the same global variables or functions as explained below.

Example

file1: main.c

#include <stdio.h>
 
int count ;
extern void write();
 
main() {

   count = 5;
   write();
}

file2: support.c

#include <stdio.h>
 
extern int count;

void write(void) {
   printf("count is %d\n", count);
}

To compile:

$gcc main.c support.c

It will produce the executable program a.out. When this program is executed, it produces the following result ?

count is 5

 

cakephp titbits usefull development tips

Some of usefull cakephp titbits. These cakephp titbits are for quick reference that are help full while developing the application. These titbits are most commonly used in the development and comes as handy reference.

To get the paths, Url and URL query string

echo $this->request->webroot  will print the webroot directory path.
echo $this->request->base will print the base path.
echo $this->request->here will print the full address to the current request.
echo $this->request->query will print the query string parameters that are passed through URL as query string.

To detect if the current page is the homepage

if ($this->here == $this->webroot){ // this is home page }
or 
if ($this->here == $this->base.'/'){ // this is home page }

To include Element in layout or view

echo $this->element('navbar');

// create the sidebar block. 
$this->start('sidebar'); 
echo $this->element('sidebar/recent_topics');
echo $this->element('sidebar/recent_comments');
$this->end();

To load CSS and JS files in layout

echo $this->fetch(‘css’);
echo $this->Html->css(array(‘forms’, ‘tables’, ‘menu’));
echo $this->fetch(‘script’);
echo $this->Html->css(array(‘forms’, ‘tables’, ‘menu’));

 

Loading vendor files

// Load the class Abc in app/Vendor/Abc.php
App::uses('Abc', 'Vendor');

// To load app/Vendor/abc.php:
App::import('Vendor', 'abc');

//To load app/Vendor/abc/xyz.php:
App::import('Vendor', 'abc/xyz');


To include one controller in another controller

// The same as require('Controller/UsersController.php');
App::import('Controller', 'Users');

// We need to load the class
$Users = new UsersController();

// If we want the model associations, components, etc to be loaded
$Users->constructClasses();

 

Cakephp HtmlHelper functions can be found here

Cakephp FormHelper functions can be found here

 

cakephp login with Email or username

This article “cakephp login with Email or username” will explain how the Auth component will allow user cakephp login with  email or username, instead of using any of username and email address.

Now a days in many application the authentication process is using  either  the username or email with password combination to allow access to the site. In cakephp framework with auth component we can also achieve the same. To allow cakephp login with email or username we need to add some extra code in the app controller which tells the auth component to allow login with either email or username with password combination.

Auth component is used in the cakephp for the process of authentication. It by default allow login with username and password combination. To make auth component  allow cakephp login with email or username we need to add below code in app controller

App controller file will available in “app/controllers” folder with the file name appcontroller.php.

For the combination of

Email + Password
OR
Username + Password

public $components = array(
    'Auth' => array(
        'authenticate' => array(
            'Form' => array(
                'fields' => array('username' => array('username', 'email'),'password' => 'password')
            )
        )
    )
);

 

To make auth component  allow cakephp login with only email address we need to add below code in app controller.

For the combination of

Email + Password

public $components = array(
    'Auth' => array(
        'authenticate' => array(
            'Form' => array(
                'fields' => array('username' =>  'email','password' => 'password')
            )
        )
    )
);

 

To make auth component  allow cakephp login with only username we need to add below code in app controller.

For the combination of
Username + Password

public $components = array(
    'Auth' => array(
        'authenticate' => array(
            'Form' => array(
                'fields' => array('username' =>  'username','password' => 'password')
            )
        )
    )
);

cakephp login with Email or username

Cakephp interview questions

Here there are some Cakephp interview questions and answers about How CakePHP Framework works. These cakephp interview questions listed below are  mainly asked in any interview. These cakephp interview questions  are use full both fresher and experienced people.

 

What is Cakephp?

CakePHP is a free, open-source, rapid development framework for PHP. It’s a foundational structure for programmers to create web applications. CakePHP goal is to enable developers to work in a structured and rapid manner–without loss of flexibility. CakePHP takes the monotony out of web development.

Server requirements for cakephp?

Here are the requirements for setting up a server to run CakePHP:
An HTTP server (like Apache) with the following enabled: sessions, mod_rewrite (not absolutely necessary but preferred)
PHP 4.3.2 or greater. Yes, CakePHP works great in either PHP 4 or 5.
A database engine (right now, there is support for MySQL 4+, PostgreSQL and a wrapper for ADODB).

Installation of cakephp

01. Go to cakephp.org and download the latest version of cakephp.
02. Cakephp comes in a .zip file so unzip it.
03. Extract the files in the localhost in the desired folder ( lets say cakephp)
04. Open the browser and run the URL localhost/cakephp
05. Follow the instruction showed in the page

what is the folder structure of Cakephp

cakephp/
app/
config/
controllers/
models/
plugins/
tmp/
vendors/
views/
webroot/
.htaccess
index.php
lib/
plugins/
vendors/
.htaccess
index.php
README

List some of the features in Cakephp?

Basic Features List

  1.     Full support of Model, View, Controller Architecture.
  2.     Good Application Scaffolding.
  3.     Very nice Code generation via Bake.
  4.     Helpers for HTML, Forms, Pagination, AJAX, Javascript, XML, RSS and more.
  5.     Access Control Lists and Authentication.
  6.     Router for mapping urls and handling extensions.
  7.     Security, Session, and RequestHandler Components.
  8.     Utility classes for working with Files, Folders, Arrays and more.
  9.     can manage multiple site(apps) developement

Advance Features List

  1.     Internationalization and Localization with static translations in gettext style or dynamic translations of model data.
  2.     Full Console environment with support for multiple tasks. Included shells: bake, schema, acl, i18 extractor, api.
  3.     CSRF protection via Security Component.
  4.     HTTP Authentication via Security Component.
  5.     Flexible Caching: use memcache, apc, xcache, the file system, or a model to speed up your applicationConfigure class to provide dynamic handling of configuration settings and App class to handle importing required classes. Supports Unit Testing (simpletest,unit test)

what is the difference between two vendors folder available in cakephp?

There will be two vendor folders available in cakephp frame work.
one in ” app ” folder and one in root folder
The vendor folder in the app folder is used to place the third-party libraries which are application specific.
The vendor folder in the root folder is used to place the third-party libraries which are used for multiple applications.

what is the naming convention in cakephp.

According to cakephp naming conventions
1) Table names should be in lowercase and always Plural
2) Model name should be in CamelCased and are always singular eg: ModelName
3) Model file names are also singular and underscored if contains two words eg : model_name.php
4) Controller name should be in CamelCased and are always plural : appended “controller ” eg: ControllerNamesController
5) Controller file names are always plural and underscored : appender “controller” eg: controller_names_controller.php

what is model,view,controller?

1. The Model represents the application data
2. The View renders a presentation of model data
3. The Controller handles and routes requests made by the client

 

what is a component?
Components in CakePHP are associated with Controllers.
Components are created and used to share common functionality bewteen controllers.

What are commonly used components of cakephp?
1. Security
2. Sessions
3. Access control lists
4. Emails
5. Cookies
6. Authentication
7. Request handling

what is a Helper?
Helpers in CakePHP are associated with Presentation layers of application.
Helpers mainly contain presentational logic which is availabel to share between many views, elements, or layouts

What are commonly used helpers of cakephp?
CacheHelper
FormHelper
HtmlHelper
JsHelper
NumberHelper
Paginator
RSS
SessionHelper
TextHelper
TimeHelper

What is a Behavior?
Behaviors in CakePHP are associated with Models.
Behaviors are used to change the way models behaves and enforcing model to act as something else.

Difference between Component, Helper, Behavior?
Component is a Controller extension, Helpers are View extensions, Behavior is a Model Extension.

What is a Element?
Element in cakephp are smaller and reusable bits of view code. Elements are usually rendered inside views.

What is a layout?
Layout in cakephp are used to display the views that contain presentational code. In simple views are rendered inside a layout

what is the first function that gets loaded from a controller?
function index()

In which file cake php database configuration is setup ?
“app/config/database.php”

The four model association types in CakePHP are
1) hasOne
2) hasMany
3) belongsTo
4) hasAndBelongsToMany (HABTM).

Database related functions in cakephp.
1) find
2) find(‘first’)
3) find(‘count’)
4) find(‘all’)
5) find(‘list’)
6) find(‘threaded’)
7) query(),
8) save()

How to set layout in the controller?

var $layout = ‘layout_name’;
to overwrite for a specific action use below code in that action
$this->layout =”layout_name”;

How to including helpers in controller ?

public $helpers = array(‘Form’, ‘Html’, ‘Js’, ‘Time’);
to in specific action use below code in that action
$this->helper[] =”helper_name”;

How to including components in controller ?

public $components = array(‘Emails’, ‘ImageUploader’, ‘Sms’);

Session write, read and delete

1)$this->Session->write(‘Person.eyeColor’, ‘Green’);
2)$green = $this->Session->read(‘Person.eyeColor’);
3)$this->Session->delete(‘Person’);

Which function is executed before every action in the controller?

function beforeFilter()

requestAction is used to ?

To call controller function form view requestAction function is used
$abc = $this->requestAction(‘/controller/action/param:1234’);
// or
$abc = $this->requestAction(‘controller’=>’controller_name’, ‘action’=>’action_name’, ‘param’ => 1234);

what all are drawbacks?

1) The biggest drawback to CakePHP is that it doesn’t completely implement object orientation
2) Can’t be used for small scale apps as it loads the complete application in beginning.
3) The learning curve

php cache engines

PHP Cache Engines

This article will explain all the available cache engines for php. PHP Cache engines are used to implement cache mechanism for the dynamic sites developed using scripting and programming languages. Cache mechanism will help to increase site performance.

CakePHP/PHP comes with several cache engines built-in, and provides an easy system to implement our own caching systems. The built-in caching engines are:

FileCache :

  • File cache is a simple cache that uses local files.
  • Not quite fast and doesn’t provide as many features for atomic operations
  • Concurrent access are not great at all, if several processes try to read/write at the same time
  • since disk storage is often quite cheap, storing large objects, or elements that are infrequently written work well in files

ApcCache  :

  • APC cache uses the PHP APC extension.
  • Really fast
  • Able to provide atomic read/write features
  • But you have less space
  • Uses shared memory on the webserver to store objects and it’s local to each server too

Wincache :

  • Wincache uses the Wincache extension.
  • Wincache is similar to APC in features and performance, but optimized for Windows and IIS.

XcacheEngine :

  • Xcache is a PHP extension that provides similar features to APC.

MemcacheEngine :

  • Uses the Memcache extension.
  • Provides a very fast cache system
  • Can be distributed across many servers, and provides atomic operations

MemcachedEngine :

  • Uses the Memcached extension.
  • It also interfaces with memcache but provides better performance.
  • Quite fast (a bit less than APC ; but still pretty fast)
  • Shared between all your servers : each item has to be cached only once, even if you have several webservers.
  • You can have several servers in your memcached cluster (which means virtually no limit in the size of the cache)

RedisEngine  :

  • Uses the phpredis extension.
  • Redis provides a fast and persistent cache system similar to memcached
  • also provides atomic operations

Read more about Memcache here