Working with Amazon S3 – Part I


In this multi-part series, I am going to run you through Amazon S3 Service.

In first part, you will see the basics of Amazon S3 Service and why one should use it. Later, we will walk through S3 Web Management Console, which is a website interface for S3.

Next parts of this article will go through, accessing S3 programmatically with REST interface and with NodeJS SDK (S3 npm module), securing content on Amazon S3

What is Amazon S3

Amazon Simple Storage Service better known as The Amazon S3.
It is Storage as a service solution and provides the simple interface for storing and retrieving the data on cloud
In other words, it provides the infrastructure for any application storage needs. Your application can store and retrieve larger files up to 5TB, which is fast, efficient and reliable. And it provides the nice interfaces for accessing such as Web Management Console for accessing from Web browsers and REST API and SOAP interface via programmatically

Why use Amazon S3

  • It is fast enough to support high-performance applications.
  • It is inexpensive. The initial setup does not require any cost. But later as the storage grows you can pay for what you utilize which is very low cost
  • It is very secure, it provides access control tools to completely provides security to your resources stored on s3
  • As the storage of files kept on S3, it makes core application becomes lighter and faster to respond to other requests
  • It provides better scalability in terms of storage, request rate.
  • It is secure. All access happens over HTTPS
  • It is well integrated with other Amazon AWS services
  • It is used by big players such as Dropbox for their backend storage facility

Basic units of Amazon S3

Before starting anything further, lets have look at little explanation basic units of Amazon S3.
While working with Amazon S3, following two main units are base of Amazon S3 that you need to know-
Buckets
Buckets are the fundamental container to storing data. Every object is stored inside bucket. Each bucket can have any no. of data objects stored inside it. Buckets name are unique across the Amazon S3. So, name the bucket carefully, once bucket is created. it cannot be renamed.
With following two formats buckets can be accessed, following is format for bucket url-
bucket name as sub-domain
or
bucket name as path
Where,
bucket-name is name of bucket
object-name is objects name which is stored inside bucket
For eg,. we stored photos file inside mypics bucket then
Buckets provide namespace at higher level and way to organize the data objects stored inside s3. Also, you can apply access control to bucket for restricting access by creating policies around it.
Objects
Objects are basic entities stored in Amazon S3. Objects are stored in buckets.
Objects consists of data and meta-data. The data part can be anything but meta-data is typically set of key-value pairs that describes object. This meta-data can be common stuff like date- created, date-modified, mime-type or custom meta-data provided
So, as a user you can read, write, delete data objects inside bucket with simple web services interface provided, which is very simple to use and anyone gets started with it in no time.

Accessing Amazon S3 with Web Management Console

As said earlier, we will go through accessing the Amazon S3 with Web Management Console, which is website interface for S3.If you do not have AWS account, sign up
If you do not have AWS account, sign up here or login if you already have an account.
After that you will redirected to access AWS console here, which provides view of services provided by Amazon AWS. Our interest is in S3, So in Click S3 in Storage and Content Delivery category.
or
After successful sign in, access the S3 here,
You will see something like following-
(i already have amazon s3 account, so few buckets i have already created. In your case, if you are new user you wont see anything)
Web interface is simple and very minimal having only required functionality.left, side we have list of buckets created in your account. right pane is floating meaning it  gives information about various aspects as such properties (meta-data) of bucket, objects meta-data, etc
on left pane, we have list of buckets created in your account. righ pane is floating meaning it  gives information about various aspects as such properties (meta-data) of bucket,  objects meta-data, etc

Creating Bucket

Now, Lets create the bucket. Click fat blue button “Create Bucket” at top-left corner as in above figure.
Upon clicking, you will see modal, as shown in following figure, for setting basic details-
Name the bucket whatever you want and select region as appropriate-
After that, bucket will be created and listed in the left panel as shown in following figure –
Now, for our bucket tags-dev, it has now unique url, to which we can access the contents inside it as follows-
Lets, go inside created bucket by clicking from list, it will initially show empty as we don’t have anything inside it.

Uploading content

Its time to put some contents inside it. Lets upload few files. Also, we will create the folders inside to further organize objects.
Now, i have uploaded two files to S3 bucket and created a folder called “tags” as shown as follows-
I created folder from top left corner from white button named “Create Folder”-
The panel on right shows upload transfer progress, for each file you uploaded.
I also uploaded file inside “tags” folder-
The folder inside bucket gives extra namespace for organizing the content inside bucket. We can have as many folder we want in a bucket. Even nesting of folders also we can do.
Now, upload is done, lets try access the files as follows –
Now, if you tried to access it file you uploaded, you won’t to access it. This is because buckets are initially accessible to everyone by just URL, we have a read only policy to bucket which brings to following new topic

Configuring bucket policies

Following are the steps to create bucket policy-
  • Click to All Buckets link in breadcrumb at top-left, which will show all all bucket in left panel
  • Right Click on our bucket of interest, in our case “tags-dev”, then from menu select “Properties”
  • Now, right panel we will have the Properties of selected bucket as shown in following figure
  • Now, from right panel, click on Permission category, which will expand to give access controls tools for the bucket
     Which are Add more permission, Add bucket policy, Add CORS configurations.
     Now, for this article, i will go into only adding bucket policy.
  • So, let’s create bucket policy for read-only content. Which is very common to give access to resources. Go ahead and click “Add bucket policy”, it will open following
For now, Go ahead and paste following and also replace name of your bucket in place of “ExampleBucket”
 
Hit Save button, which will create the policy for bucketNow, if you try to acess the any object inside bucket, it will get access to it.

Deleting bucket

Finally, you may want to delete any objects it can be done by the right clicking on file, which reveal context menu which is shown as follows –

.
For deleting bucket, you need to empty bucket. i.e remove all the objects inside it. Then, you can delete it by right clicking and selecting Delete option.Which is shown as follows-

Thats Lots thing have been covered and this is end of first part of article.

Conclusion

In this article, we saw basic units of Amazon S3, why one should use it. and after that we tried to access the one of interface of Amazon S3 Infrastructure which is Web Management Console.
In which, we saw some basics of creating buckets, uploading files, giving them read only access and finally deleting objects and buckets.
Stick around for next article, where we will walk through programmatically accessing Amazon S3 with REST interface and a NodeJS module.

Using Git- Part -I : Basics


Introduction

Git is popular distributed version control system created by Linus Torvalds, creator of Linux OS. So, as you might have guessed it is first used for version controlling the Linux Kernel code.

quote-i-m-an-egotistical-bastard-and-i-name-all-my-projects-after-myself-first-linux-now-git-linus-torvalds-273567

Its widely used in most of open source and closed source software development. Thanks to Github popularity and its own feature sets.

Most of software open source projects foundation such as Eclipse Foundation recently moved its projects SVN and CVS repositories to Git. You can read more about here and here.

This is basic tutorial targeted at fellow beginners to git version control system. It shows very basic git workflow to get started with git.

Installation

For Windows and Mac OS

For this, go ahead to git-scm downloads site then go ahead to download executable files specific to your Operating System. Click on downloaded installer file to install git on your machine.

For Debian based OS- Ubuntu/Mint

Execute following commands in terminal window, it installs git using PPA (personal package archive)

sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

Now, you have installed the git, you can check for proper installation using git –version

Image 1

Note : For this tutorial, i will be using terminal for demonstration of all git commands. So, if you are on Windows OS, make use of bash command prompt that shipped with git installation and on Mac or Linux use normal terminal.

As now, we have git executable available from command prompt/terminal to execute the git commands.

Git executable has following basic format-

git <command> <switch-options/command-options> <sub-command> <more-sub-command-options>

Doing Git global user configuration

Command to be used : git config

First step after installation, is to do user configurations that is to setup the name and email id so, that git can identify while you making commit, that basically adds ownership of each commit.

Execute the following commands-

git config --global user.name <your-name>
git config --global user.email <your-email>

Creating Git repository

Command to be used : git init

For tracking files using git, first you need to create git repository first. Lets create hello-git directory and initialize the git repository in it.

mkdir hello-git
cd hello-git
git init

Image 2

After initializing you will see the message that Initialized git repository as shown in above screenshot and if you checked out the ‘hello-git’ directory , you will notice ‘.git’ directory is created. This is the directory where git operates.

Image 6

Creating files and adding it to git

Commands to be used : git add, git commit, git status

Now, lets add few files to our newly created git project.

Image 7

I have added the hello.js and README.md files to project directory, Now lets see what git tells us about status of our project directory.

git status– this command lets you see whats the status of files in a git project directory

Image 8

Git File states

As we are discussing from start that git tracks changes for git intialized directory. So, when working with git, filles goes to few states which explained as follows-

Untracked Files: This are kind of files which are newly added to directroy and it is not tracked by git for version control

Tracked Files : This are the files which are aleady committed to git but not staged or added to Index area

Staged / Index Files :These are the files which are to be added to next commit.

Modified / Unstaged files : These are files which are modifed but not staged

As you can see it says our two files are Untracked, meaning that we haven’t told git to track our files. For telling this, we going to use git add command

git add It adds untracked or unstaged files to Staging/ Index area of git.

Adding specific files :
git add <file-1> <file-2>

Adding all files in directory and sub-directories :
git add .
or
git add --all

For our case, i am going to add all files,

Image 10

As you can see that, files are added to staging area. Now, we need to commit our files to that is to add to git repository, for this we use git commit

git commit : In simple terms, this command does two thing first it adds files to git repository and makes the commit log based on commit message you provided

git commit -m “<your message>”

For our case,

git commit -m "Initial commit for project, added hello.js and README.md"

Image 11

Now, after doing this you will see that your changes are committed to git repository.

Now, if you do git status , you will see no changes to commit, that means our working directory changes are recorded by git

remiaing-1

git log : This let you see your older commits, in this you can see commit hash , name of author, date to which commit has been made and commit message

Image 12

To see more compact commit messages, use oneline switch on log command shown as follows-

remiaing-2

Doing more changes to files

Commands to be used : git diff, git add and git commit

Lets do more changes to files in repository and learn more git commands. Now, lets add few lines to hello.js and README.md, Open this files in the your favorite editor and add few lines to it. I have added few lines to files you can see those as follows-

Image 13 Image 14

Now, if you do git status

Image 15

You can see the files are tracked but not staged for commit, as compared previous un-tracked files. Now, in middle changing files, you might want to see what is been changes since last staged.

git diff : It shows you code diff’s between what you have changed in working directory since your last commit for each files in directory.

git diff <options> <file-name>

If you just do git diff, without specifying the file-name you diff of all files

Image 17

While if you specified file-name as follows, it shows diff of only that particular file

Image 18

You can see the lines in light green color, which are started with +, indicates that these lines are added since last commit.

Lets add files to staging area using git add command

git add .

Now, if do diff of repository, you will not see the anything because, by default diff command shows the diff’s between unstaged files.

To see diff’s of files that has been staged, you have to do following-

git diff --cached

Image 20

Now, lets commit the files-

git commit -m “added the more contents”

Image 21

and now if you do git log, you see our two commits as shown as follows-

Image 22

To get more information use –stat, which gives more information about which files are changed. Look at following image

Image 25

I Hope this tutorial helped you to at-least understand basics of Git. Git has lot of cool features you might need it later as you get advanced. For that, be sure to check out this blog again for more tutorials on same.

Image Credit : http://izquotes.com/quote/273567