So here is the command for registering an EBS backed image based on a particular snapshot:

ec2-register --block-device-mapping /dev/sda1=s<snapshot id>::false
--name "<ami name>" --kernel <kernel id> --ramdisk <ramdisk id>
-K <key file> -C <cert file>

(Note: If you want the volume to persist after termination then you’ll want to change the true in the –block-device-mapping flag to false)

So the key is in the –block-device-mapping flag.  The documentation for Amazon states:

Block Device Mapping Documentation

So it says somewhere in there that you can specify the size of the AMI in that particular flag, so if I want to increase the size of the instance all I would need to do is change the flag to read:

--block-device-mapping /dev/sda1=s<snapshot id>:50:true

or whatever size you want.  Afterwards you’ll need to log on to the server and resize the drive.  You can do this by running the following (assuming you’re using an ext3 root file system, which is what I used in my previous post):

sudo resize2fs /dev/sda1

it’ll resize the drive and you should be good to go.

Share/Save

So here is how I did it:

1) Start Up an Instance

The first thing I do is start up an instance.  Everyone has instructions on how to do this from the command line.  But I’m lazy and find that Amazon’s API tools change regularly which is annoying and makes it difficult to follow the instructions other people have written.  I recommend doing it in the AWS Console.

Go to EC2 in the AWS Console.  Choose AMI and find an image that you like (I tend to lean toward Alestic’s Ubuntu images, but that’s just me).  Once I found an AMI I like I highlight it and then choose “Launch Instance”.  I follow the instructions, making sure that I choose the correct security group (aka firewall) and the correct key pair (you access ec2 servers using a key pair rather than a password).

Once the server is up and running, I update everything, configure it, transfer data onto it, blah blah blah.  The usual things that you do when you are prepping a server.  Once I’m all set I’m ready to make it an EBS backed AMI.

2) Create a New EBS Volume

The next thing I do is create an EBS Volume, again doing it in the AWS Console.  Go to the EC2 section of the AWS Console.  Choose “Volumes” and then “Create Volume”.  You’ll want to make sure of two things:

First that the volume is large enough to hold everything.  I forgot to do that the first time around and *thought* I had to do everything all over again (see next post for how to really do it).  The easiest thing would be to just size it right the first time around.

Second that the volume is in the same availability zone as the instance you started up in step 1.  If you aren’t sure what the availability zone is for your instance, go back to the instance, highlight it, and in the bottom panel you’ll see what zone its in.

3) Attach Volume to Instance

When you are done you’ll want to attach the volume to the instance that you created in step 1.  Once the volume is available, highlight it and choose “Attach Volume”.  You’ll be stepped through attaching it to your server.  If you have only one instance running its pretty straightforward.  If you have multiple ones running, you’ll want to make sure that you’re attaching it to the right place.  You’ll be asked to indicate where you want the attached, I usually do /dev/sdf.  Oh and this process won’t work if the volume and the instance aren’t in the same availability zone, hence my warning in the previous step.

4) Copy the Data from the Server to the Volume

This part of the instructions I found over at Winners Don’t Lose.  This helps you format the volume and copy the data on the current instance over to the volume:

In a root shell on the instance:

# mkfs.ext3 /dev/sdf
# mkdir /mnt/target && mount /dev/sdf /mnt/target
# rsync -avHx / /mnt/target
# rsync -avHx /dev /mnt/target
# sync;sync;sync;sync && umount /mnt/target

The above commands did the following:

formatted the entire volume /dev/sdf as an extended 3 filesystem
created directory /mnt/target and mounted /dev/sdf at /mnt/target
rsync’d the root instance-store filesystem to the ebs volume
synchronized the /dev directory from the instance-store filesystem
flush all pending write ops, and unmount the EBS volume

To be honest these are the only instructions I’ve ever been able to follow that doesn’t produce some error that I have to debug.

5) Detach the Volume

The next step is to detach the volume that you just created from the running instance.  Again you can do this in the AWS console by finding the volume, highlighting it, and choosing detach.  Make sure the you’re detaching the right volume though!

6) Create a Snapshot of the Volume

You’ll want to create a snapshot of the volume that you just created.  You can again do that in the AWS Management console but finding the volume, right clicking on it, and choosing create snapshot.

7) Register your AMI

This is the part that took me forever to figure out.  Mostly because the documentation for the ec2-register command didn’t jive with the tools that i had access too.  The command I used was:

ec2-register --block-device-mapping /dev/sda1=<snapshot id>::true
--name <ami name> --kernel <kernel id> --ramdisk <ram disk id>
-K <your pk file>.pem -C <your cert file>.pem

(note i updated this:  if you want the volume to persist after termination then you’ll want to change the true in the –block-device-mapping flag to false)

You’ll want to make sure that you specify and kernel id and a ram disk id, I usually use the same one that my instance I’m creating this new AMI from was using.  You don’t have to do it, but that would mean that if your AMI users don’t choose one during start up, then the instance won’t work.

Note that you may have your key and cert file set as part of your environment variable.  I have multiple accounts so I don’t, and instead just specify them when I’m running any commands (annoying but I don’t do much from the commandline anyway).

Share/Save

The first thing I did was start with an already existing EBS AMI. Right now there aren’t too many available but they are growing.  I chose an Ubuntu 9.04 image since I’m going to install a stack on it, and I’ve already done *most* of it before on Ubuntu.

A quick note: it is possible to create an EBS AMI from an S3 AMI, but I wasn’t able to figure that out (quickly).  Although I did find a script that will do this for you: http://gist.github.com/249915.

Next, I wanted to get the Ubuntu server prepared.  I needed to:

• install mysql — http://www.howtogeek.com/howto/ubuntu/install-mysql-server-5-on-ubuntu/
• install ruby on rails — http://www.hackido.com/2009/04/install-ruby-rails-on-ubuntu-904-jaunty.html (up to and including step 6)
• install zlib — http://www.techsww.com/tutorials/libraries/zlib/installation/installing_zlib_on_ubuntu_linux.php
• install blacklight — http://projectblacklight.org (includes solr and a number of gem dependencies)

I did all of this.  One thing I noticed, Blacklight requires me to use non-standard ports (ie not 22 and 80).  While the server was running, I went into the AWS console and opened ports for its associated security group.  The changes took effect immediately.  I definitely liked that.

Finally, I went in to the AWS Management Console and chose “Create Image (EBS AMI)”.

Then I filled in the information including a description of my image:

and voila! EBS backed AMI.  the one caveat is:

DON’T TERMINATE THE INSTANCE UNTIL THE NEW AMI HAS FINISHED BEING CREATED

because then you’ll loose all the work that you have just done.  doh!

Share/Save

Why am I rethinking it?  Well today Amazon EC2 realeased three new features; Amazon CloudWatch, Auto Scaling, and Elastic Load Balancing.

Amazon CloudWatch keeps a number of different metrics on how your AMIs are being utilized (CPU utilization, Network I/O, etc.).  And you can access the data either through the API or commandline tools (it would be nice though if this were built into the Management Console though).

Auto Scaling and Elastic Load Balancing work in conjunction with Amazon CloudWatch to help you get the most out of your EC2 instances.  Auto Scaling allows you to define certain conditions that, when present, allow you to perform certain actions (like scale up your instances or scale them down).  Elastic Load Balancing automatically spreads the load of traffic amongst your EC2 instances based on the metrics provided by CloudWatch.

All in all pretty cool, however I would love to see it all built into the management console rather than just available through the API and commandline tools (I am lazy afterall Amazon).  If you want to read more check out Amazon Web Services’ Blog or follow the links above.

Share/Save

They have broken up their grants into 3 sections: researchers, educators, students.  They also mentioned that they are working with IT companies to provide services for IT Professionals on campus (including a company that uses AWS for email archiving).

Of course Amazon mentioned some big institutions using AWS in their classrooms (including UMD and Harvard), but here at Wheaton College Professor Tom Armstrong is using AWS for a distributed computing assignment.

Share/Save

Today, we are introducing the AWS Toolkit for Eclipse. This free, open source plugin for the Eclipse IDE makes it easier and more efficient for you to develop, deploy, and debug Java applications on top of AWS.

By bringing the popular IDE (Integrated Development Environment) to AWS, they are making it easier for developers to create server clusters for their applications.

Share/Save

Also its hard to convince people that I really am saving money by using Amazon Web Services.  The problem is that the electricity, cooling, server, support, all come out of different budgets.  So if you add it all up its a lot more expensive to have a real live server than a virtual one.  But the people that pay the electricity bill aren’t going to send me the money that I’m saving them.

So what can I do?  Well here at Wheaton we’ve set up NComputing.  Essentially there is a server that the computers connect to.  You install whatever productivity software you want (and your licenses allow).  And then users sit down, log in to the teeny tiny square box you see off to the left, and bam! instant computer.

This allows an IT department to set up computers for a fraction of the cost of setting up a “real” one.  Our Director of Technical Support sent out an email letting us now we’re saving roughly $18,000 dollars by implementing these 10 boxes rather than 10 traditional computers.  That’ll pay for my Amazon Reserved Instance for at least 5 (if not more) years (don’t forget Amazon charges you for get and put requests as well as file uploading).

So if you’re looking for a way to get a bit cloudy without diving head long into utility computing, try virtualizing your desktop computers.  It provides you with the ability to save actual real live money out of your own budget, and has the added benefit of reducing electricity on campus (and maybe you can convince the guy that pays the electric bill that you should get more money from his budget).

Share/Save

When Amazon first came out, it wasn’t cost effective for me to host my website there (in truth it still isn’t cause I have a friend that gives me a deal).  But now it is; Reuven Cohen over at Cloud Interoperability Magazine writes:

In doing the math, @ $0.03 a hour, a small reserved EC2 instance will cost you about $262 a year for the uptime, and $325 for the reservation or about $48 dollars a month. Compared to about $876 a year or $73 a month using an on demand instance (not including storage and bandwidth).

Like I said, its still not cheaper for me, but the thought of being able to do whatever crazy scheme I want on a server sure is enticing.

So if you hate your hosting provider (I hear lots of people whining about Dreamhost), perhaps you should consider switching.

Share/Save

The datasets are available via Elastic Block Storage:

Amazon EC2 customers can access this data by creating their own personal Amazon EBS volumes, using the public data set snapshots as a starting point. They can then access, modify and perform computation on these volumes directly using their Amazon EC2 instances and just pay for the compute and storage resources that they use.

All of this can be done through the AWS Web Console > Elastic Block Store > Volumes.  When you click “Create Volume” a new window will up and there is a dropdown box that allows you to choose from available datasets or you can use the snapshot id numbers that are shown on the public datasets page (because not all datasets are listed in the dropdown box).

What’s even better is that you too can share your data with the world by filling out a form (also found on the public datasets page) and talking to Amazon.

a view of EBS from the AWS Web Console

Share/Save

Essentially I launch an AMI, associate it to an IP address and then connect to it using terminal.  Pretty simple really but super powerful if you have no money and need a server quickly.

The screencast below is a bit small so here’s a link to it in case you need to see it bigger http://www.screencast.com/t/kIBVibIsL5T

Share/Save