So here is the command for registering an EBS backed image based on a particular snapshot:

ec2-register --block-device-mapping /dev/sda1=s<snapshot id>::false
--name "<ami name>" --kernel <kernel id> --ramdisk <ramdisk id>
-K <key file> -C <cert file>

(Note: If you want the volume to persist after termination then you’ll want to change the true in the –block-device-mapping flag to false)

So the key is in the –block-device-mapping flag.  The documentation for Amazon states:

Block Device Mapping Documentation

So it says somewhere in there that you can specify the size of the AMI in that particular flag, so if I want to increase the size of the instance all I would need to do is change the flag to read:

--block-device-mapping /dev/sda1=s<snapshot id>:50:true

or whatever size you want.  Afterwards you’ll need to log on to the server and resize the drive.  You can do this by running the following (assuming you’re using an ext3 root file system, which is what I used in my previous post):

sudo resize2fs /dev/sda1

it’ll resize the drive and you should be good to go.

Share/Save

So here is how I did it:

1) Start Up an Instance

The first thing I do is start up an instance.  Everyone has instructions on how to do this from the command line.  But I’m lazy and find that Amazon’s API tools change regularly which is annoying and makes it difficult to follow the instructions other people have written.  I recommend doing it in the AWS Console.

Go to EC2 in the AWS Console.  Choose AMI and find an image that you like (I tend to lean toward Alestic’s Ubuntu images, but that’s just me).  Once I found an AMI I like I highlight it and then choose “Launch Instance”.  I follow the instructions, making sure that I choose the correct security group (aka firewall) and the correct key pair (you access ec2 servers using a key pair rather than a password).

Once the server is up and running, I update everything, configure it, transfer data onto it, blah blah blah.  The usual things that you do when you are prepping a server.  Once I’m all set I’m ready to make it an EBS backed AMI.

2) Create a New EBS Volume

The next thing I do is create an EBS Volume, again doing it in the AWS Console.  Go to the EC2 section of the AWS Console.  Choose “Volumes” and then “Create Volume”.  You’ll want to make sure of two things:

First that the volume is large enough to hold everything.  I forgot to do that the first time around and *thought* I had to do everything all over again (see next post for how to really do it).  The easiest thing would be to just size it right the first time around.

Second that the volume is in the same availability zone as the instance you started up in step 1.  If you aren’t sure what the availability zone is for your instance, go back to the instance, highlight it, and in the bottom panel you’ll see what zone its in.

3) Attach Volume to Instance

When you are done you’ll want to attach the volume to the instance that you created in step 1.  Once the volume is available, highlight it and choose “Attach Volume”.  You’ll be stepped through attaching it to your server.  If you have only one instance running its pretty straightforward.  If you have multiple ones running, you’ll want to make sure that you’re attaching it to the right place.  You’ll be asked to indicate where you want the attached, I usually do /dev/sdf.  Oh and this process won’t work if the volume and the instance aren’t in the same availability zone, hence my warning in the previous step.

4) Copy the Data from the Server to the Volume

This part of the instructions I found over at Winners Don’t Lose.  This helps you format the volume and copy the data on the current instance over to the volume:

In a root shell on the instance:

# mkfs.ext3 /dev/sdf
# mkdir /mnt/target && mount /dev/sdf /mnt/target
# rsync -avHx / /mnt/target
# rsync -avHx /dev /mnt/target
# sync;sync;sync;sync && umount /mnt/target

The above commands did the following:

formatted the entire volume /dev/sdf as an extended 3 filesystem
created directory /mnt/target and mounted /dev/sdf at /mnt/target
rsync’d the root instance-store filesystem to the ebs volume
synchronized the /dev directory from the instance-store filesystem
flush all pending write ops, and unmount the EBS volume

To be honest these are the only instructions I’ve ever been able to follow that doesn’t produce some error that I have to debug.

5) Detach the Volume

The next step is to detach the volume that you just created from the running instance.  Again you can do this in the AWS console by finding the volume, highlighting it, and choosing detach.  Make sure the you’re detaching the right volume though!

6) Create a Snapshot of the Volume

You’ll want to create a snapshot of the volume that you just created.  You can again do that in the AWS Management console but finding the volume, right clicking on it, and choosing create snapshot.

7) Register your AMI

This is the part that took me forever to figure out.  Mostly because the documentation for the ec2-register command didn’t jive with the tools that i had access too.  The command I used was:

ec2-register --block-device-mapping /dev/sda1=<snapshot id>::true
--name <ami name> --kernel <kernel id> --ramdisk <ram disk id>
-K <your pk file>.pem -C <your cert file>.pem

(note i updated this:  if you want the volume to persist after termination then you’ll want to change the true in the –block-device-mapping flag to false)

You’ll want to make sure that you specify and kernel id and a ram disk id, I usually use the same one that my instance I’m creating this new AMI from was using.  You don’t have to do it, but that would mean that if your AMI users don’t choose one during start up, then the instance won’t work.

Note that you may have your key and cert file set as part of your environment variable.  I have multiple accounts so I don’t, and instead just specify them when I’m running any commands (annoying but I don’t do much from the commandline anyway).

Share/Save

But now I’m getting back to where I wanted to be.  I have accepted a job just outside of Washington, DC for a web consulting firm called Viget.  I’ll be one of many project managers for the company and I’m excited that my love of organizing projects will finally find a real creative outlet (hey organization can be creative).

I’ll be sad to leave behind my friends here in Boston, but I’m excited to be moving closer to some really great ones in Washington, DC.  I’ll be starting at the end of April, so wish me luck as I try to find an apartment, pack everything up, and move back to the city I love.

Share/Save

The install process wasn’t so easy.  To be fair there are instructions for installing WebWork on Ubuntu 10.04 but if you deviate from the instructions you have to start all over again.  I learned that when I tried to install it in the /var/www directory rather than the /opt directory it asks for.  Luckily a friend had warned me, so I only made that mistake once.

We’ve made the decision to migrate from the MAA (which hosts WebWork if you have under 100 users) to our own installation.  Unfortunately, we’re doing it this week and I just got the archives of the courses last week, so I’m in a mad dash to figure out how to do all of this.  And of course, in true Rosalyn format, I’m going to write a blog post on the topic.  Hopefully these instructions will come in handy this summer as I move the rest of the courses to the WebWork server and figure out LDAP authentication for the system.  So here it goes, I know how much you all love this:

1) Determine what Archives I need to Migrate

So when I requested the archives from the folks at MAA they sent all of our courses, even older ones.  But nothing in the course name indicated what semester these archives were from.  So I needed to figure it out on my own.  After extracting one course, I noticed there was a log directory with two files, one that talked about when people submitted answers and one that showed the last time users logged in.  Beautiful.  I decided to go with the answers log, since I felt that would be a more accurate portrayal of when the course was from.  I extracted each one separately and just took a peek at the last entry.

tar -xvf {tarball.tar.gz} {path/to/file}

Turns out there were only three current courses that needed to be migrated.  And then luckily one professor said they would continue using the MAA server until the end of the semester (and option we offered them).  Yay.  Less work for me.

2) Import the Course

So the first step is to move the archives to the server.  Now in order to unarchive the course, they need to be in the /opt/webwork/courses folder.  I sftped the files to my server, and then went into the admin interface and clicked unarchive course and followed the prompts.

WebWork Unarchive Course

When all was said and done I couldn’t log into the course.  This was the moment that everything that I thought was slightly off about the software came to make sense.

3) Change the Admin Password

So each course gets their own tables in the webwork database:

• coursename_key
• coursename_password
• coursename_permission
• coursename_problem
• coursename_problem_user
• coursename_set
• coursename_set_locations
• coursename_set_locations_user
• coursename_set_user
• coursename_setting
• coursename_user

No you’re reading that right.  There isn’t a global table for users and passwords, there are user and password tables for each course.  I know, I know…my normal form loving brain hurts thinking about it too.  But I guess it makes it easier to maintain each course??? (I’m really trying to be an optimist)

Since I don’t know the password for the admin user I realized I needed to change the password if I wanted to be able to get into the imported courses.  So I went into the database on my server and ran a select query so I could find the hashed up password of an account where I knew the actual password.

select * from coursename_password;

For some added annoyance WebWork can use spaces and other characters that you wouldn’t normally include in a MySQL table name.  I have actually never run across this before so I wasn’t sure what to do.  I tried containing the table names in single quotes and double quotes.  Turns out you need to use the accent mark located below the tilde on your keyboard.  Yay for trial and error.

So once I got the hashed up password of an account I knew the password for, I updated the table for the newly imported course in WebWork:

update `coursename_password`
set password = "<newhashedpasswordhere>" where user_id = "admin";

Once you’ve done that you should be able to log in to the course as the admin user.

Share/Save

1) install apache

in order to view the reports that munin installs you’ll need to install a web server.  i, like most of the universe, use apache, so that’s what i’m going to do here.

sudo apt-get install apache2

this will install apache2 and start it up for you.  when you’re done go to the server and you’ll see this page:

The apache default page after a simple install.

fairly simple really.  nothing major and there are tons of instructions all over the interwebs that can get you to this point.

2) install postfix

you’ll also need to install a mail server if you want Munin to email you when things go into warning mode and what not.

sudo apt-get install postfix bsd-mailx

this will install postfix and the mail command (so you can test to make sure postfix is working).  as you’re installing postfix you’ll be asked to set up postfix.  first you’ll be asked to choose the mail server configuration.

The prompts for installing postfix on Ubuntu 10.04

choose internet site which will allow you to send outgoing mail from your server.  on the next screen insert the server’s web address if it is not already filled in.  if you screw up you can reconfigure postfix with the following command

sudo dpkg-reconfigure postfix

once you’ve got that set up you’ll want to test to see if you can send email to an outside email address.  you can do this by typing the following:

mail your@emailaddress.com
Subject:  your subject here
body of the email here
.

the period will end the email and send it on its way.  before doing this i usually tail the mail.log located in /var/log and watch the email get sent on its way.  watch out because it may end up in your spam folder rather than your inbox (just an fyi).

3) install munin munin-node

next you’ll want to install munin, which does the monitoring, and munin-node, which goes on any server that you want monitored.  you can do this again with apt-get

sudo apt-get install munin munin-node

3.1) email notification

if you would like to have munin notify you when something moves from OK to another state (or back again) you’ll want to set up email notifications.  you’ll need to do this by editting the /etc/munin/munin.conf file.

vi /etc/munin/munin.conf

and insert the following lines according to the instructions in the comments

# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm

contact.yourname.command mail -s "Munin notification" your@email.com

when you’re done you’ll want to start the munin-node and restart apache (just to be safe) by running the following commands

sudo start munin-node
sudo apache2ctl graceful

when you’re done go to the web address of the server and you’ll see this:

After the initial install of Munin you'll see the Forbidden error.

so I saw that and assumed, oh its clearly a permissions error.  maybe the munin user that got created on install doesn’t have access to the web server.  yeah no, i was wrong.

4) edit /etc/munin/apache.conf

so with a fresh install of munin you can only access the information from the server that does the monitoring.  so you’ll want to go in and edit the apache.conf file:

sudo vi /etc/munin/apache.conf

and change the following line:

Allow from 127.0.0.1 ::1

into:

Allow from all

you can of course allow it from only particular ip addresses, or you can set up an .htpasswd file so that the directory is locked down.  however you want to do it is up to you, but you probably shouldn’t be pushing this info out the world.  in either case when you’re done you’ll want to restart apache.

sudo apache2ctl graceful

When all is said and done you’ll see this:

and then of course you can click on your server and look at all the pretty charts and graphs.

Share/Save

So what do these three things have in common?  Each one is being hosted with Slicehost.

Slicehost is virtual private hosting and a subsidiary of cloud computing provider Rackspace.  Essentially all of their servers are in Rackspace’s cloud.  Slicehost allows you to provision servers, schedule backups, create snapshots, clone servers, etc.  But rather than being charged per hour, you’re charged a flat monthly fee (which can go up if you go over your bandwidth allowance).

So just like any best friend there are things about it that I like and thinks about it I don’t like.  I like how easy it is to clone a server or create a back up.  So if I want to upgrade WordPress for example, rather than dumping the MySQL database, backing up the files and then upgrading, I can take a snapshot of the server and then upgrade knowing that I have an exact replica of the system.  I also like how easy it is to do development work.  If there is a bug fix or something that I want to try out, I can clone the server, work on the changes, and know that it’ll work on my production instance (because essentially I’m using my production instance).

I speak so glowingly of it, what criticisms could I have?  Well what would be great is if I could have the best of both worlds.  I wish that in addition to paying a flat monthly fee, I could have the option to provision for just a few hours.  Maybe I could log into Rackspace’s cloud, see my Slicehost servers and clone them there.  This would help me with handling data loss and working on smaller more urgent bug fixes.  Maybe I’ll get lucky, they’ll read this, and figure out how to make all my dreams come true.

Share/Save

Visual depiction of the rotating Moodle architecture

I have a plan to back up Wheaton’s Learning Management System (LMS) to the cloud on a yearly basis. I talked about the basics of this architecture in my post I <3 Redmine and You Will Too.Essentially each academic year I will rotate a years worth of Moodle into the cloud, only keeping the current year and the previous year.  Here’s a picture to give you a little help in understanding.

So now I need to find a way for people that are not me to access the Moodle that lives in the cloud.  The problem is that every moodle has a config file, and in that config file, Moodle asks you to specify the URL for where your Moodle is located, this is slightly problematic because:

1. Amazon EC2 servers are not given IP addresses, they are given a Public DNS like
2. Technically they can be assigned to an IP address but Elastic IPs cost money when they aren’t being used, and frankly I would rather not have to pay extra if I don’t have to.
3. The people accessing these Moodles can’t be expected to ssh into a server and change the config file.

So I need to make this happen automagically when the server starts up.  I tried using the php variable $_SERVER but I got some weird things.  So my idea is to have a start up script call the public-hostname (using the API amazon provides) and then edit the config file.  That way when the server starts up, Moodle will automagically be available.

Share/Save

I never argued *against* PHP/Python/Perl or even use of Linux/Apache by PaaS vendor. Just noted PaaS users *shouldn’t* care.

I apologized and asked for clarification since it seemed to me that there was an argument about whether or not a PHP/Perl cloud was necessary.

I re-read the article a few times, and on the last go around the only new explanations I could see from it was:

1. if i wrote in php a vendor would deploy in another language
2. he meant that the LAM in LAMP was what didn’t matter  (although that didn’t seem to be explicitly stated)

In the meantime though, I’ll just wait for a response (I sent it late last night and people tend to sleep) rather than possibly misinterpret more.

Share/Save

James’ last question: “Is the ‘open sourceness’ of a programming stack even that important anymore?” is a good one, but orthogonal to the discussion about a LAMP/PHP cloud, in my mind.

So why has this conversation gotten me in a tizzy and forced me to write two blog posts in one week?  Well two reasons:

1. Many web based open source projects are built on the back of P languages (PHP, Perl, and Python)
2. The cloud exists to make technology easier for the masses.

Now before I go any further I will disclose that the first language I learned was PHP.  The second one was Perl.  While I know there are wars that start on listservs about what languages are better, that is not the purpose of this post, so put it out of your head and don’t say anything about how much you think PHP sucks in the comments.

Web based open source projects are built on the back of P languages

Now I could write something about this, but I like lists.  So here is a brief off the cuff list of what open source projects use P languages:

1. MediaWiki (PHP)
2. WordPress (PHP)
3. Bugzilla (Perl)
4. Drupal (PHP)
5. Plone (Python)
6. Moodle (PHP)
7. AWStats (Perl)
8. MovableType (PHP and Perl)
9. Subversion (Python and C)
10. Trac (Python)

Now, I’m not going to argue the merits of how good each of these piece of software are. Again that’s not my point.  But they are popular whether you like it or not.  And to just dismiss them would be foolish.

The cloud exists to make technology easier for the masses

Google Docs makes it easier to get to files you need to edit.  Heroku makes it easier to deploy Ruby on Rails apps.  Amazon EC2 makes it easier for you to deploy a server.  The fall out of the cloud is that technology suddenly becomes much easier to create and deploy.

So what happens when my mom wants to write her own blog.  Well, most likely she heads over to WordPress.com and creates a blog for herself.  And if she needs a little more, then she signs up for the WordPress Premium services.  But what if I want my own blog.  Well right now I have my friend hosting my stuff.  But what if I want it hosted elsewhere and I don’t want to maintain servers (since I do that all day long).  My only option right now is to do what my mom is doing, but the reality is, that’s not what I want.  I want something like Heroku but for PHP.  And if I had it, I could install any of the PHP applications listed above.

And I think this conversation is very pertinent given that Google just announced its open source learning management system written in Python and intended to be deployed in Google App Engine.  This could potentially be a big blow to Moodle who doesn’t have a cloud to turn to.

Conclusion

So yes a PHP based cloud does matter.  And yes it should be built because there are lots of people clamoring to use it (or who have already figured out a way).

And yes this blog post could be more coherent.

Share/Save

Visual depiction of the rotating Moodle architecture

I’m in the process of setting up an architecture for Moodle. Essentially using VMWare I’ll rotate out servers every academic year.  Older server images will be put in the cloud so that if we need access to them quickly, we can do that without any problems.  This will allow me to:

1) get away with not having to do necessary MySQL tuning since Moodle doesn’t really delete much from its database.

2) not have to deal with compatibility issues since Moodle has been know to create course back ups that don’t work with future versions of the software.

3) deal with file storage issues, namely that users often thing the course management software is the place to store all of your files for ever and ever (its not by the way).

So now that I’ve come up with that I need a way to figure out what I have to do to make this work and then hope that I don’t forget those tasks.  Enter Redmine.  If you haven’t used it before, I recommend checking it out.  There is a demo redmine you can use to play around with it.

tasks that need to be accomplished in order to set up the Moodle architecture.

What I like about it is its flexibility.  At MPOW I am essentially a developer, systems administrator, and project manager all rolled up into one.  Its a lot of hats to wear, and often times I forget tasks, or I need to talk out the entire process.  Essentially I use Redmine to help me with that.  I can enter new features for the developer Rosalyn to work on.  I can enter tasks for sys admin Rosalyn to work on.  And all of this is done by project manager Rosalyn who is trying to help the sys admin and developer work together (yes i do talk to myself…and answer back).

Redmine has subtasking, subprojects, and you can change all kinds of things in it so that it can essentially function however you want it to.  And its written in Ruby on Rails so extending is easy (although the plugins are plentiful so you may not even need to write your own code).  I’ve gotten my boss on board with it, and he’s slowly trying to infect others so that they use it too.

So that’s my new best friend Redmine.  It has helped me organize and focus my day on the things that need to get done.  I <3 it and if you try it out, I think you will too.

Share/Save