Ideas….

Last week Geva Perry wrote a post published on GigaOm called Who Will Build the LAMP Cloud.  In it he speculated on current cloud providers that may be interested in building a PaaS (platform as a service) offering of the traditional LAMP stack.  A few days later James Urquhart responded with his blog post entitled Does Cloud Computing Need LAMP.  In it he questions whether the Linux and Apache piece are really necessary in a PaaS offering.  He then goes on to question the usefulness of P languages (PHP, Perl, Python) and MySQL, using a comment to Geva’s post as a jumping off point.  Geva then comments back with a post on his own blog entitled Who Will Build the LAMP Cloud? And Who Cares? In it he agrees with James’ comments on not caring about the LA in LAMP, but says:

James’ last question: “Is the ‘open sourceness’ of a programming stack even that important anymore?” is a good one, but orthogonal to the discussion about a LAMP/PHP cloud, in my mind.

So why has this conversation gotten me in a tizzy and forced me to write two blog posts in one week?  Well two reasons:

1. Many web based open source projects are built on the back of P languages (PHP, Perl, and Python)
2. The cloud exists to make technology easier for the masses.

Now before I go any further I will disclose that the first language I learned was PHP.  The second one was Perl.  While I know there are wars that start on listservs about what languages are better, that is not the purpose of this post, so put it out of your head and don’t say anything about how much you think PHP sucks in the comments.

Web based open source projects are built on the back of P languages

Now I could write something about this, but I like lists.  So here is a brief off the cuff list of what open source projects use P languages:

1. MediaWiki (PHP)
2. WordPress (PHP)
3. Bugzilla (Perl)
4. Drupal (PHP)
5. Plone (Python)
6. Moodle (PHP)
7. AWStats (Perl)
8. MovableType (PHP and Perl)
9. Subversion (Python and C)
10. Trac (Python)

Now, I’m not going to argue the merits of how good each of these piece of software are. Again that’s not my point.  But they are popular whether you like it or not.  And to just dismiss them would be foolish.

The cloud exists to make technology easier for the masses

Google Docs makes it easier to get to files you need to edit.  Heroku makes it easier to deploy Ruby on Rails apps.  Amazon EC2 makes it easier for you to deploy a server.  The fall out of the cloud is that technology suddenly becomes much easier to create and deploy.

So what happens when my mom wants to write her own blog.  Well, most likely she heads over to WordPress.com and creates a blog for herself.  And if she needs a little more, then she signs up for the WordPress Premium services.  But what if I want my own blog.  Well right now I have my friend hosting my stuff.  But what if I want it hosted elsewhere and I don’t want to maintain servers (since I do that all day long).  My only option right now is to do what my mom is doing, but the reality is, that’s not what I want.  I want something like Heroku but for PHP.  And if I had it, I could install any of the PHP applications listed above.

And I think this conversation is very pertinent given that Google just announced its open source learning management system written in Python and intended to be deployed in Google App Engine.  This could potentially be a big blow to Moodle who doesn’t have a cloud to turn to.

Conclusion

So yes a PHP based cloud does matter.  And yes it should be built because there are lots of people clamoring to use it (or who have already figured out a way).

And yes this blog post could be more coherent.

Right now I’m insanely busy. But its the hurry up and wait kind of insanely busy stuff.

Visual depiction of the rotating Moodle architecture

I’m in the process of setting up an architecture for Moodle. Essentially using VMWare I’ll rotate out servers every academic year.  Older server images will be put in the cloud so that if we need access to them quickly, we can do that without any problems.  This will allow me to:

1) get away with not having to do necessary MySQL tuning since Moodle doesn’t really delete much from its database.

2) not have to deal with compatibility issues since Moodle has been know to create course back ups that don’t work with future versions of the software.

3) deal with file storage issues, namely that users often thing the course management software is the place to store all of your files for ever and ever (its not by the way).

So now that I’ve come up with that I need a way to figure out what I have to do to make this work and then hope that I don’t forget those tasks.  Enter Redmine.  If you haven’t used it before, I recommend checking it out.  There is a demo redmine you can use to play around with it.

tasks that need to be accomplished in order to set up the Moodle architecture.

What I like about it is its flexibility.  At MPOW I am essentially a developer, systems administrator, and project manager all rolled up into one.  Its a lot of hats to wear, and often times I forget tasks, or I need to talk out the entire process.  Essentially I use Redmine to help me with that.  I can enter new features for the developer Rosalyn to work on.  I can enter tasks for sys admin Rosalyn to work on.  And all of this is done by project manager Rosalyn who is trying to help the sys admin and developer work together (yes i do talk to myself…and answer back).

Redmine has subtasking, subprojects, and you can change all kinds of things in it so that it can essentially function however you want it to.  And its written in Ruby on Rails so extending is easy (although the plugins are plentiful so you may not even need to write your own code).  I’ve gotten my boss on board with it, and he’s slowly trying to infect others so that they use it too.

So that’s my new best friend Redmine.  It has helped me organize and focus my day on the things that need to get done.  I <3 it and if you try it out, I think you will too.

For those of you that aren’t Systems Administrators, here is the reason you don’t want to be.  At 12:30am when things go down, you have to wake up and figure out why.  And here I am writing a blog post about it.  Luckily its not 4am.  That was totally worse (I say this now…let’s see how long it takes me to figure this stuff out).

So I got my text message at 12:30am.  It confused me but then I woke up a bit.  It was a good thing I wrote the notes I did in my previous post, because trust me, I could have never come up with that stuff on my own right now.

First thing I did was run top.  CPU was at 100.1% (way to work hard server) for the mysqld process.  Second thing I did was check the slow query log.  There were actually a few slow queries, the first ones are still puzzling me.  Basically these were checking a table that had a whole lot of rows of data, the data didn’t need to be stored in those tables, i removed the data, and I stopped logging that data in the tables.  I probably won’t see those queries ever again. But this doesn’t really matter, those queries weren’t the ones hanging up the system, they were only take 11 t0 17 seconds to run.

I did see the last query that ran though, and this seems to be part of the problem, but not the query that was actually causing mysqld to hang. This query was:

INSERT INTO stats_daily (stattype, timeend, courseid, roleid, stat1, stat2)

SELECT ‘enrolments’, timeend, courseid, roleid, COUNT(DISTINCT userid), 0
FROM (
SELECT 1251691200 AS timeend, pl.courseid, pl.roleid, pl.userid
FROM (
SELECT DISTINCT ra.roleid, ra.userid, c.id as courseid
FROM role_assignments ra JOIN context ctx
ON ctx.id = ra.contextid
CROSS JOIN course c
JOIN role_capabilities rc
ON rc.roleid = ra.roleid LEFT OUTER JOIN course_categories cc1
ON cc1.id = c.category
WHERE ((rc.capability = ‘moodle/course:view’ )
AND rc.permission = 1 AND rc.contextid = 1
AND (ctx.contextlevel = 10
OR (c.id = ctx.instanceid AND ctx.contextlevel = 50) OR (cc1.id = ctx.instanceid AND ctx.contextlevel = 40)))
) pl
) inline_view
GROUP BY timeend, courseid, roleid;

This query took a total of 41 seconds to execute.  I’m assuming that the next query after this was the one that brought the house down.  I saw the beginning of this when I went into mysql and did a show processlist:

+——-+————+———–+——–+———+——+————–+——————————————————————————————————+
| Id    | User       | Host      | db     | Command | Time | State        | Info                                                                                                 |
+——-+————+———–+——–+———+——+————–+——————————————————————————————————+
| 45654 | moodleuser | localhost | moodle | Query   | 1776 | Sending data | UPDATE stats_daily
SET stat2 = (SELECT COUNT(DISTINCT ra.userid)
|

The full query finally showed up in my query log once I killed the process.  Here it is in case you’re curious:

090902  0:01:29      45654 Query       UPDATE stats_daily
SET stat2 = (SELECT COUNT(DISTINCT ra.userid)
FROM role_assignments ra JOIN context ctx
ON ctx.id = ra.contextid
CROSS JOIN course c
JOIN role_capabilities rc
ON rc.roleid = ra.roleid LEFT OUTER JOIN course_categories cc1
ON cc1.id = c.category
WHERE ra.roleid = stats_daily.roleid AND
c.id = stats_daily.courseid AND
((rc.capability = ‘moodle/course:view’ )
AND rc.permission = 1 AND rc.contextid = 1
AND (ctx.contextlevel = 10
OR (c.id = ctx.instanceid AND ctx.contextlevel = 50) OR (cc1.id = ctx.instanceid AND ctx.contextlevel = 40))) AND
EXISTS (SELECT ‘x’
FROM log l
WHERE l.course = stats_daily.courseid AND
l.userid = ra.userid AND l.time >= 1251604800  AND l.time  < 1251691200))
WHERE stats_daily.stattype = ‘enrolments’ AND
stats_daily.timeend = 1251691200 AND
stats_daily.courseid IN
(SELECT DISTINCT l.course
FROM log l
WHERE l.time >= 1251604800  AND l.time  < 1251691200)

I also saw a whole bunch of other queries locked, waiting for this query to run.  I thought that perhaps switching the tables over to InnoDB rather than MyISAM might help alleviate the problem, but the truth is this appears to be a “the query is just ridiculous” problem rather than a “InnoDB is better at performance” problem.

So at anyrate I looked around and it seems this is a problem with the stats that Moodle keeps about itself.  Apparently in version 1.6 people had this problem a lot (because stats was new).  Basically I enabled stats because someone thought it was a good idea and I didn’t bother to consider how resource intensive this process could be (note to self: be a better sys admin and read documentation when someone thinks something might be useful).  I changed some settings in the stats configuration page and hope that this reduces the load and processing time of the stats query.  We’ll see if this fixes my problem.

Also, the amount of time I spent on this right now totally makes up for leaving work an hour and a half early to go meet my brother in law the electrician so he could replace outlets and light fixtures (my $12 sconces from lowes look quite lovely).