Visual depiction of the rotating Moodle architecture

I have a plan to back up Wheaton’s Learning Management System (LMS) to the cloud on a yearly basis. I talked about the basics of this architecture in my post I <3 Redmine and You Will Too.Essentially each academic year I will rotate a years worth of Moodle into the cloud, only keeping the current year and the previous year.  Here’s a picture to give you a little help in understanding.

So now I need to find a way for people that are not me to access the Moodle that lives in the cloud.  The problem is that every moodle has a config file, and in that config file, Moodle asks you to specify the URL for where your Moodle is located, this is slightly problematic because:

1. Amazon EC2 servers are not given IP addresses, they are given a Public DNS like
2. Technically they can be assigned to an IP address but Elastic IPs cost money when they aren’t being used, and frankly I would rather not have to pay extra if I don’t have to.
3. The people accessing these Moodles can’t be expected to ssh into a server and change the config file.

So I need to make this happen automagically when the server starts up.  I tried using the php variable $_SERVER but I got some weird things.  So my idea is to have a start up script call the public-hostname (using the API amazon provides) and then edit the config file.  That way when the server starts up, Moodle will automagically be available.

Share/Save

James’ last question: “Is the ‘open sourceness’ of a programming stack even that important anymore?” is a good one, but orthogonal to the discussion about a LAMP/PHP cloud, in my mind.

So why has this conversation gotten me in a tizzy and forced me to write two blog posts in one week?  Well two reasons:

1. Many web based open source projects are built on the back of P languages (PHP, Perl, and Python)
2. The cloud exists to make technology easier for the masses.

Now before I go any further I will disclose that the first language I learned was PHP.  The second one was Perl.  While I know there are wars that start on listservs about what languages are better, that is not the purpose of this post, so put it out of your head and don’t say anything about how much you think PHP sucks in the comments.

Web based open source projects are built on the back of P languages

Now I could write something about this, but I like lists.  So here is a brief off the cuff list of what open source projects use P languages:

1. MediaWiki (PHP)
2. WordPress (PHP)
3. Bugzilla (Perl)
4. Drupal (PHP)
5. Plone (Python)
6. Moodle (PHP)
7. AWStats (Perl)
8. MovableType (PHP and Perl)
9. Subversion (Python and C)
10. Trac (Python)

Now, I’m not going to argue the merits of how good each of these piece of software are. Again that’s not my point.  But they are popular whether you like it or not.  And to just dismiss them would be foolish.

The cloud exists to make technology easier for the masses

Google Docs makes it easier to get to files you need to edit.  Heroku makes it easier to deploy Ruby on Rails apps.  Amazon EC2 makes it easier for you to deploy a server.  The fall out of the cloud is that technology suddenly becomes much easier to create and deploy.

So what happens when my mom wants to write her own blog.  Well, most likely she heads over to WordPress.com and creates a blog for herself.  And if she needs a little more, then she signs up for the WordPress Premium services.  But what if I want my own blog.  Well right now I have my friend hosting my stuff.  But what if I want it hosted elsewhere and I don’t want to maintain servers (since I do that all day long).  My only option right now is to do what my mom is doing, but the reality is, that’s not what I want.  I want something like Heroku but for PHP.  And if I had it, I could install any of the PHP applications listed above.

And I think this conversation is very pertinent given that Google just announced its open source learning management system written in Python and intended to be deployed in Google App Engine.  This could potentially be a big blow to Moodle who doesn’t have a cloud to turn to.

Conclusion

So yes a PHP based cloud does matter.  And yes it should be built because there are lots of people clamoring to use it (or who have already figured out a way).

And yes this blog post could be more coherent.

Share/Save

Visual depiction of the rotating Moodle architecture

I’m in the process of setting up an architecture for Moodle. Essentially using VMWare I’ll rotate out servers every academic year.  Older server images will be put in the cloud so that if we need access to them quickly, we can do that without any problems.  This will allow me to:

1) get away with not having to do necessary MySQL tuning since Moodle doesn’t really delete much from its database.

2) not have to deal with compatibility issues since Moodle has been know to create course back ups that don’t work with future versions of the software.

3) deal with file storage issues, namely that users often thing the course management software is the place to store all of your files for ever and ever (its not by the way).

So now that I’ve come up with that I need a way to figure out what I have to do to make this work and then hope that I don’t forget those tasks.  Enter Redmine.  If you haven’t used it before, I recommend checking it out.  There is a demo redmine you can use to play around with it.

tasks that need to be accomplished in order to set up the Moodle architecture.

What I like about it is its flexibility.  At MPOW I am essentially a developer, systems administrator, and project manager all rolled up into one.  Its a lot of hats to wear, and often times I forget tasks, or I need to talk out the entire process.  Essentially I use Redmine to help me with that.  I can enter new features for the developer Rosalyn to work on.  I can enter tasks for sys admin Rosalyn to work on.  And all of this is done by project manager Rosalyn who is trying to help the sys admin and developer work together (yes i do talk to myself…and answer back).

Redmine has subtasking, subprojects, and you can change all kinds of things in it so that it can essentially function however you want it to.  And its written in Ruby on Rails so extending is easy (although the plugins are plentiful so you may not even need to write your own code).  I’ve gotten my boss on board with it, and he’s slowly trying to infect others so that they use it too.

So that’s my new best friend Redmine.  It has helped me organize and focus my day on the things that need to get done.  I <3 it and if you try it out, I think you will too.

Share/Save

So I got my text message at 12:30am.  It confused me but then I woke up a bit.  It was a good thing I wrote the notes I did in my previous post, because trust me, I could have never come up with that stuff on my own right now.

First thing I did was run top.  CPU was at 100.1% (way to work hard server) for the mysqld process.  Second thing I did was check the slow query log.  There were actually a few slow queries, the first ones are still puzzling me.  Basically these were checking a table that had a whole lot of rows of data, the data didn’t need to be stored in those tables, i removed the data, and I stopped logging that data in the tables.  I probably won’t see those queries ever again. But this doesn’t really matter, those queries weren’t the ones hanging up the system, they were only take 11 t0 17 seconds to run.

I did see the last query that ran though, and this seems to be part of the problem, but not the query that was actually causing mysqld to hang. This query was:

INSERT INTO stats_daily (stattype, timeend, courseid, roleid, stat1, stat2)

SELECT ‘enrolments’, timeend, courseid, roleid, COUNT(DISTINCT userid), 0
FROM (
SELECT 1251691200 AS timeend, pl.courseid, pl.roleid, pl.userid
FROM (
SELECT DISTINCT ra.roleid, ra.userid, c.id as courseid
FROM role_assignments ra JOIN context ctx
ON ctx.id = ra.contextid
CROSS JOIN course c
JOIN role_capabilities rc
ON rc.roleid = ra.roleid LEFT OUTER JOIN course_categories cc1
ON cc1.id = c.category
WHERE ((rc.capability = ‘moodle/course:view’ )
AND rc.permission = 1 AND rc.contextid = 1
AND (ctx.contextlevel = 10
OR (c.id = ctx.instanceid AND ctx.contextlevel = 50) OR (cc1.id = ctx.instanceid AND ctx.contextlevel = 40)))
) pl
) inline_view
GROUP BY timeend, courseid, roleid;

This query took a total of 41 seconds to execute.  I’m assuming that the next query after this was the one that brought the house down.  I saw the beginning of this when I went into mysql and did a show processlist:

+——-+————+———–+——–+———+——+————–+——————————————————————————————————+
| Id    | User       | Host      | db     | Command | Time | State        | Info                                                                                                 |
+——-+————+———–+——–+———+——+————–+——————————————————————————————————+
| 45654 | moodleuser | localhost | moodle | Query   | 1776 | Sending data | UPDATE stats_daily
SET stat2 = (SELECT COUNT(DISTINCT ra.userid)
|

The full query finally showed up in my query log once I killed the process.  Here it is in case you’re curious:

090902  0:01:29      45654 Query       UPDATE stats_daily
SET stat2 = (SELECT COUNT(DISTINCT ra.userid)
FROM role_assignments ra JOIN context ctx
ON ctx.id = ra.contextid
CROSS JOIN course c
JOIN role_capabilities rc
ON rc.roleid = ra.roleid LEFT OUTER JOIN course_categories cc1
ON cc1.id = c.category
WHERE ra.roleid = stats_daily.roleid AND
c.id = stats_daily.courseid AND
((rc.capability = ‘moodle/course:view’ )
AND rc.permission = 1 AND rc.contextid = 1
AND (ctx.contextlevel = 10
OR (c.id = ctx.instanceid AND ctx.contextlevel = 50) OR (cc1.id = ctx.instanceid AND ctx.contextlevel = 40))) AND
EXISTS (SELECT ‘x’
FROM log l
WHERE l.course = stats_daily.courseid AND
l.userid = ra.userid AND l.time >= 1251604800  AND l.time  < 1251691200))
WHERE stats_daily.stattype = ‘enrolments’ AND
stats_daily.timeend = 1251691200 AND
stats_daily.courseid IN
(SELECT DISTINCT l.course
FROM log l
WHERE l.time >= 1251604800  AND l.time  < 1251691200)

I also saw a whole bunch of other queries locked, waiting for this query to run.  I thought that perhaps switching the tables over to InnoDB rather than MyISAM might help alleviate the problem, but the truth is this appears to be a “the query is just ridiculous” problem rather than a “InnoDB is better at performance” problem.

So at anyrate I looked around and it seems this is a problem with the stats that Moodle keeps about itself.  Apparently in version 1.6 people had this problem a lot (because stats was new).  Basically I enabled stats because someone thought it was a good idea and I didn’t bother to consider how resource intensive this process could be (note to self: be a better sys admin and read documentation when someone thinks something might be useful).  I changed some settings in the stats configuration page and hope that this reduces the load and processing time of the stats query.  We’ll see if this fixes my problem.

Also, the amount of time I spent on this right now totally makes up for leaving work an hour and a half early to go meet my brother in law the electrician so he could replace outlets and light fixtures (my $12 sconces from lowes look quite lovely).

Share/Save

Recently Nagios started informing me that HTTPS was going down infrequently (but started to become more frequent).

***** Nagios  *****
Notification Type: PROBLEM
Service: HTTPS
Host: Moodle Server
Address: 123.456.78.912
State: CRITICAL
Date/Time: Fri Aug 28 01:06:50 EDT 2009
Additional Info:
CRITICAL – Socket timeout after 10 seconds

What was strange was that HTTP wasn’t going down.  I felt like Nagios was lying to me so I figured I would have to wake up late at night (early in the morning?) and catch it lying to me.  To do this I decided I’d try to figure out if my cellphone had an email to associate to text messaging.  Mine wasn’t working so I had to talk to Sprint about it.

In the mean time though, it was still going down.  Luckily last night at 4am I randomly woke up. I ran downstairs (imagine kid at Christmas…but like a reverse Christmas) and fired up the laptop.  Sure enough when I went to the Moodle homepage I saw this:

Error: Database connection failed.

It is possible that the database is overloaded or otherwise not running properly.

The site administrator should also check that the database details have been correctly specified in config.php

Now in hindsight, there were so many things I could have done at this point.  But it was 4am and my house was freezing so I made some mistakes.  The first thing I did was restart Apache.  It appeared to be running and the restart didn’t throw any errors, but it didn’t fix the problem either.  Next I “turned off SSL”.  I was able to get to the error page without the HTTPS so I determined at this point that Nagios clearly was lying to me, something else was wrong (p.s. i am thankful to Nagios for telling me that something was wrong).  So then I decided that maybe it was a database problem.  I went into MySQL and was able to access the Moodle database and do a “show tables;”.  So I thought at that point I should restart MySQL, in hindsight there are so many other things I should have done first, but I will document those below.  A restart of MySQL fixed the issue.

So what will I do next time (and yes…there will be a next time):

1) run top on the server: See if MySQL is hogging the CPU.  This could lead you to a number of options.

2) check the mysql-slow-queries.log: I set up the mysql option log-slow-queries.  this in conjunction with the long-query-time option will log any query that takes over a certain amount of time (default is 10s).

3) show mysql processlist: If you haven’t set up the log slow queries option this might also be helpful in determining what mysql is doing.  Here is a brief explaination of show processlist and its companion of kill <processid>.  I also found this handy script that you could set up as part of a cron to kill any runaway processes.

4) query is poorly written: I’ve found a number of reports that MySQL queries are poorly written in Moodle and that these can cause the CPU usage to top out at %100 which causes Moodle to be unusable.  Below are a few Moodle Tracker reports of this:

• http://tracker.moodle.org/browse/MDL-14911
• http://tracker.moodle.org/browse/MDL-19584
• http://tracker.moodle.org/browse/MDL-17051
• http://tracker.moodle.org/browse/MDL-10848

To see if this is the case with your Moodle use the slow query logging (or just the plain old query logging) with a backup of your database to see if you can replicate the problem somewhere else.  If you can the problem is probably the query and maybe you can figure out a way to rewrite the query (or get it rewritten) to see if that fixes the problem.

5) check your persistent db connection: Check to see if you have persistent database connections set up in the Moodle config.php file and in the php.ini file.  If you do, turn it off and see if this solves the problem.  it may be that your connections aren’t closing and they are killing your database regularly.

6) check your max connections: Are the maximum number of Apache connections greater than your maximum number of MySQL connections?  If so that might be a problem.  Basically users could get to the website but then when they go to request something in the database they can’t because the # of connections are taken up.  I have a feeling though that this may be a completely different error than what I was experiencing (people…including myself…really need to take screenshots or document errors better).

7) check your MySQL performance: Perhaps you haven’t configured Moodle right.  Take a look at these performance tuning tips to see if you can get it working better.

8) check your database tables: Stop mysqld and run a check on your Moodle tables.  See if some of the tables are corrupted, if they are fix them.

So if you’re wondering what I’m doing at 4am tomorrow, there’s the list.  Wish me luck.

Share/Save

We have a bunch of placement tests and advising courses and that sort of thing we want our first year students to be enrolled in.  Moodle happens to have a flat file enrollment feature.  Below are the instructions from that enrollment feature:

This method will repeatedly check for and process a specially-formatted text file in the location that you specify. The file is a comma separated file assumed to have four or six fields per line:

*  operation, role, idnumber(user), idnumber(course) [, starttime, endtime]
where:
*  operation        = add | del
*  role             = student | teacher | teacheredit
*  idnumber(user)   = idnumber in the user table NB not id
*  idnumber(course) = idnumber in the course table NB not id
*  starttime        = start time (in seconds since epoch) - optional
*  endtime          = end time (in seconds since epoch) - optional

It could look something like this:

   add, student, 5, CF101
   add, teacher, 6, CF101
   add, teacheredit, 7, CF101
   del, student, 8, CF101
   del, student, 17, CF101
   add, student, 21, CF101, 1091115000, 1091215000

So great, I need to pull a randomly generated number rather than a username in order to enroll students.  So now I need to figure out a way to take a list of usernames in banner and get their corresponding idnumbers in Moodle.

Now if this were a one time thing I would take the lazy man’s way out and I would create a table in the database and do a simple

select table.username, user.idnumber
from table, user
where table.username = user.username;

But I’m going to have to do this every year until we either get rid of Moodle or I leave Wheaton.  Since plans are not in the work for either, I’ll actually do something more substantive.

First I made the decision to use the MySQL select in clause.  For those of you who don’t know, you can do:

select username, idnumber
from user
where username in ('value1', 'value2', 'valueN');

Its pretty useful really.  I can’t really find a MySQL page that talks about the in() clause but I did find a good post in the MySQL lists.  The next thing I had to figure out was how to push a file into a php array.  Again I found a great post on using the file() command to do this.  The only drawback is that when i go to print it’ll just show:

value1 value2 valueN

When that’s not what I want.  I need quotes and commas and crap around it.  So I found another great post that talks about using the implode function to put a php array into an in() clause.  The implode function will allow me to put quotes and what not around each one of the values in the array so it will look like this:

'value1', 'value2', 'valueN'

So now that I have the component parts, all I need to do is create a php page that will let me upload a file, run the query, and retrieve the outfile.  Of course the idea of writing this code makes me tired and since its 5pm I shall just publish this post and write the code tomorrow (hopefully).

UPDATE

I was wrong.  The value is not a randomly generated number, well it is, but its randomly generated by Banner.  In the Banner XML the value is the <sourcedid> for a <person>.

Luckily this randomly generated value is something that we can pull out of Banner using our reporting system.  The value (in our Banner) is gobsrid_sourced_id.  Not sure if it will be the same in your Banner, but hopefully this will put someone on the right track.

My next step is to document how to do all of this in our Moodle Documentation Wiki.  I might do a Jing Video.

Share/Save

NITLE hosts a Moodle Exchange (NME).

For faculty, technologists, and librarians at participating campuses that use or are interested in using Moodle. Participants in this collaborative user community focus on the use of Moodle at smaller, independent colleges and universities, sharing ideas and best practices and helping each other troubleshoot technical or use issues specific to liberal education and undergraduate colleges and universities. In the NME, Moodle-using or Moodle-interested campuses share strategies and ask and answer questions about implementation, evaluation, migration, customization, and other Moodle-related topics.

NME also works together to develop a Liberal Arts Edition of Moodle as well as plugins specifically focused on the needs of Liberal Arts Colleges.  So how does this get cloudy?  Well they use Google Code to distribute their developments.  This allows everyone involved in the project to access or share code that has been developed for the community.

If you’re doing something similar let me know about it.  I’m very interested in how different academic institutions are using the cloud to develop systems and applications.

Share/Save

1. GD Library — By default the GD library’s aren’t part of the PHP package in Server 10.5.  I was poking around in this Step by Step Installation guide trying to upgrade my PHP library on my server.  Turns out I didn’t need to for various reasons (also the package recommended screwed with my Apache).  So instead i went into my php.ini file and uncommented the line:
   

   extension=php_gd2.dll

   This turned on the GD Library and Moodle looks much better now.

2. MySQL — I was getting a weird error in Moodle under Admin > Server > Environment.
   

   It is required that you store all your data in Unicode format (UTF-8). New installations must be performed into databases that have their default character set as Unicode. If you are upgrading, you should perform the UTF-8 migration process (see the Admin page).

   I thought that was a bit weird cause I installed the database from scratch. But sometimes you just need to do what it tells you to. So I ran the following query:

   alter database moodle collate = utf8_unicode_ci;

   And voila the error went away.

So now my Moodle is a bit happier.  I am of course (like always) recording this so that when I get my new server and I run into these problems again, I can remember what the heck I did.

Share/Save

Documenting it here for future reference.

Share/Save

I know, I know, my blog’s tag cloud is becoming Moodle heavy. Its true. But the boss man thinks that students shouldn’t be able to enroll themselves into any Moodle class at the drop of a hat. Sadly this makes sense (professors can be very protective of their class information because often times it informs their research). Sigh. So I’m off to make this happen.

So I’ve come up with a few ideas:

1. I can just run a MySQL query every month:
   

   update course set enrollable = 0;

   While yes this will work, how much of a pain will that be. I guess I could create a little script to do it for me. But honestly that’s not a solution that’s a patch.

2. I can try going into moodle/course/edit_form.php and find the line:
   

   $mform->setDefault('enrollable', 1);

   And change it to:

   $mform->setDefault('enrollable', 0);

   Wow Rosalyn you’re genius! Yeah, no, I’m not I just found that in the Moodle Enrollment Plugins forum. A guy named Michael Skwara came up with it.  So I tried it out.  Unfortunately it doesn’t effect (or is that affect) courses I create using the LMB Enrollment Plugin.  You can still enroll yourself into those.  Darn it!  Foiled again.

3. Just change the database schema (oh yeah rosalyn…like that’s just so easy).  I took a look in MySQL and the default for that column is set to 1, so if I set it to 0 perhaps that will make all new courses created with LMB default to not enrollable.  Of course I’m assuming that LMB doesn’t really update things like the enrollable field in the course table (I mean where would it get that information).  So I set about testing my theory.
   First things first, I asked in the forum I referenced above if anyone saw a problem with what I was about to do before I did it. One of the Moodlers, Iñaki Arenaza, said that there shouldn’t be any problem with this. Which is what I suspected, unless of course Moodle goes crazy and revamps their DB schema…which would cause me other headaches.So on to my next portion of the testing, does LMB update the field enrollable or does it just leave it alone. So I installed a fresh Moodle (including the LMB plugin).  From there I logged into the MySQL and changed the default value for the column enrollable by running the query:

   alter table course alter column enrollable set default 0;

   Your query might be different if you are using a prefix for your Moodle tables. I then ran an import using the LMB plug in. After that had finished I logged in using LDAP and tried to enroll myself in a course. And lo and behold I received the message:

   This course is not enrollable at the moment.

   SUCCESS!  So students who aren’t attached to the course in Banner, can’t attach themselves to the course in Moodle.  Professors however can still go in and add students (or delete students) from a course the way they did before.

So that’s my foyer into the world of altering the Moodle database schema in order to make the boss man happy. My next step is to upgrade Moodle so I can take advantage of the fun new features.

UPDATE

I noticed a few people have visited this post recently.  While this fixed worked for me I recommend that you TEST IT OUT FIRST.

Share/Save