Moodle’s MySQL Keeps Going Down

If you’re not interested in Moodle or troubleshooting MySQL problems, than you probably don’t want to read this post.  I however wanted to write some notes for myself and need it to be portable.  Since that’s technically the purpose of this blog, I figured I should put it here.

Recently Nagios started informing me that HTTPS was going down infrequently (but started to become more frequent).

***** Nagios  *****
Notification Type: PROBLEM
Service: HTTPS
Host: Moodle Server
Address: 123.456.78.912
State: CRITICAL
Date/Time: Fri Aug 28 01:06:50 EDT 2009
Additional Info:
CRITICAL – Socket timeout after 10 seconds

What was strange was that HTTP wasn’t going down.  I felt like Nagios was lying to me so I figured I would have to wake up late at night (early in the morning?) and catch it lying to me.  To do this I decided I’d try to figure out if my cellphone had an email to associate to text messaging.  Mine wasn’t working so I had to talk to Sprint about it.

In the mean time though, it was still going down.  Luckily last night at 4am I randomly woke up. I ran downstairs (imagine kid at Christmas…but like a reverse Christmas) and fired up the laptop.  Sure enough when I went to the Moodle homepage I saw this:

Error: Database connection failed.

It is possible that the database is overloaded or otherwise not running properly.

The site administrator should also check that the database details have been correctly specified in config.php

Now in hindsight, there were so many things I could have done at this point.  But it was 4am and my house was freezing so I made some mistakes.  The first thing I did was restart Apache.  It appeared to be running and the restart didn’t throw any errors, but it didn’t fix the problem either.  Next I “turned off SSL”.  I was able to get to the error page without the HTTPS so I determined at this point that Nagios clearly was lying to me, something else was wrong (p.s. i am thankful to Nagios for telling me that something was wrong).  So then I decided that maybe it was a database problem.  I went into MySQL and was able to access the Moodle database and do a “show tables;”.  So I thought at that point I should restart MySQL, in hindsight there are so many other things I should have done first, but I will document those below.  A restart of MySQL fixed the issue.

So what will I do next time (and yes…there will be a next time):

1) run top on the server: See if MySQL is hogging the CPU.  This could lead you to a number of options.

2) check the mysql-slow-queries.log: I set up the mysql option log-slow-queries.  this in conjunction with the long-query-time option will log any query that takes over a certain amount of time (default is 10s).

3) show mysql processlist: If you haven’t set up the log slow queries option this might also be helpful in determining what mysql is doing.  Here is a brief explaination of show processlist and its companion of kill <processid>.  I also found this handy script that you could set up as part of a cron to kill any runaway processes.

4) query is poorly written: I’ve found a number of reports that MySQL queries are poorly written in Moodle and that these can cause the CPU usage to top out at %100 which causes Moodle to be unusable.  Below are a few Moodle Tracker reports of this:

To see if this is the case with your Moodle use the slow query logging (or just the plain old query logging) with a backup of your database to see if you can replicate the problem somewhere else.  If you can the problem is probably the query and maybe you can figure out a way to rewrite the query (or get it rewritten) to see if that fixes the problem.

5) check your persistent db connection: Check to see if you have persistent database connections set up in the Moodle config.php file and in the php.ini file.  If you do, turn it off and see if this solves the problem.  it may be that your connections aren’t closing and they are killing your database regularly.

6) check your max connections: Are the maximum number of Apache connections greater than your maximum number of MySQL connections?  If so that might be a problem.  Basically users could get to the website but then when they go to request something in the database they can’t because the # of connections are taken up.  I have a feeling though that this may be a completely different error than what I was experiencing (people…including myself…really need to take screenshots or document errors better).

7) check your MySQL performance: Perhaps you haven’t configured Moodle right.  Take a look at these performance tuning tips to see if you can get it working better.

8) check your database tables: Stop mysqld and run a check on your Moodle tables.  See if some of the tables are corrupted, if they are fix them.

So if you’re wondering what I’m doing at 4am tomorrow, there’s the list.  Wish me luck.

Posted in Work | Tagged , | 2 Comments

My Backup Script

About a month ago I made the mistake of deleting a backup script from the server I was maintaining.  This was my only copy of it and I hated myself two seconds after i typed rm -f backupscript.

Now I have quite a few more servers than I did a couple of months ago and they are all functioning in a production environment.  Therefore I need a more organized way to back stuff up.  Sure we do tape backups and what not, but I like to have everything all in one place in case i need to port it somewhere or other (for testing and dev purposes).

Anyway the thing backs up my MySQL databases, app directories, and some key config files (apache and php for now).  It also prints out a lovely log just in case something goes wrong (in the crontab i have it print things to the log).  Eventually I’d like the log to do basic error checking beyond the minimal it does now (ie listing out the files it just created).

Below you’ll find my backupscript.  Could it be worse? Yes.  Could it be better? Hell yes.  Do I want to answer questions about it?  NO!  In other words, use at your own risk. (Disclaimer: this blog is for me and my awful memory, the stuff here is not intended for use outside of that).

#!/bin/bash
#first define some variables that are used throughout the script
MYSQL=`which mysql`;
MYSQLDUMP=`which mysqldump`;
GZIP=`which gzip`;
TAR=`which tar`;
DATE=`date`;

#have the user of the script define variables for their system

#list the directories you want backed up (leave out final slash)
APPDIRS=( /dir/to/app1 /dir/to/app2 );
#list the above directories corresponding application name
APPNAMES=( app1 app2 );
#list root mysql user information
USER="root";
PWD="*****";
#list what directory you want things backed up to
DEST="/dir/to/backup";

echo " ";
echo " ";
echo "-------------------------------------------------";
echo "BEGIN BACK UP LOG FOR $DATE";
echo "-------------------------------------------------";
echo " ";
echo " ";
echo " ";
echo "-------------------------------------------------";
echo "PROCESSING DATABASE FILES";
echo " ";

#call the databases from mysql that will be backed up
#do not include the test database or the information schema
DBS=(`$MYSQL  -u $USER -p$PWD  -Bse 'show databases' | grep -v "test" | grep -v "information_schema"`);

#now display the databases that are being backed up
#so that they will be printed in the log
echo "The following databases will be backed up...";
for db in ${DBS[@]};
do
 echo "  $db";
done;

#begin backing up the database
#show that you are backing up the database for benefit of the log
echo " ";
echo " ";
echo "Backing up databases...";
for db in ${DBS[@]};
do
 echo -n "  backing up $db.....";
 # a mysqldump of the database and then gzip it up
 `$MYSQLDUMP --default-character-set=utf8 --set-charset -u $USER -p$PWD $db | $GZIP -9 > $DEST/$db.sql.gz`
 echo "DONE";
done;

echo " ";
echo " ";
echo "Created the following backup files...";
set $DEST/*.sql.gz

for file in ${@};
do
 echo "  $file";
done;

echo " ";
echo "FINISHED PROCESSING DATABASE FILES";
echo "-------------------------------------------------";
echo " ";
echo " ";
echo " ";
echo "-------------------------------------------------";
echo "PROCESSING CONFIGURATION FILES";
echo " ";

#now lets get other important files and back them up here too
cp /etc/php.ini $DEST/php.ini.bak
cp /etc/httpd/conf/httpd.conf $DEST/httpd.conf.bak

#display the files that you just copied over
echo "Copied the following configuration files...";
set $DEST/*.bak
for file in ${@};
do
 echo "  $file";
done;

#now we need to tar them up
echo " ";
echo " ";
echo "Tarring up the configuration files...";
#and then zip them up
$TAR cvzf conf.tgz $DEST/*.bak

#and delete the unneeded .bak files
rm -f $DEST/*.bak
echo "DONE";
echo " ";
echo "FINISHED PROCESSING CONFIGURATION FILES";
echo "-------------------------------------------------";
echo " ";
echo " ";
echo " ";
echo "-------------------------------------------------";
echo "PROCESSING APPLICATION FILES";
echo " ";
echo "Backing up the following application directories...";

#take the directories that are in the array and print them
for appdir in ${APPDIRS[@]};
do
 echo "  $appdir";
done;
echo " ";
echo " ";
echo "Tarring up the directories...";
for (( i = 0 ; i < ${#APPDIRS[@]} ; i++ ));
do
 echo "creating tar archive for ${APPNAMES[$i]}";
 tar cvzf ${APPNAMES[$i]}.tgz ${APPDIRS[$i]}/;
 echo "DONE";
 echo " ";
done;
echo " ";
echo "Created the following backup files...";
for name in ${APPNAMES[@]};
do
 ls $name.tgz;
done;

echo " ";
echo "FINISHED PROCESSING APPLICATION FILES";
echo "-------------------------------------------------";
echo " ";
echo " ";
echo " ";
echo "-------------------------------------------------";
echo "END BACKUP LOG FOR $DATE";
echo "-------------------------------------------------";
Posted in Work | Tagged , | 2 Comments

Article — Women in IT: Success and struggle

RT @VanessaAlvarez1: RT @neilweinberg :The proud. The few. The brave. Women in IT. http://bit.ly/yHlMn

This article on Women in IT really hit home for me.  I often come back to work and look at my tweets and feedreader and think, “When do these people have time to think about this stuff after the end of the work day?”.  Now I know.

When I’m at home cooking for my husband (I love you honey), he’s spending time being productive for work.  So if I could convince him to do all the cooking maybe I could keep up too (yeah…like that’s going to happen (still love you honey)…he always pulls out the I make more money card).

I guess my one saving grace is I’m in academia where the demands aren’t as ridiculous as the business world.

Posted in Work | Tagged , | Leave a comment