In my case the output of mdstat looks like this:

# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda2[0] sdb2[1]
33615936 blocks [2/2] [UU]

md1 : active raid1 sda3[2](F) sdb3[1]
2096384 blocks [2/1] [_U]

md0 : active raid1 sda1[0] sdb1[1]
128384 blocks [2/2] [UU]

unused devices:

So I have three partitions on two drives raided together. And sda3 is failing. This is the message I received in email.

This is an automatically generated mail message from mdadm
running on host.domain.com

A Fail event had been detected on md device /dev/md1.

Faithfully yours, etc.

Device Boot Start End Blocks Id System
/dev/sda1 * 1 16 128488+ fd Linux raid autodetect
/dev/sda2 17 4201 33616012+ fd Linux raid autodetect
/dev/sda3 4202 4462 2096482+ fd Linux raid autodetect

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 16 128488+ fd Linux raid autodetect
/dev/sdb2 17 4201 33616012+ fd Linux raid autodetect
/dev/sdb3 4202 4462 2096482+ fd Linux raid autodetect

Disk /dev/md0: 131 MB, 131465216 bytes
Disk /dev/md1: 2146 MB, 2146697216 bytes
Disk /dev/md2: 34.4 GB, 34422718464 bytes

Removing the failed partition(s) and disk:

I used the mdadm command to first fail

mdadm –manage -dev/md0 –fail /dev/sda2
mdadm –manage -dev/md1 –fail /dev/sda3
mdadm –manage -dev/md2 –fail /dev/sda1

then remove the raid devices on the failing drive.

madam –manage /dev/md0 –remove /dev/sda2
madam –manage /dev/md1 –remove /dev/sda3
madam –manage /dev/md2 –remove /dev/sda1

Then I shut down the system

shutdown -h now

and replaced the drive with a new one. Then I tried to reboot. But because the failed drive was the first drive in the scsi chain, it failed to boot with the message.

No Operating System Present

Adding the new disk to the RAID Array:

So I ended up having to switch the drives, putting sdb in as sda and then proceeding. I used sfdisk to mirror the partitioning between the two drives.

sfdisk -d /dev/sd1 | sfdisk /dev/sdb

Add the partitions back into the RAID Arrays:

mdadm –manage /dev/md0 –add /dev/sdb2
mdadm –manage /dev/md1 –add /dev/sdb3
mdadm –manage /dev/md2 –add /dev/sdb1

cat /proc/mdstat

I could see the drive rebuilding. When it finished I hot swapped out sda and did the whole process over again, this time without rebooting the system, since the system uses hot swap drives. It worked fine and I had both drives up and running. I could have done the whole process without rebooting the machine.

Install Grub on new hard drive MBR:

# grub
grub> find /grub/stage1
(hd0,0)
grub> device (hd0) /dev/sdb
grub> root (hd0,0)
grub> setup (hd0)
grub> find /grub/stage1
(hd0,0)
(hd1,0)

grub> quit

So now I have the boot manager mirrored on both drives. I can reboot with either single drive and it will work fine.

I had two options. I could directly monitor the databases from monitor using the check_mysql or
run check_mysql on the database server and call it through check_by_ssh. I started out configuring
the method over ssh.

First, in each database server I created a user nagios and set a password for that user. I then
created a set of keys:

ssh-keygen -t dsa

I set no passphrase for the key, since I intended it to have a single purpose and limited
access to the database servers. I then tested the access to see if there were any glitches.
It worked on one server but not the other. After a quick once over, I decided to proceed and
solve that problem later. Since I needed check_mysql I compiled the plugins

From the nagios plugins web site, I downloaded nagios-plugins-1.4.14.tar.gz to each of the
database servers. However, because not all the necessary mysql packages were in place, it
threw some errors during the configure stage.

./configure –with-nagios-user=nagios –with-nagios-group=nagios –with-mysql=/usr

The main error that interfered with my plans was the failure to build the check_mysql plugin.
After some research I discovered that the failure was probably due to the absence of some
mysql libraries that would be in a development package. However, RHEL5 doesn’t have such a
package in their repository for RHEL5. It is available in some of the alternate repositories. It’s not really that surprising, when I think about it, that the package is unavailable, RHEL isn’t intended to be a development platform, it’s a server platform. I didn’t want to add alternate repositories without permission from the client.

So I went for plan B. I decided to configure the check-mysql to run on the monitor server
and attach to the mysql database over the network. There is a danger that If not carefully configured this could represent a security vulnerability for the database server. To make it as secure as possible
I logged into mysql on each of the database servers and created special access rules for this
purpose. I created a special nagios user with it’s own password and gave it read only permissions
and only on one database.

grant select on database.* to nagios@monitor identified by “password”

Now the user nagios can read that database. It doesn’t have any more privileges, so it’s unlikely to be used to damage the database, even if the monitor were compromised. The bad guys won’t
be able to use the compromise of the monitor to also compromise or damage the database server.

To test my work I issued the following command:

/usr/local/nagios/libexec/check_mysql -d database -u nagios -p password -H $HOSTNAME$

$HOSTNAME is the ip of the database server.
The data came back:

Uptime: 528011 Threads: 61 Questions: 83799845 Slow queries: 38527 Opens:
11365 Flush tables: 1 Open tables: 1003 Queries per second avg: 158.709

To prevent the username and password from being exposed in the web interface I put some of the command values in resource.cfg

###########################################################################
#
# RESOURCE.CFG – Sample Resource File for Nagios 3.0b6
#
# Last Modified: 09-10-2003
#
# You can define $USERx$ macros in this file, which can in turn be used
# in command definitions in your host config file(s).  $USERx$ macros are
# useful for storing sensitive information such as usernames, passwords,
# etc.  They are also handy for specifying the path to plugins and
# event handlers – if you decide to move the plugins or event handlers to
# a different directory in the future, you can just update one or two
# $USERx$ macros, instead of modifying a lot of command definitions.
#
# The CGIs will not attempt to read the contents of resource files, so
# you can set restrictive permissions (600 or 660) on them.
#
# Nagios supports up to 32 $USERx$ macros ($USER1$ through $USER32$)
#
# Resource files may also be used to store configuration directives for
# external data sources like MySQL…
#
###########################################################################

# Sets $USER1$ to be the path to the plugins
$USER1$=/usr/local/nagios/libexec
# Sets $USER2$ to be the path to event handlers

#$USER2$=/usr/local/nagios/libexec/eventhandlers
# Store some usernames and passwords (hidden from the CGIs)

$USER3$=nagios
$USER4$=password

###############################################################################################

commands.cfg is where I put my tested command

############################### commands.cfg ######################
define command{
command_name check-mysql
command_line $USER1$/check_mysql -d tracking -u $USER3$ -p $USER4$ -H $HOSTADDRESS$
}
####################################################################

Next I need to tell nagios where to send messages.

############################### contacts.cfg #######################
define contact{
contact_name pacneil
use generic-contact
alias Neil Schneider
email pacneil@linuxgeek.net
}

define contactgroup {
contactgroup_name pacneil
alias Test Group
members pacneil
}
######################################################################

What group of servers are we going to monitor?

############################### host_groups.cfg #####################
define hostgroup{
hostgroup_name db-host-group
alias Database Servers Host Group
hostgroup_members db-slave-host-group
}

define hostgroup{
hostgroup_name db-slave-host-group
alias Slave Database Servers Host Group
}
######################################################################

And we need to configure some parameters how we want to display the hosts in the web interface.

############################### hosts.cfg ############################
define host{
use db-server
host_name db3.servers.pmc
hostgroups db-slave-host-group,lb1-host-group,rackspace-host-group
alias db3
display_name Db3
address 74.205.65.35
parents app2.servers.pmc
2d_coords 100,0
3d_coords -5,4,1
}

define host{
use db-server
host_name db4.servers.pmc
hostgroups db-slave-host-group,lb1-host-group,rackspace-host-group
alias db4
display_name Db4
address 74.205.65.36
parents app2.servers.pmc
2d_coords 200,0
3d_coords -5,4,-1
}

######################################################################

And I create a service group just for database servers.

########################## service_groups.cfg ########################

define servicegroup{
servicegroup_name db-server-service-group
alias Database Server Service Group
servicegroup_members server-service-group
}
######################################################################

Then I define the service

############################ services.cfg ############################
define service{
use server-service
name db-server-service
servicegroups db-server-service-group
hostgroup_name db-host-group
register 0
}

######################################################################

The server release is touted as providing “the most cost-effective, flexible, and scalable environment”. There are two flavors, Red Hat Enterprise Linux and Red Hat Enterprise Linux Advanced Platform. The number of guests on Advanced Platform is unlimited while the standard release is limited to four guests.

Redhat Enterprise Linux is certified as a guest OS on the following platforms:

• VMware ESX and VMware ESXi
• BM POWER LPARs
• IBM System z

Redhat Enterprise Linux supports three distributed system technologies:

• Logical Volume Manager (LVM)
• Global File System (GFS)
• Distributed Lock Manager (DLM)

Redhat also has new managment tools for managing virtualized environments. And of course it provides tools like Samba for integration into Windows environments, Apache Web server, MySQL and Postgresql databases. And of course all the enterprise network services such as DNS, dhcp, and firewall capabilities are included.

Redhat is also promoting Redhat Enterprise Desktop as an alternative to “proprietary” desktop operating systems. They’re selling the “security” features and cost benefits of managing their system. And if you must run one of those “proprietary” systems, they offer virtualization to run it as a guest OS.

While not everything on RHEL 5.4 is the latest release, it does provide the kind of system and support that will make management comfortable.

Subscribers to RHEL 5 will get the updates automatically for free. New subscribers will pay about $349 for the server version and prices start at $80 for the desktop version.

If you’ve been here before then you know this is my hobby site. I’ve been using Unix/Linux since 1988. Linux is my primary OS and I rarely touch Windows unless I have to.

I’m the president of the local Linux User Group Kernel Panic Linux User Group I participate in local installfests, TechFests and of course our monthly meetings, where I and other members of the group give presentations on Linux topics.

I also do Computer consulting, including network security, network design and integration, system administration and of course my specialty is Linux . The name of my company is Pacific Rim Computer Products. Feel free to contact me, if you need some help.