I had two options. I could directly monitor the databases from monitor using the check_mysql or
run check_mysql on the database server and call it through check_by_ssh. I started out configuring
the method over ssh.

First, in each database server I created a user nagios and set a password for that user. I then
created a set of keys:

ssh-keygen -t dsa

I set no passphrase for the key, since I intended it to have a single purpose and limited
access to the database servers. I then tested the access to see if there were any glitches.
It worked on one server but not the other. After a quick once over, I decided to proceed and
solve that problem later. Since I needed check_mysql I compiled the plugins

From the nagios plugins web site, I downloaded nagios-plugins-1.4.14.tar.gz to each of the
database servers. However, because not all the necessary mysql packages were in place, it
threw some errors during the configure stage.

./configure –with-nagios-user=nagios –with-nagios-group=nagios –with-mysql=/usr

The main error that interfered with my plans was the failure to build the check_mysql plugin.
After some research I discovered that the failure was probably due to the absence of some
mysql libraries that would be in a development package. However, RHEL5 doesn’t have such a
package in their repository for RHEL5. It is available in some of the alternate repositories. It’s not really that surprising, when I think about it, that the package is unavailable, RHEL isn’t intended to be a development platform, it’s a server platform. I didn’t want to add alternate repositories without permission from the client.

So I went for plan B. I decided to configure the check-mysql to run on the monitor server
and attach to the mysql database over the network. There is a danger that If not carefully configured this could represent a security vulnerability for the database server. To make it as secure as possible
I logged into mysql on each of the database servers and created special access rules for this
purpose. I created a special nagios user with it’s own password and gave it read only permissions
and only on one database.

grant select on database.* to nagios@monitor identified by “password”

Now the user nagios can read that database. It doesn’t have any more privileges, so it’s unlikely to be used to damage the database, even if the monitor were compromised. The bad guys won’t
be able to use the compromise of the monitor to also compromise or damage the database server.

To test my work I issued the following command:

/usr/local/nagios/libexec/check_mysql -d database -u nagios -p password -H $HOSTNAME$

$HOSTNAME is the ip of the database server.
The data came back:

Uptime: 528011 Threads: 61 Questions: 83799845 Slow queries: 38527 Opens:
11365 Flush tables: 1 Open tables: 1003 Queries per second avg: 158.709

To prevent the username and password from being exposed in the web interface I put some of the command values in resource.cfg

###########################################################################
#
# RESOURCE.CFG – Sample Resource File for Nagios 3.0b6
#
# Last Modified: 09-10-2003
#
# You can define $USERx$ macros in this file, which can in turn be used
# in command definitions in your host config file(s).  $USERx$ macros are
# useful for storing sensitive information such as usernames, passwords,
# etc.  They are also handy for specifying the path to plugins and
# event handlers – if you decide to move the plugins or event handlers to
# a different directory in the future, you can just update one or two
# $USERx$ macros, instead of modifying a lot of command definitions.
#
# The CGIs will not attempt to read the contents of resource files, so
# you can set restrictive permissions (600 or 660) on them.
#
# Nagios supports up to 32 $USERx$ macros ($USER1$ through $USER32$)
#
# Resource files may also be used to store configuration directives for
# external data sources like MySQL…
#
###########################################################################

# Sets $USER1$ to be the path to the plugins
$USER1$=/usr/local/nagios/libexec
# Sets $USER2$ to be the path to event handlers

#$USER2$=/usr/local/nagios/libexec/eventhandlers
# Store some usernames and passwords (hidden from the CGIs)

$USER3$=nagios
$USER4$=password

###############################################################################################

commands.cfg is where I put my tested command

############################### commands.cfg ######################
define command{
command_name check-mysql
command_line $USER1$/check_mysql -d tracking -u $USER3$ -p $USER4$ -H $HOSTADDRESS$
}
####################################################################

Next I need to tell nagios where to send messages.

############################### contacts.cfg #######################
define contact{
contact_name pacneil
use generic-contact
alias Neil Schneider
email pacneil@linuxgeek.net
}

define contactgroup {
contactgroup_name pacneil
alias Test Group
members pacneil
}
######################################################################

What group of servers are we going to monitor?

############################### host_groups.cfg #####################
define hostgroup{
hostgroup_name db-host-group
alias Database Servers Host Group
hostgroup_members db-slave-host-group
}

define hostgroup{
hostgroup_name db-slave-host-group
alias Slave Database Servers Host Group
}
######################################################################

And we need to configure some parameters how we want to display the hosts in the web interface.

############################### hosts.cfg ############################
define host{
use db-server
host_name db3.servers.pmc
hostgroups db-slave-host-group,lb1-host-group,rackspace-host-group
alias db3
display_name Db3
address 74.205.65.35
parents app2.servers.pmc
2d_coords 100,0
3d_coords -5,4,1
}

define host{
use db-server
host_name db4.servers.pmc
hostgroups db-slave-host-group,lb1-host-group,rackspace-host-group
alias db4
display_name Db4
address 74.205.65.36
parents app2.servers.pmc
2d_coords 200,0
3d_coords -5,4,-1
}

######################################################################

And I create a service group just for database servers.

########################## service_groups.cfg ########################

define servicegroup{
servicegroup_name db-server-service-group
alias Database Server Service Group
servicegroup_members server-service-group
}
######################################################################

Then I define the service

############################ services.cfg ############################
define service{
use server-service
name db-server-service
servicegroups db-server-service-group
hostgroup_name db-host-group
register 0
}

######################################################################

I hope it saves you some time and you find it useful.

…… read more

The server release is touted as providing “the most cost-effective, flexible, and scalable environment”. There are two flavors, Red Hat Enterprise Linux and Red Hat Enterprise Linux Advanced Platform. The number of guests on Advanced Platform is unlimited while the standard release is limited to four guests.

Redhat Enterprise Linux is certified as a guest OS on the following platforms:

• VMware ESX and VMware ESXi
• BM POWER LPARs
• IBM System z

Redhat Enterprise Linux supports three distributed system technologies:

• Logical Volume Manager (LVM)
• Global File System (GFS)
• Distributed Lock Manager (DLM)

Redhat also has new managment tools for managing virtualized environments. And of course it provides tools like Samba for integration into Windows environments, Apache Web server, MySQL and Postgresql databases. And of course all the enterprise network services such as DNS, dhcp, and firewall capabilities are included.

Redhat is also promoting Redhat Enterprise Desktop as an alternative to “proprietary” desktop operating systems. They’re selling the “security” features and cost benefits of managing their system. And if you must run one of those “proprietary” systems, they offer virtualization to run it as a guest OS.

While not everything on RHEL 5.4 is the latest release, it does provide the kind of system and support that will make management comfortable.

Subscribers to RHEL 5 will get the updates automatically for free. New subscribers will pay about $349 for the server version and prices start at $80 for the desktop version.

Before I was called in, the customer had installed a new ethernet card, on the assumption that the on-board card had failed. They came to that conclusion because they couldn’t ping out, and no other machines could ping the server. A good assumption, I think. However, the new ethernet card didn’t fix the problem. That’s when they called me in.

We went through the configuration with a fine tooth comb. We disabled Network Manager, since I’ve known it to cause issues on some systems. Everything seemed fine, but the network still didn’t work. The network configuration was static. So we changed to using dhcp to get an address. This worked, it got an address, but we still couldn’t ping either direction, from the server or to the server. We looked for alternate kernels we could boot, to see if it was a kernel specific problem, however there was only one kernel available. As I recall there wasn’t even a failsafe option in the boot menu.

As part of the diagnostic process we put a laptop on the wired network and tried pinging it from the server. No response. we checked the ARP tables and there was an entry for the server we was working on, with it’s IP and MAC addresses. We concluded that the server could “talk” but couldn’t “hear”. We checked for firewall rules that might be blocking. There were none. We checked the arp tables on the server, and there were entries, without host names, for other machines on the network. None of us had see this before. The plot thickens.

So next we decided to try a Knoppix live CD to test the hardware. we figured if Knoppix worked, then there was a software configuration problem. Sure enough, Knoppix booted and networking was working. We could ping, other machines on the network could ping the server. So we concluded that it must be something in the software.

We again, went through the network configuration files. Suspecting that there may have been some corruption in the files, we manually rewrote them from scratch. We thought there was a possibility of unprintable characters in the configuration files that was causing them not to work properly. We flushed the firewall rules for good measure with iptables -F just to make sure there were no rules, that were not being reported by iptables -L. Still no change.

Still operating under the assumption that there was a configuration errors somewhere, we decided to try an upgrade, to OpenSuSE 11.1. We backed up all the configuration files and data to an external USB drive. Then we did an upgrade. Upgrading is a pretty slow process, relative to a fresh install. Nothing changed.

Next we tried a fresh install, no change. Then I tried another distributions, CentOS 5. Still no change. Finally as my frustration mounted, I did ifup eth0 and a root console popped up displaying an error message. “…. irq #66 disabled” So we googled the error and found that this kind of error appears when there are spurious interrupts on the bus. After a couple more tests we concluded that there was some kind of hardware problem, that we hadn’t seen before. Since the system was under warranty, we contacted the manufacturer and they replaced the motherboard. After a fresh install, the system came up and ran perfectly.

This is one of the things I both love and hate about this business. Everyday there is a new problem to be solved that we’ve never seen before. I hope this saves someone else some time diagnosing their hardware problems.


uncompressing linux
OK booting the kernel
Red Hat nash version 4.2.1.10 starting
Ext3-fs: unable to read superblock
mount error 22 mounting ext3
mount error 2 now mounting none
switchroot: mount failed 22
umount /initrd/dev failed 2
Kernel panic not syncing
Attempted to kill init


It’s been a while since I had issues like this. So I went looking through Google for answers. Isn’t that what everyone does these days? Everything search showed seemed to indicate a problem with the the raid. So I tried to look for a problem with the raid partitions. First I used Knoppix. I found a web site Recover Data From RAID1 LVM Partitions With Knoppix Linux LiveCD I followed the instructions, using mdadm –examine –scan to examine the partitions and manually set up the raid. Everything worked fine. I could see the raid partitions, I could mount them and view the data. cat /proc/mdstat showed the proper raided partitions. So what’s the problem?

Next I tried the rescue option of the Centos CD. The system booted fine, and the rescue mode dutifully mount all the partitions in the right order to /mnt/sysimage. But still no love when I tried to boot. Next I got on irc and talked to some of my friends from KPLUG.

One suggested trying to boot to the grub prompt and running boot options by hand, using tab completion. He suggested that perhaps the system wasn’t recognizing the drive. It seemed plausible, so I tested it. But the system seemed to be finding the /boot partition just fine and tab completion found the init ram disk fine.

Another suggested that the failure of the system to recognize the drives was caused by not loading the proper controller module. This had not occurred to me, because I thought I remembered that all the modules for hard drive host controllers were built into the kernel. At least that’s the way I used to build them. I remembered wrong. My friend suggested that I look at /etc/modprobe.conf and check the scsci_hostadapter parameter. So I booted into the Centos CD rescue mode, as the system booted I noticed the message loading ata_piix. This was the clue I had been missing. I edited the /etc/modprobe.conf file and changed the reference there. When I attempted to reboot the system the raid partitions were recognized and the system finished booting.

Once upon a time, modules for hard drives were included in the kernel, not loaded at boot time. Now with the size of kernels getting very large, nearly everything is loaded at boot. That’s fine for things like ethernet and sound cards. It’s always been a point of pride with me, that we could take the hard drives from one system, and install them into another system and it would just run. I guess that’s no longer true.

IMNSHO this is not progress.

So I started looking for it on my system. Not to be found. Not where it belonged in /etc/ldap/schema not in /usr/share/docs/samba. I used find
to search my system, looking for the schema file. In Debian, even if you install packages that support the ldap authentication method for samba, you don’t get the samba.schema file necessary to make it work. You have to install samba-doc and then move or copy the samba.schema file to the right place to make it work.

I installed the samba-doc package and the schema file in /usr/share/doc/samba-doc/examples/LDAP I have to question what the Debian developers were thinking when they put the schema file in the samba-doc package. I didn’t install most of the optional doc packages, because my space is a little tight and I am trying to conserve it the best I can.

Once I found the samba.schema file and put it in the right place, I used ldapadd to install my ldif file and restarted slapd. After that, everything worked fine and samba is up and running as it was before I did the upgrade to Lenny.

Onward ……………….

So I found the instruction Chapter 4: Upgrading from previous releases. I followed the instructions, except for recording the session. Next time I’ll probably consider doing this too. Everything seemed to go well. I managed to do the entire upgrade, while logged into the machine, and reading the web page in the browser running on that same machine. I was pretty impressed, I have to say. I was even able to chat on IRC with my friends while I did the upgrade.

Nothing ever works perfectly. For some reason, every time I upgrade even the kernel, it messes up my /boot/grub/menu.lst My boot drive is /dev/sda2 but it keeps getting set to /dev/sdb2, go figure. After repeatedly having this problem, it really doesn’t bother me that much anymore. It’s just a minor annoyance.

I have a samba share where I store all my Windows data files. I don’t trust Windows to keep my data safe, so I store it on my Linux machine. I use ldap for authentication. So after my upgrade I tripped over <a href=”http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=495954″ bug #495954</a> I found it interesting that this bug seemed to have been reported and verified by several people, but no one seemed to want to report a work around. Either my Google foo is weak, or no one has reported how to work around this. I’m in the process of reconstructing my authentication system from scratch. I also notice there is no samba3 schema file. I’m still trying to track this down.

More later ………………………………..

I was importing some music into my collection and the entire collection disappeared from the left panel. I could still view the files and I could still play, but the collection panel was empty. I ran Rescan Collection, it progressed to about 82% and then quit. I searched all over and this appears to be a common problem. And it doesn’t seem to matter if you’re using MySql as I am or sqlite. I tried all the suggestions posted to the amarok forums, nothing worked. I basically wiped out all my amarok configuration files and started over, with the same result.

So I checked around and found there is an IRC channel #amarok. I thought I would give that a try. Although there were a lot of people in the channel, no one responded to my questions. Finally a smart-ass posted and said that I should try something else since Amarok was bad software. Although I wasn’t quite ready to accept this at face value, I was somewhat surprised that someone would make a statement like this on a support channel.

I continued to search for a solution, to no avail. It seems there is a bug in Amarok that causes it to fail to manage the collection list once it gets to a certain size, or if it encounters a file with some kind of error. I even ran amarokcollectionscanner manually to see if I could spot an error. Nothing appeared, though it seems to not complete scanning the entire collection.

I checked and it appears that my MySQL database is intact. It looks like, for whatever reason, Amarok can’t or won’t access it.

Finally out of frustration, I started looking at other applications. So far Rhythmbox seems to be working well. It does most all the things that Amarok does. It doesn’t seem to have the tagging capabilities that Amarok has, but it sees and displays my entire collection, so that’s a start. I’ll continue to look at other applications to replace Amarok. I’ll write about it when I have time.

Update: I went from Etch to Lenny and tried Amarok again. This time it scanned my collection and rebuilt everything just fine. Probably a bug in the old version that’s been fixed. I’m happy.