Just a word about this post:
If this seems disjointed, illogical with more than a few misspelled words, come back later. This started out as my notes on configuring a set of virtual servers inside a client’s network. So I started writing as I went along. Eventually I’ll have it edited and make it final, but for now it’s just my notes. When it’s finished, I’ll remove this note.
Just sayin.
I have been tasked with setting up three virtual machines on a single host. These hosts will each have only one basic function on the company intranet. They will be a cvs server, a ftp server and a samba server. This will be a place to keep my notes.
Since I’m working with x86-64 hardware and all guest OS’s will be Linux, KVM seems the best choice. What I’ve read says it imposes very low overhead, and I like the ability to use a logical volume directly. I planned to install each virtual host into a separate logical volume, with the intention of being able to adjust the size of the hard drive inside the virtual machine, as needed by a changing business. We chose CentOS 5.5 as it seems a good choice for the standard on all this company’s servers. Most of their servers are RH or clones thereof.
Hardware
Dell PowerEdge R210 Intel® Core™ I3 540 3.06GHz, 4M Cache, 2C/4T 16 Gb Memory (4x4Gb) 2 x 2TB 7.2K RPM SATA 3.5in Cabled Hard Drive mirrored, DVD-ROM Drive and BMC.
On some Dell hardware, you also need to disable “Trusted Execution”, otherwise VT will not be enabled. That was not the case on this hardware. The CPU does have the VT extensions.
ftp server
200 Gb
1 CPU
4 Gb RAM
cvs server
300Gb
1 CPU
4 Gb RAM
samba server
800Gb
1 CPU
4 Gb RAM
The machine came with 2 drives so that we could mirror them. The system came with software raid controller, so I just chose to use Linux built in software raid. I configured most of the the drive space in RAID I. To install the system I needed to create a 100M partition outside the raid, because /boot cannot be within the software raid.
Raid Devices
/boot 100MB
/opt 100MB
After the install, I changed the options in fstab to ro,noauto,nouser,sync and then did a poor mans mirror
dd if=/dev/sda1 of=/dev/sdb1
The unexpected result was that this changed the label so the partition would no longer mount on /opt. I’ll have to relabel the partition and then add the entry back to the fstab file. I wonder what happens when you have two disks with the same disk label?
I found later one consequence, when I got in a state where I needed to try to upgrade to fix some things. The boot CD found two partitions /dev/sda1 and /dev/sdb1 with disklabel /boot. It refused to continue, telling me to fix that first. When I do this next time I will use tune2fs to relabel the partition. If we ever lose the drive with /boot, my hope is that it will allow us to continue running the system. This is a mirrored drive setup, and with a bootable partition, it can be recovered more simply.
The rest of the 2 terrabyte drive became the Volume Group System. I’m not a fan of the default names used during the install. Which logical volume inside VOL_GROUP00 contains the /usr partiton? Is it VOL00 or VOL01 or VOL05? I override the names and give them names that will help me identify the data, when I have to boot from a rescue CD and start copying the data off a failing system, or make a change to the fstab to get the system to boot from the still good drive in a mirrored pair.
Inside that I created logical volumes
logical volume root mounted on /
logical volume swap
Since the installer always wants a mount point for each partition and logical volume, I wait until after the install of the host system to create:
logical volume ftp for the ftp server
lvcreate -L 200G -n ftp System
logical volume cvs for the cvs server
lvcreate -L 200G -n cvs System
logical volume samba for the samba server
lvcreate -L 200G -n samba System
Resizing logical volumes inside logical volumes
1. shutdown virtual machine
use kparx to add the logical volumes
kpartx -a /dev/System/ftplvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
samba System -wi-a- 1.00T
cvs System -wi-a- 198.00G
ftp System -wi-ao 248.00G
root System -wi-ao 3.91G
swap System -wi-ao 1.94G
LogVol00 VolGroup00 -wi— 192.22G
LogVol101 VolGroup00 -wi– 5.66G
2. extend HOST logical volume
lvextend --L+50G /dev/mapper/System-ftp
Extending logical volume ftp to 248.00 GB
Logical volume ftp successfully resized
vgdisplay -v
3. resize the physical volume on the virtual machine
pvresize --setphysicalvolumesize 248G /dev/vda2
lvextend -L+50G /dev/VolGroup00/LogVol00
This failed. I was able to resize the logical volumes on the host, but I kept getting errors similar to:
device-mapper: reload ioctl failed: Invalid argument Failed to suspend LogVol00
I found plenty of links to other people who’ve encountered this problem, but no solution. So we decided to fix the size of the logical volumes for each host and move on.
So I set fixed sizes for the logical volumes.
400G for the ftp server
557G for the cvs server
900G for the samba server
Then I installed CentOS 5.5 on each.
Networking
Getting the machines to connect to the LAN with addresses on that LAN was another challenge. The default CentOS install set up a virtual network between the hosts with an outbound NAT connection. I wanted each machine to have a discreet IP address and be as separated as possible. Same reason I had them running in their own logical volumes.
I found lots of descriptions of what I needed to do, but they all seemed to lacking a small piece of information. I discovered you must create the bridge device on the host first, before you install the virtual hosts. At least for me, I wasn’t able to install and then change the network configuration. I may not have understood the required modifications well enough.
I used the instructions from this site:
CentOS / Redhat: KVM Bridged Network Configuration
I first created the bridge device. I made backups of original files before I started. Remember to use prefix, not postfix notation. bak.ifcfg. The scripts look for any file that starts with ifcfg and then acts upon them. First a new file for the bridge device.
vi /etc/sysconfig/network-scripts/ifcfg-br0DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.1.5
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
ONBOOT=yes
Then edit the ethernet configuration file, after making a backup:
vi /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0
BOOTPROTO=none
HWADR=12:34:56:78:91:23
BRIDGE=br0
ONBOOT=yes
Other interesting links:
Libvirt overwrites the existing iptables rules
Redhat Hypervisor Deployment Guide
How to Get Maximum Network Performance using paravirtual drivers and bridged networking
Using bridged networking with Virt-manager
A Quick Guide to Using KVM with CentOS-5.1
KVM is interesting again.. and how I setup my virtual network…
KVM Bridged Network – solutions and problems
CentOS / Redhat: KVM Bridged Network Configuration