Thought of sharing my recent experiences in exploring other Virtualization providers for our Organization in an effort to move away from VMWare(licenses ofcourse .. sure the Tools work great! )
There definitely are a few Open Source providers that have been around for a while and still gaining steam for example, Xen Server and KVM, other Tools that integrate well with Puppet/Chef like Razor etc. We chose KVM and gave it a run.
These option suit well for Small/Medium Companies while the trend with Larger Companies is to adopt more of a PaaS/IaaS like Tools, for example, OpenStack etc
1. New releases of VMware (ESXi 5 onwards) have inbuilt resource limitations. Ex – A Vmware VM cannot have more than 4 virtual CPUs.
What is KVM ?
KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 & x86_64 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.
Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc.
The following note is more about how we setup the Standard Bridged Networking with STP(Spanning Tree Protocol) enabled with Centos 6 as Base OS on a RAC Server…
1. Enable virtualization on the physical CPU using BIOS settings.
You can verify this with this command: cat /proc/cpuinfo | grep -e vmx(Intel CPU’s) -e svm(AMD CPU’s)
If you see some output for this command, you don’t need to re-visit the BIOS settings on the server.
2. We start with a physical machine that has CentOS 6.4 x86_64 installed. During installation, I recommend going with ‘Software Development Workstation’ option. This ensures all the dependant packages get installed.
As a Best Practice, when installing this Base OS, create it with a different hostname to that of the name given to the RAC Server using Vendor Software (we use Dell’s idrac) at the time if racking the server in the chassis, configuring it with a Static IP.to that on idrac and with its own Static IP.
Allocate All available Disk Space on the RAC Server to this installation, we can later allocate memory from this Server to the VM’s we provision.
3. Make sure that you have terminal access to the machine
4. Stop/ Disable Selinux, IPtables, IP6tables, NetworkManager(Its pretty buggy, and intervened with my setup at times) if possible.
5. Configure the machine with Static IP the standard way. your NW interface config file shud look something like:
/etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=”em1″
#BOOTPROTO=”dhcp” ← Disable DHCP
HWADDR= ← Use the existing value in the file
NM_CONTROLLED=”no” ← Disable NetworkManager
ONBOOT=”yes”
TYPE=”Ethernet”
UUID= ← Use the existing value in the file
IPV6INIT=no
IPADDR=*.*.*.* ← Replace as per your environment
NETMASK=*.*.*.* ← Replace as per your environment
GATEWAY=*.*.*.* ← Replace as per your environment
DNS1=*.*.*.* ← Replace as per your environment
Save the file
Restart the network – /etc/init.d/network restart
Verify that the new static IP address now shows up with ifconfig.
5.
Install packages yum install kvm libvirt python-virtinst qemu-kvm bridge-utils
If working on a Centos Installation that is done with a “Minimal” Installer, installing the following modules with yum will be sufficient to getting started with KVM:
yum groupinstall “virtualization client”
yum groupinstall “virtualization platform”
yum groupinstall “virtualization tools”
yum install tunctl
yum install -y python-hashlib
yum install dejavu-lgc-sans-fonts
yum install -y policycoreutils-python
Start the libvirtd daemon /etc/init.d/libvirtd start
6. Ensure that libvirtd starts correctly /etc/init.d/libvirtd status
7. To check if KVM has successfully been installed, run virsh -c qemu:///system list
It should display something like this:
[root@server1 ~]# virsh -c qemu:///system list
Id Name State
———————————-
Setup network bridge
Please complete these steps logged into the terminal.
1. cd /etc/sysconfig/network-scripts/
2. Create a new network bridge configuration file by copying the existing em1 network configuration file
cp ifcfg-em1 ifcfg-br0
3. Edit the ifcfg-em1 file and comment out the following variables – IPADDR, NETMASK , GATEWAY.
Edit the ifcfg-br0 file and setup the following values in there
DEVICE=”br0″ ← Device called br0
BOOTPROTO=”none”
#HWADDR= ← Comment it out
NM_CONTROLLED=”no”
ONBOOT=”yes”
TYPE=”Ethernet”
#UUID= ← Comment it out
IPADDR=*.*.*.* ← Replace as per env
NETMASK=*.*.*.* ← Replace as per env
GATEWAY=*.*.*.* ← Replace as per env
DNS1=*.*.*.* ← Replace as per env
NAME=”System br0″ ← Setup system name
DEFROUTE=yes
IPV6INIT=no
Save and restart the network.
Switch back to the command line, please issue these commands
# Command to setup a CentOS network bridge
brctl addbr br0
# Add em1 as the NIC for the bridge we just created
brctl addif br0 em1
# Enable stp on the bridge
brctl stp br0 on
To make sure everything was setup correctly, issue the command
brctl show
This is the expected output.
We are looking for br0 to be setup on interface em1 with STP enabled.
bridge name bridge id STP enabled interfaces
br0 8000.7845c49d1d28 yes em1
virbr0 8000.525400eae031 yes virbr0-nic
8. Now restart the network again. If a network restart does not setup the bridge correctly, you might have to restart the machine and then setup the bridge after restart.
9. Now check the IP address again.
This is the expected output
# br0 now has the static IP address.
br0 Link encap:Ethernet HWaddr 78:45:C4:9D:1D:28
inet addr:10.1.5.208 Bcast:10.1.7.255 Mask:255.255.252.0
inet6 addr: fe80::7a45:c4ff:fe9d:1d28/64 Scope:Link
…….
# em1 does not have an IP address
em1 Link encap:Ethernet HWaddr 78:45:C4:9D:1D:28
inet6 addr: fe80::7a45:c4ff:fe9d:1d28/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
…….
10. If everything has been setup correctly, you should be able to remotely login to the KVM host using SSH.
11. After completing steps #1 – 10, if you have issues connecting to the internal network, please make sure to setup a correct default route.
route add default gw <gateway> br0
To make sure that the route got correctly added, issue the command route -n.
Note:
Observed the Bridge Commands need to be run again, if the Server is rebooted,as a workaround, Add the following commands to /etc/rc.local
brctl addbr br0
brctl addif br0 em1
brctl stp br0 on
service network restart
With this the Bridge gets automatically created after reboot.
Creating virtual machines:
We can now use the GUI ( type sudo virt-manager in a terminal) or from command-line.
Creating VM’s with GUI is pretty similiar to that of VMWare Tools, we right-click on loclhost(qemu), create vm -> point to ISO -> Name VM -> Allocate memort, RAM etc(can be re-configured anytime) -> Install the Guest OS the standard way.
Before you start the VM, just make sure to check the network settings of the VM.
Browse to the virtual machine setting and then NIC, It should point to the Bridge ‘br0’ that we setup for the server.
How to create a VM using the command line
The virt-install tool can be used to provision a VM using the command line.
Here is an example of the command to create a
1. A VM on the local QEMU server
2. Name of the VM – test2
3. 4 GB ram
4. 4 virtual CPUs
5. Disk stored at /home/virtualmachines/test2
6. Disk size 20 GB
7. Networking using bridge – br0
8. OS Linux and version rhel 6
9. OS install binaries under /home/software
10. Root password mnlinux
virt-install –connect qemu:///system –name TestVM2 \
–ram 4024 –vcpus 4 –disk path=/home/virtualmachines/TestVM2,size=20 \
–network=bridge:br0 –os-type=linux –os-variant=rhel6 \
–cdrom /home/software/CentOS-6.4-x86_64-netinstall.iso \
–graphics spice,password=mnlinux –autostart
Central Administration of KVM Infrastructure
The virt-manager can be setup to centrally manage all the BRE KVM infrastructure.
We can setup a central KVM manager and have all the KVM host instances added to it.
This provides a central place to manage all VMs, start stop VMs, reconfigure them, etc
Basic Administration tasks
1. How to find out the virtual machines running on a host
virsh list –all
2. How to find the state of the virtual machines running on a host
virsh list
3. Can I check the list of VMs running on another host remotely
Yes, just pass the location of the remote host to the virsh command. In the below example, I am connecting to another KVM host and listing out the VMs and their state on it.
virsh -c qemu+ssh://root@servername/system list
Id Name State
—————————————————-
4 hdqpdbugs2 running
4. Can I provision a VM via command line.
Yes, using virt-install. Please see the previous section
5. Stop a VM running on a KVM server
virsh shutdown hdqpdtest1
6. Start a VM running on a KVM server
virsh resume hdqpdtest2
7. Save the virtual machine state (VMWare checkpoints)
(To save the current state of servername to a file)
virsh save servername 5623GA.file
(To restore the state of servername to a version that was saved on a file)
virsh restore servername 5623GA.file
Virtual Machine Cloning
The following section has examples on how to use cloning of virtual machines.
1. Clone a virtual machine locally
In this example we clone VM – TestVM into TestVM2.
The VM TestVM needs to be shutdown
And we have the storage for the new VM defined unser /home/virtualmachines
virt-clone -o TestVM -n TestVM2 -f /home/virtualmachines/TestVM2
2. Clone a virtual machine remotely (ie between hypervisors)
In this example, we clone VM – TestVM1 which is on host-1 to a another hypervisor host-2.
Setup KVM on host-2
Login to host-2 and issue these commands.
virt-clone –connect qemu+ssh://root@host-1/system -o TestVM1 -n TestVM2 -f /home/virtualmachines/TestVM2
This command creates a new VM – TestVM2 on host-1. Please note the VM is created on host-1 (not host-2)
Next we have to move the data and config files over to host-2
Copy the VM data files from host-1 to host-2
scp host-1:/home/virtualmachines/TestVM2 TestVM2
Copy the VM config file from host-1 to host-2
scp host-1:/etc/libvirt/qemu/TestVM2.xml /etc/libvirt/qemu/TestVM2.xml
On host-2 issue these commands to register the new VM
virsh define /etc/libvirt/qemu/TestVM2.xml
virsh create /etc/libvirt/qemu/TestVM2.xml
Unregister the TestVM2 from host-1 using the GUI console.