VM Performance Checklist - Before you Complain that your Virtual Machine is Slow

I continue to meet folks who complain that their Virtual Machine performance is slow. Yes, it would be great if VMs somehow were able to self-tune the relationship between themselves and the host OS, but that's sadly not the case.

When you're running an OS within and OS and maintaining a FileSystem within a FileSystem, not to mention sharing a hard drive spindle, there's lots of opportunities for things to go very slowly.

If you're experiencing poor VM performance, I would encourage you to go through a Performance Checklist.

Also, before you start, remember what you goals are. You'll not get your VMs running at 100% of native speed, at least not this year, so just stop aiming for that as a goal.

Here's some more realistic goals:
Ideally Virtual PC performance is at:

• CPU: 96-97% of host
• Network: 70-90% of host
• Disk: 40-70% of host

Try to make all of these changes if you can. If you can't do one or more of these recommendations, then you can't complain. ;)

Virtual PC Performance Checklist

Make sure your Host Operating System's disk is defragmented.
This includes the System Disk (the disk your OS boots off of) as well as the Disk that holds your Virtual Hard Disk File.
For a quick fix, use a single-file defragmenter like Contig from SysInternals. With the Virtual Machine shut down, run Contig -a to analyze single file fragmentation and run without -a to defragment the file.

Run Fewer Applications.
I'm continually amazed when folks complain about VM performance and when I get to their desk I see that they are running Outlook. That 200+megs could be better used by the system. Are you running a VM or checking your email? Consider checking your email on a schedule, or using Outlook Web Access while you work on your VM.
If you have 2 GIG or more of memory, consider running your Host Operating System without a Paging File. This doesn't mean you get to keep 50 applications, plus Outlook running all at once, but it does take the pressure off your Host OS's disk, and you might find things run considerably snappier.

Run the Virtual Machine on a separate spindle.
There's no better tip, as anyone who has run VMs (I've been using VMWare since it was in Beta) will tell you. The #1 bottleneck is disk.
Try to use a 7200RPM or 10000RPM drive for your VM disk
Use USB2 or SATA or Fireware.
If you're using USB2, make sure the Eternal Hard Drive is on it's own USB root hub, all alone. Don't share it with your keyboard, mouse, or webcam.

Optimize your VM for your current task.
Personally, I use and highly recommend Invirtus Virtual Machine Optimizer for this. It's inexpensive if you value your time. Considering getting a site license and actually do the math at how much time it'll save your company when you're trying to convince your boss. I run it over lunch on a VM and move on. You can also do a lot of the work manually if you have the time using tools like XPLite and CrapCleaner (although less so with CrapCleaner if the box is already fresh).
Remove any application that's not needed.
Shut down every service you can possibly get away with.

Enable Hardware Assisted Virtualization
If you've got this on your computer, turn it on. There IS some concern about really sophisticated Trojans that can use this technology for evil, but for me, it's all good as it speeds most Guest Operating Systems (especially non-Microsoft ones) up quite a bit.

Give your Virtual Machines LESS MEMORY
I've found that 512 megs is just about the Ideal Amount of memory for 90% of your Virtual Machines. Don't bother trying to give them 1024 megs, it's just not worth the pressure it'll put on the Host Operating System.

Considering making a custom Windows install for your VMs.
Rather than going to all the effort to REMOVE things, why not create a Windows installation that can be shared across your organization that doesn't include the crap ahead of time. There's a Windows Installation Customizer called nLite that lets you prepare Windows installations so they never include the stuff you don't want. Makes it easier if Solitaire is never installed, eh?

Make sure the Guest Operating System is defragmented.
Jeff likes this free Disk Defragmenter that runs in that "Text Mode" place before Windows really starts up. This allows it to get at files that don't always get defragmented.

Squish your VM Hard Drive.
Again, I use Invirtus so it does this for me, but you can also zero out the free space on your VM hard drive with the Virtual PC Pre-Compactor that comes with Virtual PC when hosting Windows, and there are Linux options for shrinking VM hard drives as well.

Don't use NTFS Compression on the Virtual Machine Hard Drive File in the Host Operating System
NTFS Compression doesn't work on files larger than 4 gigs, and can cause corruption.

Don't Remote Desktop or VNC into Host Operating Systems that are hosting Virtual Machines.
If you're remoting into a machine where THAT machine is running a VM, note that to the Remote Desktop protocol (and VNC) the VM just looks like a big square bitmap that is constantly changing. That guarantees you slow performance. If you can, instead, Remote Desktop into the Virtual Machine itself.

Make sure you've install the Virtual Machine Additions (or Tools, or Utilities, or Whatever)
Virtual PC and VMWare and Parallels all include drivers and tools that improve the performance of your Virtual Machine. They are there for good reason, make sure you've installed them.
Also, if you're running a Virtual Machine created under and older version, like Virtual PC 2004, and you're now running under a newer one, like 2007, pay attention to the upgrade warnings and install the latest drivers and Virtual Machine Additions.

Optimize Painting and the "Perception of Responsiveness"
If you're running a VM, you don't need to have eye candy like menu fades, smooth scrolling or shadows.

• Turn off wallpaper
• Turn off Window Dragging and Shadows under Menus (under Effects in the Display Control Panel).
• Consider removing all effects like fading as well as ClearType.
• Consider running the Classic Theme if you're running an XP VM, or consider "net stop themes" altogether.
• Turn off the Mouse Pointer Shadow in the Mouse Control Panel.
• Turn off Mouse
• Use TweakXP or change the Registry to remove the Menu Delay for the Start Menu and other Menus via the MenuShowDelay setting in HKEY_CURRENT_USER\Control Panel\Desktop.

Invirtus Virtual Machine Optimizer

I've got a pile of demo VMs that Stuart and I and others work on and once they're all setup they tend to get pretty big. After some work recently an 8 gig VM ballooned to nearly 10 gig! There's all sorts of tips and tricks on how to compact VMs, usually by defragging and zero'ing out free space. However, I don't have the time to do this kind of stuff manually, and when I DO run through these processes manually I'm rarely satisfied with the results.

Invirtus VM Optimizer is a clever little tool that solved this particular problem. It's an ISO image that you mount within your guest OS. I used the "Automatic Corporate" edition. It autoruns on mount and I waited for about 20 minutes. After it was done, I ran the Virtual Disk Wizard that comes with Microsoft Virtual PC (Invirtus works on VMWare also) and compacted the 9.9 gig VM down to 3.4 gigs in another 10 minutes.

So, a total of 30 minutes later I had a VM that was 1/3 the size, ran faster and now fits on a single-layer DVD when the original one wouldn't have fit on even a dual-layer one.

Full disclosure: The first Microsoft Virtual Disk Wizard compact operation failed with an obscure error message. I ran chkdsk on both the host and guest OSes, and ran again and everything worked fine. I don't think this had anything do to with Invirtus.

Here's what I can say about Invirtus VM Optimizer - It did exactly what Invirtus said it would do! Always nice to have software work EXACTLY as advertised. Frankly, I would have been thrilled with a 50% size reduction, and was VERY surprised to get 66% as a bonus. I use VMs all the time and the price is a no brainer, US$40 for personal and US$160 for Corporate. Paid for itself in one use.

Access VM Serial Console - Named Pipe

VMTN Discussion Forums: Access VM Serial Console - Named Pipe ...
http://www.dest-unreach.org/socat/

xCAT VMware Workstation (vmCAT) HOWTO

http://xcat.org/doc/vmware-HOWTO.html

This document is for xCAT 1.2.0 and VMware Workstation 5.0.0.

This HOWTO is not about running xCAT in a VMware session (that is no different than any other xCAT management node install). This HOWTO is about xCAT controlling VMware VMs.

xCAT supports VMware VMs like any other machine. rpower, getmacs, rcons using serial console, etc... all function as expected.

vmCAT can be setup the following ways:

Basic. All VMs are running on the xCAT management node. This is not recommend for production environments, however, it is adequate for development environments. E.g. I use them to build diskless images quickly for the rest of the system. (One VMware license required.)
Advanced. All VMs are running on physical nodes other than the xCAT management node. (One VMware license required/node). Follow the Basic notes first before exploring Advanced.
Accelerated. Builds on Advanced by allowing running virtual machines to migrate from node to node.
FAQ:

Q: Why?
A: This HOWTO is a POT (Proof of Technology) to explore grid and utility computing using virtual machines with embedded applications and is not recommended for production environments. YMMV.

Q: Is there a practical use for this HOWTO now?
A: Yes. I use the Basic setup to test and develop xCAT installation support. For image development and testing VMware is a very nice tool.

Q: What about Xen?
A: xenCAT is under development.

Q: What about Windows?
A: For Windows hosts with xCAT in a Linux VMware guest controlling Windows to launch additional Linux VMware guests as xCAT nodes is under development (lapCAT). lapCAT can be used to develop xCAT on airplanes.
Please read through all sets of instructions before setup.

Basic

Install xCAT 1.2.0.

Install VMware Workstation 5.0.0. (Build 13124 tested). Take the defaults.

Run VMware, create VMs. HINT: Create one, test it, and use copyvm.

Recommendations/Requirements:

Virtual Machine names and locations must not have any spaces. Since xCAT can image or network boot any type of OS consider using a generic name, e.g. vnode01, vnode02, etc...
You must use bridged networking.
VMware requires the creation of a virtual disk even in a diskless environment. Go ahead and create the disk, afterwards remove it if you like. It is so small, I would not worry about it.
After the VM is created edit and remove the Audio, CD-ROM, floppy, and USB devices. Remove HD if planning to use diskless.
While editing the VM add a serial port. The serial port must be a named pipe, the path must be unique (e.g. /root/.vmserial/vnode01). The first dropdown box should read "This end is the server.", the second dropdown box should read "The other end is an application.". Click on the "Advanced" button and select "Yield CPU on poll".
Do not put VMs on NFS mount points. VMs on NFS will not suspend (problems with writing out RAM contents). Always use local disk (RAM disk OK).

Consider hardcoding the MAC address. (Optional but required for Advanced and Accelerated setups).

As virtual machines wander from physical machine to physical machine VMware will generate new MAC addresses. This is undesirable. Edit each .vmx file and remove all lines start with ethernet0, then append to the end of the file:

ethernet0.present = "TRUE"
ethernet0.address = "00:50:56:XX:YY:ZZ"
ethernet0.addressType = "static"

Where XX is in the range 00-3f, and YY and ZZ are in the range 00-ff, e.g.:

00:50:56:00:00:01

Create required directorie(s) for the serial port socket. E.g. /root/.vmserial.

mkdir -p /root/.vmserial

Stage1 each VM: HINT: Create one, test it, and use copyvm.

Power on the VM.
Press F2 (quickly).
Right arrow to "boot".
Down arrow to "Network boot".
Press '+' until "Network boot" is at the top of the list.
Press F10 to save.

If all your VMs are going to be identical use the copyvm command (Optional). The virtual machine must be turned off, e.g.:

# cd /root/vmware
# copyvm vnode01 vnode05

copyvm: vnode05 created from vnode01

Update /opt/xcat/etc/mac.tab with:

vnode5-eth0 00:50:56:36:7e:09

You can ignore the "Update" message and use getmacs instead.

NOTE: copyvm will create a random VMware approved static MAC. A check against /etc/dhcpd.conf and $XCATROOT/etc/mac.tab for duplicates is also performed.

NOTE: copyvm will fail if any of the support files (e.g. HD images) are not contained within the VM directory. Normally not a probelm.

VMs have virtual physical consoles that need to be redirected.

Edit $XCATROOT/etc/site.tab, add/edit vmwdisplay and set to an X display to monitor the VMware VGA consoles, e.g.:

vmwdisplay mercury:21

It is recommended that the X display for VMware VGA console management be a VNC session so that remote management is possible. E.g., if the display is mercury:21 then type from mercury as root:

cd /root
mkdir .vnc
cd .vnc
cp $XCATROOT/build/vnc/xstartup .
vncpasswd
vncserver :21 -geometry 1024x768 -depth 24

To remotely access the VGA consoles use any VNC client, if using Linux type:

vncviewer -shared display

e.g.:

vncviewer -shared mercury:21

To prevent VMware from prompting/warning append to /etc/vmware/config (required):

msg.autoAnswer = "TRUE"

Add each VM node to the following xCAT tables like any other node:

$XCATROOT/etc/nodelist.tab
$XCATROOT/etc/nodetype.tab
$XCATROOT/etc/conserver.tab

For each VM node add an entry in $XCATROOT/etc/nodehm.tab, e.g.:

vnode01 vmware,vmware,NA,NA,NA,conserver,NA,NA,vmware,pxe,pcnet32,vnc,N,NA,NA,57600

For each VM node add an entry in $XCATROOR/etc/vmware.tab:

nodename vmware_host,VMX_path,serial_socket_path

e.g.,

vnode01 mercury,/root/vmware/vnode01/vnode01.vmx,/root/.vmserial/vnode01

For each VM node add an entry in $XCATROOT/etc/conserver.cf, e.g.:

vnode01:|conserver.vmserial vnode01::&:

Restart conserver:

service conserver restart

At this point the VMs should behave like any other node (assuming that xCAT is setup correctly). Test with:

getmacs noderange
makedhcp noderange
winstall singlenode

There is one exception to the above step. There is a new rpower function (suspend). You can suspend VMs with:

rpower noderange suspend

To resume:

rpower noderange on

Advanced

Complete the Basic setup first. Test.

Use xCAT to diskfull install (diskless not tested) any nodes that will act as a virtual node host. Install all packages.

Disable swap on each node. (Optional, but recommended for performance reasons, YMMV).

Install VMware Workstation on each virtual node host.

To automate the installing of VMware Workstation on a large number of nodes do the following:

Obtain the proper number of VMware Workstation licenses (one/node).
From the management node copy the /usr/bin/vmware-config.pl script to /root and edit:

# cp /usr/bin/vmware-config.pl /root
# vi /root/vmware-config.pl

Search for "# NAT networking", type:

[ESC]/\# NAT networking

Then change:

# NAT networking
$answer = get_answer('Do you want to be able to use NAT networking '
. 'in your virtual machines? (yes/no)', 'yesno', 'yes');

to

# NAT networking
$answer = get_answer('Do you want to be able to use NAT networking '
. 'in your virtual machines? (yes/no)', 'yesno', 'no');

Next search for "show_EULA", type:

[ESC]/show_EULA()

Then change:

show_EULA()

to

#show_EULA()

The above changes allow VMware to be configured without prompts. Since you already installed VMware manually as part of the Basic setup, you have already agreed to the EULA. The other change has NAT networking disabled by default.

Create a script to do the following, you will obviously need to customize for your environment:

rpm -i VMware-workstation-5.0.0-13124.i386.rpm
scp managementnode:/root/vmware-config.pl /usr/bin
vmware-config.pl -d -c
mkdir /root/.vmware /root/vmware /root/.vmserial
echo "msg.autoAnswer = \"TRUE\"" >>/etc/vmware/config
echo "pref.tip.startup = \"FALSE\"" >>/root/.vmware/preferences
scp managementnode:/root/.vmware/license.ws.5.0 /root/.vmware

That last command assumes that you have a site license allowing one instance/physical node.
Alternatively just install VMware manually per node.


Test display. From any target physical node export display to vmwdisplay entered into $XCATROOT/etc/site.tab, e.g.:

export DISPLAY=mercury:21

Then run vmware manually (if you installed VMware manually enter the serial number), e.g.:

vmware

NOTE: If you get a "Xlib: connection to "host:display" refused by server" error, then you need run xhost + from the host:display session, e.g. mercury:21.

Create virtual machines on the xCAT management node following the complete Basic setup. Test a few.

NOTE: VMs must not be on NFS exported directories (migration will fail).
NOTE: VMs must not be on NFS mount points (suspend will fail).
NOTE: The VMs must have hardcoded static MACs (see Basic setup).

Migrate physical machines:

rpower noderange off (required)

For each virtual machine type:

rmigrate vnode pnode

Where vnode is a VM defined in $XCATROOT/etc/vmware.tab and pnode is the physical node that this VM will run on, e.g.:

rmigrate vnode01 node01
rmigrate vnode02 node01
rmigrate vnode03 node02
rmigrate vnode04 node02
...

NOTE: migration will remove the virtual machine from the management node. You can always migrate back if needed.

Test VMs on virtual node host, e.g.:

Open a VNC connection to the display defined as vmwdisplay in $XCATROOT/etc/site.tab. If not using VNC monitor the physical display that VM VGA consoles will be redirected.
Power up the first node and watch:

rpower vnode01 on

Serial console redirection support. Since the VMs will be running on different physical nodes it will be required to setup a Conserver server on each node.

For each node create the required directories for the serial port sockets. E.g. /root/.vmserial.

mkdir -p /root/.vmserial

For each node setup conserver.cf. Type from each node (not the management node):

cp $XCATROOT/etc/conserver.cf /etc/conserver.cf

Edit conserver.cf

Remove physical node lines.
Define any missing VMs.
It is OK to setup entries for VMs not destine for this physical node. (It may actually be desired and easier, all nodes can have the same conserver.cf).
Change trusted: to IP of management node.
Example conserver.cf:

LOGDIR=/var/log/consoles
vnode01:|conserver.vmserial vnode01::&:
vnode02:|conserver.vmserial vnode02::&:
vnode03:|conserver.vmserial vnode03::&:
vnode04:|conserver.vmserial vnode04::&:
%%
trusted: 199.88.179.26


Setup Conserver init scripts.

For RH, type:

cp $XCATROOT/rc.d/conserver /etc/rc.d/init.d

For SuSE type:

cp $XCATROOT/rc.d/conserver.suse /etc/init.d/conserver
cd /usr/sbin
ln -s -f /etc/init.d/conserver rcconserver

Edit the conserver init script (/etc/rc.d/init.d/conserver or /etc/init.d/conserver):

Change:

CONCONFIG=$CONSPREFIX/etc/conserver.cf

to

CONCONFIG=/etc/conserver.cf

Startup Conserver:

RH:

service conserver start
chkconfig --level 345 conserver on

SuSE:

rcconserver start
chkconfig --level 345 conserver on

Test from management node:

console -Mpnode vnode

Where vnode is a VM defined in $XCATROOT/etc/vmware.tab and pnode is the physical node that this VM will run on. (Use ctrl-E c to exit).

From the management node edit $XCATROOT/etc/conserver.tab. For each remote virtual node (VM) change "localhost" to the physical node name hosting that VM, example conserver.tab:

node01 localhost,node01
node02 localhost,node02
node03 localhost,node03
node04 localhost,node04
node05 localhost,node05
vnode01 node01,vnode1
vnode02 node01,vnode2
vnode03 node02,vnode3
vnode04 node02,vnode4
vnode05 node03,vnode5
vnode06 node03,vnode6
vnode07 node04,vnode7
vnode08 node04,vnode8

For each VM node add/edit an entry in $XCATROOT/etc/nodehm.tab, e.g.:

vnode01 vmware,vmware,NA,NA,NA,conserver,NA,NA,rcons,pxe,pcnet32,vnc,N,NA,NA,57600

The only difference between Basic and Advanced is the MAC address collection method. It must be rcons and screen scraped from the serial console. The vmware method is for local virtual nodes only.

Alternatively just edit $XCATROOT/etc/mac.tab manually since the MACs are located in human readable .vmx files, or collect MACs using the vmware method before migration.

At this point the VMs should behave like any other node (assuming that xCAT is setup correctly).

New functionality. You may use the rmigrate command to move any virtual node (VM) to any physical node with if the physical node has been properly prepped to support VMware.

NOTE: The virtual node (VM) must be powered off first.
NOTE: To move running VMs read the Accelerated setup.
NOTE: Edit $XCATROOT/etc/site.tab and change bufferedcons to no. To avoid problems with migration.

Accelerated

Accelerated setup is built on Basic and Advanced setups.

NOTE: This setup is to support the migration of active virtual machines.

Disable buffered console support. At this time support for roaming active VMware serial redirection is not supported. Edit $XCATROOT/etc/site.tab and change bufferedcons to no.


Kill all wcons and rcons sessions, if recently switching off bufferedcons, e.g.:

killall screen (careful)


Running VMs cannot be suspended and restarted on different processor types. At this time no checking is performed. E.g. a VM started on a PIII will fail to start on a P4.


At this time migration does not check that the target node has the proper resources to support the VM. E.g. memory and disk space.


Support

http://xcat.org


Egan Ford
egan@us.ibm.com
November 2005