Performance of Citrix Presentation Servers hosted on VMware ESX can have a number of areas that can cause performance issues. Below are steps that can assist in tuning the environment for maximum performance.
1) Use a suitable virtualisation product.
Use VMware ESX, not VMware Server or GSX for a production environment.
2) Do not over subscribe ESX host resources
Never create more Citrix virtual machines requiring more resources than are available to allocate.
e.g. an ESX host with 20GB RAM and Dual processor Quad Core CPUs should host not more than 4 Citrix virtual machines with Dual vCPU and 4GB RAM
Typically Citrix servers would be created as a separate cluster of ESX hosts.
3) Set Virtual Machine Reservations
Set reservations for 100% CPU and 100% of RAM. This has the effect of enforcing the point above, and has the additional
benefit that once the virtual machine is shutdown and then started it will no longer have a virtual swap (.vswp) file, or a zero byte swap file, depending on the version.
4) Give I/O the Best Possible Chance
Citrix Servers are I/O intensive. Host them on a RAID 10 VMFS datastore and ensure all I/O components are capable of sustaining the throughput, avoiding bottlenecks.
Typically this would be a Fibre Channel SAN with FC/SCSI disks. Avoid SATA for production servers.
5) Assign Appropriate Resources
Citrix servers are CPU intensive and will require dual processor. If the ESX host resources are not over subscribed, CPU scheduling will not cause a bottleneck. Typically configuration:
- 2 x vCPU
- 4GB RAM
- Single SCSI Controller (SCSI IDs 0:0, 0:1, 0:2...)
- Remove the floppy drive
6) Scale Out, Not Up
Give Citrix virtual machines 4GB of memory. A 32 bit Windows machine accessing more than 4GB of memory is nothing more than a hack. Use more virtual machines not bigger virtual machines
7) Don't Install All VMware Tools Components
Dont install VMware Tools components that can impede performance. These include:
- Descheduled Time Accounting (experimental)
- Filesystem Sync Driver (LGTO_SYNC - can halt I/O disrupting services/users)
- Shared Folders (vmhgfs – not supported on ESX)
An unattended install would have the following command line:
msiexec -i "VMware Tools.msi" ADDLOCAL=ALL REMOVE="Hgfs,Sync,vmdesched" /qn reboot=supress
Note: Previously the Memory Control Driver (vmmemctl) was recommended to be removed. After discussions with VMware, this should be retained as it will only be used when the host is under memory pressure and will intelligently select memory to balloon in conjunction with the operating system. The alternative is the host paging out virtual machine memory to disk, which may be highly active, possibly suspending a virtual machine for milliseconds at a time. This situation should not arise if suitable resources are available, but an unexpected event such as a host failure may cause this.
8) Never Use Snapshots on Production Virtual Machines
Performance will suffer badly. Stage and test changes, don't rely on rolling back to a snapshot.
9) Keep ESX Patched
There have been some recent patches to ESX 3.5 that appear to have improved the performance of Windows virtual machines. Regardless it's good practice.
10) Monitor CPU Ready Times
Monitor CPU Ready (%RDY) times in esxtop. Typically they should be under 5%, if they are any higher the host resources may be over subscribed.
Also ProcessExplorer from SysInternals/Microsoft is good to identify issues such as svchost/dllhost subprocesses and high hardware interrupts.
11) Use Standard Citrix Performance Tuning
Use all the standard performance tuning used in a physical Citrix server environment such as:
- NtfsDisableLastAccessUpdate
- DisablePagingExecutive
- DisableDHCPMediaSense
- MaxCmds
- MaxThreads
- MaxCollectionCount
12) Align Partitions
Align VMFS and NTFS file system partitions for performance benefits.
http://www.vmware.com/pdf/esx3_partition_align.pdf
13) Isolate Performance Issues
Try and isolate performance issues to a particular application or configuration. Examples of performance issues include:
Microsoft Outlook 2007
There are a number of known performance issues with Outlook 2007. Test the server environment with Outlook 2003 with the latest Service Pack to determine if the performance issue is application specific. A number of performance hotfixes have been released for Outlook 2007:
Description of the update for Outlook 2007: April 13, 2007
http://support.microsoft.com/kb/933493
This update fixes the issue that is described in the following Microsoft Knowledge Base article:
932086 (
http://support.microsoft.com/kb/932086/) You may experience performance problems when you are working with items in a large .pst file or in a large .ost file in Outlook 2007
NOTE: This hotfix appears to improve Outlook 2007 performance when using PST/archive files
Description of the 2007 Office hotfix package: February 26, 2008
http://support.microsoft.com/kb/948950
This hotfix fixes the following issues that were not previously documented in a Microsoft Knowledge Base article:
- A server that is running Windows Server 2003 experiences high CPU usage, and Microsoft Office Outlook 2007 runs very slowly.
This issue occurs when many users run Outlook 2007 in Terminal Server client sessions that are connected to the Windows Server 2003-based server.
Note: This hotfix appears detrimental to performance despite the comments in the article.
The use of Personal Folders or Archives (.PST files) is unsupported and causes performance issues:
Personal folder files are unsupported over a LAN or over a WAN link
http://support.microsoft.com/kb/297019
Network Stored PST files ... don't do it!
http://blogs.technet.com/askperf/archive/2007/01/21/network-stored-pst-files-don-t-do-it.aspx
Update: Service Pack 2 resolves a preview pane utilisation issue in Outlook.
Project VRC Phase II version 1.0
http://www.projectvrc.nl/
HAL Selection
If the incorrect HAL is selected a VM may show signs of high processor utilisation and sluggish response. Try a single vCPU with the Standard PC HAL as a baseline to troubleshoot performance issues.
Switch Flow Control
Look for switch 802.3x Flow Control pauses on the switch (e.g. Cisco):
show flowcontrol
Disable on a port with (e.g. Cisco):
flowcontrol receive off
flowcontrol send off
Hardware Interrupts
Adding multiple SCSI HBA's has been seen to cause significant processor utilisation through hardware interrupts. This can be seen in Process Explorer. If assigning multiple hard disks assign them to the same HBA e.g. use SCSI ID's 0:0 and 0:1, not 0:0 and 1:0
Also, just having the following entries in the .vmx file without actually having the device connected (even with floppy or CD-ROM set as a client device) has been seen to create ~40% CPU utilisation from hardware interrupts:
floppy0.fileName = "/dev/fd0"
ide0:0.fileName = "/usr/lib/vmware/isoimages/windows.iso"
Remove the floppy drive from the virtual machine, and remove any floppy0 references from the .vmx. Realistically you are never going to need it.
These hardware interrupt examples may all be symptoms of IRQ sharing with USB devices. Refer to the tuxyturvy.com and VMware article below. If the issues affects the ESX host the USB modules can be removed using the following commands:
modprobe -r usb-uhci
modprobe -r ehci-hcd
To avoid loading these modules in future comment out the following lines in /etc/modules.conf:
# alias usb-controller usb-uhci
# alias usb-controller1 ehci-hcd