vSphere High Performance Cookbook: A cookbook is the ideal way to learn a tool as complex as vSphere. Through experiencing the real-world recipes in this tutorial you'll gain deep insight into vSphere's unique attributes and reach a high level of proficiency.

To step through this recipe, you need a running ESXi Server, a Virtual Machine, vCenter Server, and a working installation of the vSphere Client. No other prerequisites are required.

Let's get started:

The VMM determines a set of possible monitor modes to use, and then picks one to use as the default monitor mode, unless something other than Automatic has been specified. The decision is based on:

There are three valid combinations for the monitor mode, as follows:

BT, HV, and HWMMU are abbreviations used by ESXi to identify each combination.

When a virtual machine is powering on, the VMM inspects the physical CPU's features and the guest operating system type to determine the set of possible execution modes. The VMM first finds the set of modes allowed. Then it restricts the allowed modes by configuration file settings. Finally, among the remaining candidates, it chooses the preferred mode, which is the default monitor mode. This default mode is then used if you have left Automatic selected.

For the majority of workloads, the default monitor mode chosen by the VMM works best. The default monitor mode for each guest operating system on each CPU has been carefully selected after a performance evaluation of available choices. However, some applications have special characteristics that can result in better performance when using a non-default monitor mode. These should be treated as exceptions, not the rule.

The chosen settings are honored by the VMM only if the settings are supported on the intended hardware. For example, if you select Use software instruction set and MMU virtualization for a 64-bit guest operating system running on a 64-bit Intel processor, the VMM will choose Intel VT-x for CPU virtualization instead of BT. This is because BT is not supported by the 64-bit guest operating system on this processor.

The virtual CPU consists of the virtual instruction set and the virtual memory management unit (MMU). An instruction set is a list of instructions that a CPU executes. The MMU is the hardware that maintains the mapping between the virtual addresses and the physical addresses in the memory.

The combination of techniques used to virtualize the instruction set and memory determines the monitor execution mode (also called the monitor mode). The VMM identifies the VMware ESXi hardware platform and its available CPU features, and then chooses a monitor mode for a particular guest operating system on that hardware platform. The VMM might choose a monitor mode that uses hardware virtualization techniques, software virtualization techniques, or a combination of hardware and software techniques.

We always had a challenge in hardware virtualization. x86 operating systems are designed to run directly on the bare metal hardware, so they assume that they have full control on the computer hardware. The x86 architecture offers four levels of privilege to operating systems and applications to manage access to the computer hardware: ring 0, ring 1, ring 2, and ring 3. User-level applications typically run in ring 3, the operating system needs to have direct access to the memory and hardware, and must execute its privileged instructions in ring 0.

Binary translation allows the VMM to run in ring 0 for isolation and performance, while moving the guest operating system to ring 1. Ring 1 is a higher privilege level than ring 3 and a lower privilege level than ring 0.

VMware can virtualize any x86 operating systems by using a combination of binary translation and direct execution techniques. With binary translation, the VMM dynamically translates all guest operating system instructions and caches the results for future use. The translator in the VMM does not perform a mapping from one architecture to another; that would be emulation not translation. Instead, it translates from the full unrestricted x86 instruction set issued by the guest operating system to a subset that is safe to execute inside the VMM. In particular, the binary translator replaces privileged instructions with sequences of instructions that perform the privileged operations in the virtual machine rather than on the physical machine. This translation enforces encapsulation of the virtual machine while preserving the x86 semantics as seen from the perspective of the virtual machine.

Meanwhile, user-level code is directly executed on the processor for high-performance virtualization. Each VMM provides each virtual machine with all of the services of the physical system, including a virtual BIOS, virtual devices, and virtualized memory management.

In addition to software virtualization, there is support for hardware virtualization. This allows some of the work of running virtual CPU instructions to be offloaded onto the physical hardware. Intel has the Intel Virtualization Technology (Intel VT-x) feature. AMD has the AMD Virtualization (AMD-V) feature. Intel VT-x and AMD-V are similar in aim but different in detail. Both designs aim to simplify virtualization techniques.

To step through this recipe, you need a running ESXi Server, a Virtual Machine, and a working installation of vSphere Client. No other prerequisites are required.

Let's get started:

The CPU scheduler uses processor topology information to optimize the placement of vCPUs onto different sockets.

The CPU scheduler spreads the load across all the sockets to maximize the aggregate amount of cache available.

Cores within a single socket typically use a shared last-level cache. Use of a shared last-level cache can improve vCPU performance if the CPU is running memory-intensive workloads.

By default, the CPU scheduler spreads the load across all sockets in under-committed systems. This improves performance by maximizing the aggregate amount of cache available to the running vCPUs. For such workloads, it can be beneficial to schedule all of the vCPUs on the same socket, with a shared last-level cache, even when the ESXi host is under committed. In such scenarios, you can override the default behavior of the spreading vCPUs across packages by including the following configuration option in the virtual machine's VMX configuration file, sched.cpu.vsmpConsolidate=TRUE. However, it is usually better to stick with the default behavior.

To step through this recipe, you need a running ESXi Server, a couple of CPU-hungry virtual machines, VMware vCenter Server, and a working installation of vSphere Client. No other prerequisites are required.

Let's get started:

In the following screenshot you will see the high ready time for the virtual machine:

Notice the amount of CPU this virtual machine is demanding and compare that to the amount of CPU usage the virtual machine is actually being able to get (usage in MHz). The virtual machine is demanding more than it is currently being allowed to use.

Notice that the virtual machine is also seeing a large amount of ready time.

Bad performance is when the users are unhappy. But that's subjective and hard to measure. We can measure other metrics easily, but they don't correlate perfectly with whether user's expectations are met. We want to find metrics that correlate well (though never perfectly) with user satisfaction. It's always the case that the final answer to "Is there a performance problem?" is subjective, but we can use objective metrics to make reasonable bets, and decide when it's worth asking the users if they're satisfied with the performance.

A vCPU is in ready state when the vCPU is ready to run (that is, it has a task it wants to execute) but is unable to run because the vSphere scheduler is unable to find physical host CPU resources to run the virtual machine on. One potential reason for elevated ready time is that the virtual machine is constrained by a user-set CPU limit or resource pool limit, reported as max limited (MLMTD). The amount of CPU denied because of a limit is measured as the metric max limited (MLMTD).

Ready time is reported in two different values between resxtop/esxtop and vCenter Server. In resxtop/esxtop, it is reported in an easily-understood percentage format. A figure of 5 percent means that the virtual machine spent 5 percent of its last sample period waiting for available CPU resources (only true for 1-vCPU VMs). In vCenter Server, ready time is reported as a time measurement. For example, in vCenter Server's real-time data, which produces sample values every 20,000 milliseconds, a figure of 1,000 milliseconds is reported for a 5 percent ready time. A figure of 2,000 milliseconds is reported for a 10 percent ready time.

                                                 Metric Value (In Millisecond)
Metric Value (In Percent) = ------------------------------------------------	x 100
                                                 Total Time of Sample Period
                            (By default 20000 ms in vCenter for real-time graphs)

Although high ready time typically signifies CPU contention, the condition does not always warrant corrective action. If the value for ready time is close in value to the amount of time used on the CPU, and if the increased ready time occurs with occasional spikes in CPU activity but does not persist for extended periods of time, this might not indicate a performance problem. The brief performance hit is often within the accepted performance variance and does not require any action on the part of the administrator.

To step through this recipe, you need a running ESXi Server, a running virtual machine, VMware vCenter Server, and a working installation of vSphere Client. No other prerequisites are required.

Let's get started:

There are three different HT sharing methods, as follows:

The following table elaborates the three methods of core sharing:

These options have no effect on the fairness or CPU time allocation. Regardless of a virtual machine's hyperthreading settings, it still receives CPU time proportional to its CPU shares, and constrained by its CPU reservation and CPU limit values.

If there are running VMs on the same virtual infrastructure cluster with different numbers of vCPU (for example, one vCPU and two vCPUs) then there is a good chance that one vCPU of your dual vCPU VM can work alone on one physical CPU and the other vCPU has to share a physical CPU with another VM. This causes tremendous synchronization overhead between the two vCPUs (you don't have this in physical multi-CPU machines because this sync is hardware based) which can cause the system process within the VM to go up from 50 percent to 100 percent CPU load.

To step through this recipe, you need a running ESXi Server, a couple of running CPU-hungry virtual machines, a SSH client (Putty), vCenter Server, and a working installation of vSphere Client. No other prerequisites are required.

The following table elaborates on Esxtop CPU Performance Metrics:

To spot CPU overcommitment there are a few CPU resource parameters which you should monitor closely. Those are:

This example uses esxtop to detect CPU overcommitment. Looking at the pCPU line near the top of the screen, you can determine that this host's two CPUs are 100 percent utilized. Four active virtual machines are shown, Res-Hungry-1 to Res-Hungry-4. These virtual machines are active because they have relatively high values in the %USED column. The values in the %USED column alone do not necessarily indicate that the CPUs are overcommitted. In the %RDY column, you see that the three active virtual machines have relatively high values. High %RDY values, plus high %USED values, are a sure indicator that your CPU resources are overcommitted.

From the CPU view, navigate to a VM and press the E key to expand the view. It will give a detailed vCPU view for the VM. This is important because at a quick level, CPU ready as a metric is best referenced when looking at performance concerns more broadly than a specific VM. If there is high ready percentage noted, contention could be an issue, particularly if other VMs show high utilization when more vCPUs than physical cores are present. In that case, other VMs could be leading to high ready time on a low idle VM. So, long story short, if the CPU ready time is high on VMs on a host, it's time to verify that no other VMs are seeing performance issues.

You can also use vCenter performance chart to spot the CPU overcommitment, as follows:

Now you will see the ready time and the used time in the graph and you can spot the overcommitment. The following screenshot is an example output:

The following example shows that the host has high used time.

Although high ready time typically signifies a CPU contention, the condition does not always warrant corrective action. If the value for ready time is also accompanied by high used time then it might signify that the host is overcommitted.

So used time and ready time for an host might signal contention. However, the host might not be over-committed, due to workload availability.

There might be periods of activity and periods that are idle. So the CPU is not over-committed all the time. Another very common source of high ready time for VMs, even when pCPU utilization is low, is due to storage being slow. A vCPU, which occupies a pCPU, can issue a storage I/O and then sits in the WAIT state on the pCPU blocking other vCPUs. Other vCPUs accumulate ready time; this vCPU and this pCPU accumulate wait time (which is not a part of the used or utilized time).

To step through this recipe, you need a running ESXi Server, a couple of running CPU-hungry virtual machines, vCenter Server, and a working installation of vSphere Client. No other prerequisites are required.

To spot CPU overcommitment in the guest OS there are two CPU resource parameters which you should monitor closely as follows:

The preceding example shows the high usage and used value. We can see it is 100 percent.

The preceding example shows that after the CPU increase in the VM, the percentage of CPU usage dropped down to 52 percent.

So for a SMP VM if you see it is the high CPU resources demanding, it may happen that either the application is single threaded or the guest operating system is configured with uniprocessor HAL.

Many applications are written with only a single thread of control. These applications cannot take advantage of more than one processor core.

In order for a virtual machine to take advantage of multiple vCPUs, the guest operating system running on the virtual machine must be able to recognize and use multiple processor cores. If the virtual machine is doing all of its work on vCPU0, the guest operating system might be configured with a kernel or a HAL that can recognize only a single processor core.

You have two possible approaches to solving performance problems related to guest CPU saturation:

Adding CPU resources is often the easiest choice, particularly in a virtualized environment. If a virtual machine continues to experience CPU saturation even after adding CPU resources, the tuning and behavior of the application and operating system should be investigated.

To step through this recipe, you need a running ESXi Server, a couple of running CPU hungry virtual machines, vCenter Server, and a working installation of vSphere Client. No other prerequisites are required.

Let's get started:

Now if there is a lower limit configured on the VM and at the same time if it is craving for a resource, then you will see a high ready time and a low used metric. An example of what it may look like is given in the following image:

Look at the preceding example and see when the VM is craving for more CPU resource, if you put a limit on top of it, then it will experience a high ready time and a low used time. Here in the above example this VM is set with a limit of 500MHz.

Now to rectify this, we can change the limit value and the VM should perform better with a low ready time and a high used value.

Now the VM should look and perform as shown in the following screenshot:

To step through this recipe, you need a running ESXi Server, a couple of running CPU-hungry virtual machines, and a SSH Client (for example, Putty). No other prerequisites are required.

Let's get started:

In the preceding example, you can see our pCPU 0 and pCPU 1 are heavily being used (100 percent and 73 percent UTIL respectively) and it shows the following figure:

Now in the preceding example, you see that the %Used value for the four CPU-hungry virtual machines are pretty high.

Also look at the %RDY screen, and you will see a high ready time, which indicates a performance problem.

The following list is a quick explanation for each of these metrics:

To step through this recipe, you need a running ESXi Server, a couple of running virtual machines, and a working installation of vSphere Client. No other prerequisites are required.

Let's get started: