Why Guest OS Task Manager is Showing Different Value Compare to vSphere Performance Monitor?

Demystifying CPU States in vCPU World

Have you experienced a situation where your guest OS task manager is showing different value compare to vSphere performance monitor? Or you get a request for additional vCPU from the application team which uses your VM because they see their VM utilizing almost all vCPU they have, but when you check vCPU usage of that VM in your vSphere web client, it only shows low utilization? Is there something wrong? Before you think there's something wrong with vSphere performance monitor, read this article to understand what's causing that situation.

Figure 1. Windows task manager shows ~100% CPU Utilization

Before going further, let me first describe the situation clearer. Figure 1 and 2 are coming from the same Virtual Machine, perf-worker-01b. The first figure shows Windows Task Manager where the CPU utilization is hitting 100% for most of the last 4 minutes. The second figure shows vSphere performance monitor which taken about the same time as Figure 1, and this figure reveals that for the last 4 minutes, VM CPU usage is only around 50%. FYI, I actually ran a CPU benchmark tool on perf-worker-01b for about 30 minutes, and for most of the time in that period, Windows task manager showed 100% CPU utilization, while vSphere performance monitor showed around 50% CPU usage. Why vSphere performance monitor only showed 50% CPU usage when Windows task manager showed ~100%?

Figure 2. vSphere performance monitor shows ~50% CPU Usage

To answer that question, first I would suggest you to read this VMware technical whitepaper with title The CPU Scheduler in VMware vSphere 5.1 here and read the Terminology part which talks about vCPU or world state. An easier to understand graphic which explains about CPU state can also be found in one of VMware Hands-on Labs lab manual. Check Module 1 in HOL-1704-SDC-1 lab manual here and scroll down until you find the following figure which shows a vCPU or World will always in one of four state: WAIT, RDY, CSTP, or RUN.

Figure 3. vCPU or World CPU State
Source: http://docs.hol.vmware.com/HOL-2017/hol-1704-sdc-1_html_en/

Still coming from the same lab manual, the definition of the four states are:

• WAIT: This can occur when the virtual machine's guest OS is IDLE (waiting for work), or the virtual machine could be waiting on vSphere tasks. Some examples of vSphere tasks that a vCPU may be waiting on include waiting for I/O to complete or waiting for ESX level swapping to complete. These non-idle vSphere system waits are called VMWAIT.
• Ready (RDY): A CPU is in the Ready state when the virtual machine is ready to run but unable to run because the vSphere scheduler is unable to find physical host CPU resources to run the virtual machine on. One potential reason for elevated Ready time is that the VM is constrained by a user-set CPU limit or resource pool limit, reported as max limited (MLMTD).
• CoStop (CSTP): Time the vCPUs of a multi-vCPU virtual machine spent waiting to be co-started. This gives an indication of the co-scheduling overhead incurred by the virtual machine.
• RUN: Time the virtual machine was running on a physical processor.

The other principle that you need to remember is:

WAIT + RDY + CSTP + RUN = 100% per World

As you can see here, the state where a VM gets its turn to use a physical CPU is actually only during two states: RUN and IDLE (part of WAIT). A guest OS inside a VM only knows or sees the time when it gets its turn to use physical CPU, and don't know or realize when it is in "paused" state. Back to the first situation, Windows task manager shows ~100% CPU utilization, but only for part of CPU state times. Because this is a fully utilize VM, we can assume that ~100% CPU utilization should comes from only one CPU state which is RUN state, where RUN state in a situation where CPU contention happened most likely will be much less than 100% of total CPU state times (WAIT + RDY + CSTP + RUN). This is the reason why from the point of view of the guest OS, the CPU utilization is ~100%, but from the point of view of the Virtual Machine is only ~50%.

Figure 4. CPU State Times for perf-worker-01b VM

OK, let's check further. In Figure 4, I plotted the four CPU states for perf-worker-01b during CPU contention. Here we can see the time for that VM's vCPU in RUN state is about the same as in RDY state, which is around 10,000 ms, while WAIT and CSTP is about 0 ms. I add IDLE metric which mostly 0 ms to show that the VM is not in idle state.

• This value comes from vSphere performance monitor, which each cycle is 20 seconds (20,000 ms). Taking example for the last state, the value for RUN is 9,963 ms, RDY is 10,038 ms, while WAIT and CSTP are 0 ms. If we sum up all value for the last CPU state, 0 + 10,038 + 0 + 9,963, it equals to 20,001 ms. This is consistent with the principle WAIT + RDY + CSTP + RUN = 100% per World.
• VM spent its time around 50% in RUN state and 50% in RDY state, means VM actually can only utilize 50% from the total time. This explains why even Windows task manager shows 100% CPU utilization, CPU usage in vSphere performance monitor shows 50%.

Figure 5. perf-worker-01b VM CPU Usage vs. CPU Demand

One last thing that I want to show you is the comparison of CPU Usage and CPU Demand for perf-worker-01b VM. Remember that CPU Demand is the CPU counter of what a VM really ask. If there's no CPU contention, then CPU Demand should be about the same value as CPU Usage. But here in our case, CPU Usage is only half of CPU Demand. That's once again consistent in showing since RUN is only 50% from total time, then VM could only get 50% from what it demands.

Figure 6. perf-worker-01b without CPU contention

If I managed to remove CPU contention on the host where perf-worker-01b is running, see that now RDY value drops to nearly 0 ms and RUN spikes to nearly 20,000 ms; and CPU Usage is about the same as CPU Demand. We can see that the CPU Utilization in Windows task manager is tally with CPU Usage (%) in vSphere performance monitor which is both has value ~100%. Seeing this, I hope now you understand that there's situation where CPU utilization/usage value can be different from the point of view of a VM and guest OS. I hope that's informative for you and I'll see you again next time.