VCAP5-DCA-Objective 3.2–Optimize Virtual Machine Resources

For this objective I used the following documents:

  • Documents listed in the Tools section

Objective 3.2 – Optimize Virtual Machine Resources



  • Compare and contrast virtual and physical hardware resources
    • At its most basic form, virtual resources allow you to overcommit your virtual machines. Virtual resources are the makeup of physical resources and allow flexibility
    • Over commitment of virtual resources is a good idea as long as its managed well. Configuring X amount of virtual resources on a virtual machine does not meant the commensurate physical resources will be used, which is where the flexibility of virtual resources comes in
    • While virtual hardware resources add overhead that physical resources do not, using virtual resources you can get the most out of the physical resources
          • I don’t have a lot more to say about this. I didn’t see any reference to this topic in the documentation and I believe the basic comparisons of physical vs. virtual still apply. Please, if anyone has a reference from the documentation please let me know in the comments
  • Identify VMware memory management techniques
    • With the introduction of vSphere 5 new management techniques were introduced to further optimize memory management
    • Hosts allocate memory to virtual machines based on their most recent working set size and relative shares to the resource pool. The working set size is monitored for 60 seconds (default period). This interval can be changed by modifying the advanced setting Mem.SamplePeriod
    • A cool new feature introduced with vSphere 5 is VMX Swap. I’ve explained this in previous objectives, but I’ll go through it real quick. Every VM has memory overhead, and that memory is reserved during power on. A chunk of that memory is reserved for the VMX process. Instead of using physical memory for the VMX process, VMX swap files are created during power on and memory needed for the VMX process is swapped to the VMX swap files instead of using memory. This feature is enabled by default and is invoked when the host memory is overcommitted
    • ESXi memory sharing allows virtual machines running the same operating systems and/or applications to, when possible, share the memory pages. This technique is called Transparent Page Sharing (TPS). You can set advanced settings per-host to specify a custom interval of how often the host scans for memory and host much CPU resources to consume doing it. Those two settings are Mem.ShareScanTime and Mem.ShareScanGHz. The defaults are 60 (minutes) and 4 respectively
    • Memory Compression is a technique that is used right before pages start getting swapped to disk. Memory pages that can be condensed into 2KB or less are stored in what’s called the virtual machine’s compression cache. You can set the maximum size of the compression cache with the Mem.MemZipMaxPct advanced setting. The default is 10%. If you want to enable/disable memory compression us the Mem.MemZipEnable advanced setting. Use the value 0 to disable and 1 to enable
    • Before getting into the memory reclamation techniques lets talk about the idle memory tax. The idle memory tax is a construct that, during a time of contention, will reclaim idle memory that is held by a virtual machine. Jason Boche (blog / twitter) has an older, but still excellent and relevant blog post on the Idle Memory Tax (IMT). The more idle memory a virtual machine has, the more the tax goes up, effectively reclaiming more memory. There are two advanced settings associated with the idle memory tax; Mem.IdleTax and Mem.IdleTaxType. Mem.IdleTax is the maximum percentage of total guest memory that can be reclaimed by the idle memory tax, with a default of 75%. Mem.IdleTaxType specifies whether the tax increases/decreases based on the amount idle memory (this is called variable and is the default). For this setting, 1 is the default (variable) and 0 is for a flat rate
    • There are two memory reclamation techniques that are used when memory contention exists amongst virtual machines; memory ballooning and memory swapping
    • Memory ballooning uses the memory balloon driver, known as vmmemctl which is loaded into the guest operating system as part of the VMware tools installation. Obviously, if the virtual machine doesn’t have VMware tools installed, it won’t have the balloon driver, which means the ballooning technique will be skipped and swapping may occur (this is bad!). When memory pressure exists the balloon driver determines the least valuable pages and swaps them to the virtual disk of the virtual machine (this is not host swapping). Once the memory is swapped to virtual disk, the hypervisor can reclaim that physical memory that was backing those pages and allocate it elsewhere. Since ballooning is performing swap to virtual disk, there must be sufficient swap space within the guest operating system. You can limit the amount of memory that gets reclaimed on a per-virtual machine basis by adding sched.mem.maxmemctl line to the virtual machine configuration file. The value is specified in MB
    • There are two types of swap to disk mechanism; swap to disk and swap to host cache. Swapping to disk is the same as it’s been in previous versions; a swap file is created (by default in the same location as the virtual machine’s configuration file) during power-on and during times of memory contention, if ballooning doesn’t slow/stop contention or the balloon driver isn’t working or available, swapping to disk occurs. Alternatively, for swap file location, you can change this per-VM, per-host, or specify a datastore for an entire cluster
    • Host cache is new in vSphere 5. If you have a datastore that lives on a SSD, you can designate space on that datastore as host cache. Host cache acts as a cache for all virtual machines on that particular host as a write-back storage for virtual machine swap files. What this means is that pages that need to be swapped to disk will swap to host cache first, and the written back to the particular swap file for that virtual machine
  • Identify VMware CPU load balancing techniques
    • CPU affinity is a technique that doesn’t necessarily imply load balancing, but it can be used to restrict a virtual machine to a particular set of processors. Affinity may not apply after a vMotion and it can disrupt ESXi’s ability to apply and meet shares and reservations
    • ESXi hosts can take advantage of multicore processors and use them to produce the most optimized performance for your virtual machines. The ESXi CPU scheduler is aware of the processor topology within the system and can see how the sockets, cores and logical processors are related to each other
      • By default, the CPU scheduler will spread the workload across all sockets in the system in undercommitted systems
      • You can override the default behavior by adding sched.cpu.vsmpConsolidate = True to the virtual machine configuration file. This setting will prevent the workload from being spread across all sockets it and limited it to the same socket
    • Hyperthreading is a feature that only exists in certain Intel processor families. Hyperthreading breaks up a single core on into two logical threads. This allows vCPU1 to execute instructions on thread1 while vCPU2 can execute instructions on thread2
      • Be careful when setting manual CPU affinity when hosts have hyperthreading enabled. The scenario exists where two virtual machines get bound to the same core (one on thread1 and one on thread2) which could be detrimental to the performance of those workloads
      • Hyperthreading is needs to be enabled in the Host BIOS and once that is done, should automatically be enabled in vSphere
    • NUMA (Non-Uniformed Memory Access) nodes work differently then your standard x86 system, and therefore, ESXi has a separate CPU scheduler; the NUMA scheduler. At a very high level, NUMA is an architecture that provides more than one memory bus. Each socket has its own bus to memory and the physical processors have the option to access memory that isn’t located on its dedicated bus (that’s the non-uniform part)
      • When a virtual machine is allocated memory, it takes memory locality into mind, meaning it will provide best effort in assigning memory that is from the home node (the home node is a term used to describe a processor and memory local to that processor)
      • If there is an imbalance in the load, the NUMA scheduler can change a virtual machines home node on-the-fly (CPU DRS for NUMA?). Even though the home node moves to a new home node, it does not automatically mean that the memory is relocated to its new home node, however the scheduler has the ability to relocate remote memory to once again make it local
      • The dynamic load balancing algorithm will exam the load and decide whether a rebalance is needed, this happens every two seconds by default


  • Identify pre-requisites for Hot Add features
    • There are a lot of different virtual hardware items that can be hot added to a virtual machine. Even though the topic doesn’t specifically refer to CPU and memory, that is what I’m going to focus on
    • Hot add cannot be abled for all virtual machines, here are some of the prerequisites:
      • Only certain guest operating systems are supported for hot add, so ensure the guest operating system you are using supports it
      • Hot add must be enabled per virtual machine and the virtual machine must be powered off in order to enable it
      • If you are hot-adding multocore vCPUs then the virtual machine must be using hardware version 8
      • If you are hot-adding a vCPU to a virtual machine using virtual hardware 7, the number of cores per socket must be set to 1
      • The virtual machine MUST have at least hardware version 7 or later
      • Install VMware tools
    • You can perform hot-add operations through the standard vSphere client or the vSphere web client

Skills and Abilities

  • Tune Virtual Machine memory configurations
    • In this section (and the rest of the “tuning” sections) I will not go over how to identify bottlenecks or misconfigurations (such as using ESXTOP to diagnose). I will simply be listing some recommended practices that should optimize and make your virtual machines more efficient. ESXTOP and vscsiStats will be covered in section 3.4 and in section 6
    • Pay attention to your virtual machine memory allocation. You don’t want to overcommit to the point where the VM starts swapping to host cache, or worse, disk. You can use the built-in performance charts and esxtop / resxtop to determine whether the VM is swapping pages to virtual disk or the host is swapping to disk (these items are covered in detail in section 3.4 and section 6)
    • Don’t oversize memory on your virtual machines
      • Even if you have the available physical memory, don’t allocate anymore then what’s needed. Over-allocating memory will waste physical memory as the more memory you allocate to a virtual machine, the more memory the vmkernel takes for overhead
    • Proceed cautiously when setting memory reservations and limits. Setting these too low or too high can cause unnecessary memory ballooning and swapping
    • Ensure VMware tools is installed and up-to-date
    • If you need to control priority over memory, use memory shares to determine relative priority
    • Use an SSD disk to configure Host cache
  • Tune Virtual Machine networking configurations
    • Here’s an easy one – use the paravirtualized network adapter, also known as the VMXNET3 adapter
      • Requires VMware tools to be installed
      • Requires virtual machine hardware version 7 or later
      • Ensure the guest operating system is supported
      • Enable jumbo frames for the VM if the rest of the infrastructure is using jumbo frames
        • Is set in the guest OS driver


  • Tune Virtual Machine CPU configurations
    • If hyperthreading is enabled for the host, ensure that the Hyperthreaded Core Sharing Mode for your virtual machines are set to Any

ht_advcpu (1)

    • If you need to disable hyperthreading for a particular virtual machine, set the Hyperthreaded Core Sharing Mode to None
    • Select the proper hardware abstraction layer (HAL) for the guest operating system you are using
        • This only applies for the guest operating systems that have different kernels for single processor (UP) and multiple processors (SMP). Single vCPU would use UP and all others will use SMP
    • If your application or guest OS can’t leverage multiple processors then configure them with only 1 vCPU
    • If your physical hosts are using NUMA, ensure the virtual machines are hardware version 8 as this exposes the NUMA architecture to the guest operating systems allowing NUMA aware applications to take advantage of it. This is known as Virtual NUMA


  • Tune Virtual Machine storage configurations
    • Logical disks you create inside the guest OS should be separated into separate VMDK files. In other words, have a 1:1 for logical disks and VMDKs for your OS disk and data disks
    • Ensure the guest operating system disks are aligned with the VMFS volumes they reside on
      • Some guest operating systems (such as Windows Server 2008) do this automatically
    • Consider using the paravirtualized SCSI (PVSCSI) adapter
      • The PVSCSI adapter can provide higher throughput and lower CPU utilization
      • Requires virtual machine hardware version 7 or later
    • Large I/O requests have the potential to be broken up into smaller requests by the device driver within the guest OS. Modify the registry to increase the block size as fragmented I/O requests can reduce performance


  • Calculate available resources
    • In vSphere 5 there are many visualizations within the GUI that will show you what the available resources are for a cluster, host or a virtual machine. You can also determine available resources for a host using esxtop
    • Determining available resources for a cluster can be done by viewing the Resource Distribution Chart
      • Log into the vSphere client
      • Navigate to the Hosts and Clustersview > select a cluster from the inventory tree
      • In the vSphere DRS  pane on the right, click the View Resource Distribution Charthyperlink
      • Here you can see CPU and Memory usages in MHz or MB or the percentage of each



    • Viewing available host memory in the GUI
      • Log into the vSphere client
      • Navigate to the Hosts and Clusters view > select a host from the inventory tree
      • You can view the current host resource utilization, as well as available resources by taking the total capacity and subtracting the current usage > located in the Resources pane


    • You can also calculate available host resources using esxtop
      • SSH into a host and type esxtopat the command line
      • On the CPU screen (default screen when running esxtop, press C to get to it) you’ll see two lines at the top called PCPU USED (%) and PCPU UTIL (%)


      • There is some difference between PCPU USED and PCPU UTIL, but for calculating available CPU resources, lets focus on PCPU USED (%). You’ll see each physical CPU represented on this line and its corresponding USEDpercentage
      • PCPU USED (%)represents the effective work of that particular VCPU, thus, allowing you to calculate the available resources per PCPU
      • You can also look at AVG, which is the last field in the PCPU USED (%) line and that averages all PCPUs. This would tell you the overall CPU resources used for the host (thus enabling you to calculate the available resources)
      • To calculate the available memory for a host in esxtop press the M button to navigate to the memory screen. This time we’ll focus on the second and third lines, which are PMEM /MB and  VMKMEM /MB, respectively. This is physical memory represented in megabytes and vmkernel memory represented in megabytes


      • PMEM /MBwill show you your total amount of physical memory, how much is being used by the vmkernel and how much memory is free on the host
      • VMKMEM /MB important items are the rsvd and ursvdfields. These represent, in MB how much memory is reserved and unreserved for the host. Here is why these are important:
        • If your PMEM is showing 20GB of memory available, but the VMKMEM only shows 15GB ursvd (unreserved) then your virtual machines only have 15GB available to them
    • You can view the available virtual machine resources through the GUI as well
      • Log into the vSphere client
      • Navigate to the Hosts and Clustersview > select a virtual machine from the inventory tree
      • On the right, click on the Resource Allocationtab
      • Here you can see the physical CPU and Memory that is allocated to the virtual machine. You can also see what is being consumed (CPU) or is active (MEM ) within the guest operating system


      • Above you can see that this particular virtual machine could consume up to ~4.5Ghz of CPU, but is only consuming 113MHz. You can also see that the VM has the potential to use ~9GB of memory, and it has consumed 6.4GB, but there is only 645MB active


  • Properly size a Virtual Machine based on application workload


  • Modify large memory page settings
    • Large memory page settings are configured per-host. Here are a list of existing large page settings:


    • You can modify any of the settings above by doing the following
      • Log into the vSphere client
      • Navigate to the Hosts and Clustersview > select a host from the inventory tree
      • On the right, click the Configurationtab
      • In the Software pane click the Advanced Settingshyperlink
      • Click the Mem object on the left > find the setting you want to modify (see list above)


  • Understand appropriate use cases for CPU affinity
    • CPU affinity is a tricky thing, and as a general practice, shouldn’t be used.
    • Some use cases (there are very few) that you want to use CPU affinity:
      • Simulating a workload
      • Load testing for an application
      • Certain workloads can also benefit from this
        • From a blog post by Frank Denneman: “When the virtual machine workload is cache bound and has a larger cache footprint than the available cache of on CPU, it can profit from aggregated caches”
    • Along with understanding the use cases for CPU affinity, it is important to understand some potential issues that are associated with it:
      • If you are using NUMA hardware, the NUMA scheduler may not be able to manage virtual machine with CPU affinity, essentially disabling NUMA for that virtual machine
      • Hyperthreaded enabled hosts may not be able to fully leverage hyperthreading on a virtual machine with CPU affinity
      • Reservations and shares may not fully be respected for a virtual machine configured for CPU affinity
      • CPU affinity might not exist for a virtual machine across all hosts in a cluster during a migration


  • Configure alternate virtual machine swap locations
    • Configuring an alternate location for virtual machine swap files is a simple task, but can be mundane if you have to do it for a lot of virtual machines
      • Alternatively you could configure an alternate swap location for the host
        • Select host from inventory tree
        • Choose Configuration > click Virtual Machine Swapfile Locationhyperlink
        • Click the Edit… hyperlink (if this is disabled then you have to choose the Store the swapfile in datastore specified by host option for the cluster


        • Select the datastore where you want to store the swapfile for all virtual machines on that host


        • Click OK
    • To configure the alternate swapfile location for a particular virtual machine
      • Log into the vSphere client
      • Navigate to the Hosts and Clustersview
      • Right-click a virtual machine from the inventory > click Edit Settings…
      • Click the Options tab > at the bottom click the Swapfile Location


      • Select one of the following options (as you can see in the screenshot above)
        • Default – which is either the cluster default or host default
        • Always store with the virtual machine – swapfile is stored in the same directory as the host
        • Store swap file in the host’s swapfile directory


Comments 1

  1. Pingback: Calculate available resources - The Foglite

Leave a Reply

Your email address will not be published. Required fields are marked *