VCAP5-DCA-Objective 3.1–Tune and Optimize vSphere Performance

For this objective I used the following documents:

Objective 3.1 – Tune and Optimize vSphere Performance

Knowledge

**ITEMS IN BOLD ARE TOPICS PULLED FROM THE BLUEPRINT**

  • Identify appropriate BIOS and firmware setting requirements for optimal ESXi host performance
    • BIOS settings on your hosts is an important thing to take into consideration when optimizing your environment. Here are some general guidelines (pulled from the aforementioned whitepaper) you can follow that will assist you in your optimization efforts
      • Ensure you are using the most up-to-date firmware for your host
      • Ensure all populated sockets are enabled
      • Enable “Turbo Boost” if your processor supports it (is this like the turbo button on my x486?)
      • If your processor(s) support hyper-threading, make sure it is enabled
      • Set Disable node interleaving to Enabled (disabling this setting will essentially disable NUMA)
      • Disable any hardware devices that you won’t be using
      • Depending on your workload characteristics you may, or may not, want to disable cache prefetching features. Workloads that randomly access memory may get a performance boost if these features are disabled
      • Set the CPU power-saving features to “OS Controlled”
        • this will allow the hypervisor to control and manage these features
      • Last, but certainty not least, enable Hardware Virtualization (VT). You will know right away if this is NOT enabled if you try and boot a 64-bit virtual machine and get a ‘longmode’ error
  • Identify appropriate driver revisions required for optimal ESXi host performance
    • I don’t know exactly what it is they are looking for here, and I can’t find it in any of their product documentation. A few things that come to mind though:
      • Check the VMware HCL
        • From the dropdown you can select a category of what you are looking for

image

        • In this example I chose IO Devices
        • You can then select which VMware product and version and then select which vendor and I/O Device type
        • Click Update and View Results
        • Scroll through the list until you find the device you are looking for. The model of the device should be a hyperlink, click the hyperlink
        • Here you will see pertinent information for the release and the device driver to use

image

        • I have no idea if this is the ‘optimized’ driver, but logically you would think it’s the best device driver for that device, based on the product version
      • The next best place to look would be to check the vendor’s website and see if they have made a separate driver to use with your version of vSphere

Skills and Abilities

  • Tune ESXi host memory configuration
    • In this section (and the rest of the “tuning” sections) I will not go over how to identify bottlenecks or misconfigurations (such as using ESXTOP to diagnose). I will simply be listing some recommended practices that should optimize and make your hosts more efficient. Troubleshooting will be covered in section 6
    • One thing that you will see a lot of are blanket memory configurations for virtual machines, such as all Windows Server 2K8 R2 VMs will get a base of 4GBs of RAM and will only be increased if needed. On the surface this seems like a good practice, but what if that VM only needs 3GB?
      • Virtual machine memory overhead is dependent on the configured memory size of a virtual machine, the more you configure, the more overhead it takes, the less memory is available for your other virtual machines.
        • Don’t under-configure the memory where the working set can’t keep up because of too little memory (thrashing)
        • Don’t over-configure the memory where the working set doesn’t use all the configured memory and now you have wasted more memory than needed on memory overhead
    • The same concept above applies to the number of  vCPUs you configure for a virtual machine. The more vCPUs you configure increases the amount of memory overhead.
      • Don’t give you virtual machines more vCPUs then what is needed. Doing so increases memory overhead
    • Memory over-commitment is a feature of vSphere, and VMware has 5 different mechanisms to deal with over-commitment. There are a few things to keep in mind when talking about over-commitment, and tuning our hosts to use it effectively
      • The biggest degradation of performance to a virtual machine is when the host starts swapping to disk. There are four other memory over-commitment techniques that are used before swapping to disk
        • Don’t disable these other memory over-commitment techniques; ballooning, page sharing and memory compression
    • Use the new swap to host cache feature
      • This is a new memory over-commitment technique that allows the host to swap to cache instead of to disk. The ‘cache’ it is referring to is a SSD disk
        • Configure a SSD as host cache, which will get much better performance than swapping to traditional disk
    • Virtual machine swap files are created in the VM working directory by default (typically where the .vmx file is located)
      • Ensure that location of those swap files have enough free disk space. The swap file is created dynamically during a power on operation and the size is determined using the following calculation:
      •                              allocated mem – reserved mem = swap file size (.swp)
      • Don’t place swap files on thin-provisioned disks
    • The biggest take away here should be, its OK to overcommit memory, but not to the point where you are swapping out to disk.
  • Tune ESXi host networking configuration
    • One thing that you want to monitor when thinking about virtual networking and how to make it perform as efficient as possible is your CPU utilization. Virtual networking relys heavily on the CPU to process the network queues. The higher CPU utilization you have, the less throughput you make get
    • DirectPath I/O may provide you a bump in network performance, but you really need to look at the use case. You can lose a lot of core functionlity when using this feature, such as vMotion and FT (some special exceptions when running on UCS for vMotion) so you really need to look at the cost:beneift ratio and determine if it’s worth the tradeoffs
    • You can control your bandwidth and how it is allocated by using Network I/O Control (NIOC). You allocat bandwidth to resource pools and use shares/limits to establish priority. There seven pre-defined network resource pools:

image

      • There is also something called a user-defined resource pool in which you can create your own resource pool in order to prioritize other traffic not covered by the pre-defined pools
    • User-defined pools are pretty archaic, all you can do is assign shares and a QoS priority tag. Let’s go through an example of creating a user-define network resource pool:
      • Log into the vSphere client
      • Switch to the Networking view by selecting the View menu > select Inventory > select Networking (Ctrl + Shift + N)
      • Select a vDistributed Switch from the inventory on the left (remember that NIOC requires an enterprise+ license) > click the Resource Allocation tab
      • Click the New Network Resource Pool… hyperlink

image

      • Enter in a Name and Description
      • Set the Physical Adapter Shares value (Low, Normal, High or Custom)
      • If you Uncheck the Unlimited option be sure to enter in what amount, in Mbps that you want to set it to
      • Set a QoS Priority Tag if desired and select a tag from the dropdown (1-7)
      • Click OK

image

    • Use separate vSwtiches with different physical adapters. Doing so should help avoid unnecessary contention between the VMkernel and virtual machines
    • The us of the VMXNET3 paravirtualized adapter should be used as the standard, not the exception. When creating new virtual machines you should be asking yourself “Why shouldn’t I use VMXNET3?”, not “Why should I use VMXNET3?”
    • If you have network latency sensitive applications you want to adjust the ESXi host power management settings to the maximum performance.You do this so resources aren’t asleep for some reason when your application needs them
      • Log into the vSphere client and navigate to the Hosts and Clusters view
      • Select a host from the inventory > click the Configuration tab
      • In the Hardware pane click the Power Management hyperlink
      • Click the Properties hyperlink in the upper right
      • Select the High Performance option
      • Click OK
    • Also for applications that are sensitive to network latency you want to disable C1E and other C-states in the BIOS of the host(s) that the application may run on
    • When using the VMXNET3 networking adapter, there is a feature called virtual interrupt coalescing. Disabling this feature can improve performance for certain network latency-sensitive applications. However, be careful when enabling this as it may reduce performance for other types of workloads. This is enabled per-VM with the ethernetX.coalescing.Scheme advanced configuration option, which we’ll go over configuring in a later section
    • SplitRx Mode is a new feature that was introduced with vSphere 5.0 and it can improve performance for virtual machines in certain circumstances. Typically, networking traffic comes into a network queue and is processed by a single physical CPU. SplitRx Mode is a per-VM setting that allows network traffic coming into a single network queue to be processed by multiple physical CPUs
      • If the VM is a network appliance that is traversing traffic between virtual machines on the same host using the API, then throughput may be increased with the use of SplitRx
      • If you have more than one virtual machine on the same host receiving multicast traffic from the same location then SplitRx can improve throughput and CPU efficiency
      • Enable SplitRx mode using the ethernetX.emuRxMode advanced configuration setting
  • Tune ESXi host CPU configuration
    • This may be a given, but turn on DRS. You don’t want a host getting overloaded with VMs and maxing out the CPU when there are other hosts in your cluster that have idle CPU cycles
    • Don’t configure your VMs for more vCPUs then their workloads require. Configuring a VM with more vCPUs then it needs will cause additional, unnecessary CPU utilization due to the increased overhead relating to multiple vCPUs
    • If your hardware supports Hyper-threading (the hardware itself and BIOS) then the hypervisor should automatically take advantage of it. If your hardware does support hyper-threading but it doesn’t show enabled in vCenter, ensure that you enable it in your hardware BIOS
      • Here you can see that hyper-threading is enabled

ht_procs

      • In vCenter you can enable/disable hyper-threading by going to the Configuration tab of the host > click the Processors hyperlink > click the Properties hyperlink

ht_enable

    • When using hyper-threading ensure that you leave the per-VM advanced CPU setting to Any. Changing this setting to None will essentially disable hyper-threading for that particular virtual machine as it will place the other ‘core’ in a halted state

ht_advcpu

    • When dealing with NUMA systems, ensure that node interleaving is disabled in the BIOS. If node interleaving is set to enabled it essentially disables NUMA capability on that host
    • When possible configure the number of vCPUs to equal or less than the number of physical cores on a single NUMA node
      • When you configure equal or less vCPUs:physical cores the VM will get all its memory from that single NUMA node, resulting in lower memory access and latency times
  • Tune ESXi host storage configuration
    • Enable Storage DRS. Even if you set it to manual, enable Storage DRS in order to get the initial placement recommendations. Storage DRS is enabled by creating a Datastore Cluster. This has been covered in Objective 1.2 – Manage Storage Capacity in a vSphere Environment so I won’t go over it again, but just know that you should enable this when possible
    • Turn on Storage I/O Control (SIOC) to split up disk shares globally across all hosts accessing that datastore. SIOC will proportionally assign disk shares per-host based on the sum of VM disks shares and total disk shares for that datastore
    • Ensure that the storage configuration setup has enough IOPs to support the virtual machine workloads running on said storage
    • One of the key metrics you want to monitor in r/esxtop are GAVG counters, these are the “guest average” counters and they indicate what the guest VM is seeing. For example, the GAVG/cmd counter will show what latency the guest VM is seeing when accessing that particular storage device. Again, this will be covered more in-depth in Section 6
    • Ensure that your multi-pathing policies are set in accordance with the best practices from VMware and your storage vendor. Even if the multi-pathing policy you are currently using might e working, it doesn’t mean that there isn’t a better one out that that is more efficient
  • Configure and apply advanced ESXi host attributes
    • There are many advanced host attributes that can be set, such as for memory or CPU
    • Configure Advanced ESXi Host Attributes
      • Log into the vSphere client
      • Click on a host from the inventory > click the Configuration tab
      • On the right, in the Software pane click the Advanced Settings hyperlink

advsettings

      • Choose the item on the left where the attribute is located, such as Cpu
      • On the right, locate the proper attribute and make the required change
      • A list of Memory and CPU advanced attributes can be found in the vSphere Resource Management guide on pages 104-106
  • Configure and apply advanced Virtual Machine attributes
    • Advanced virtual machine attributes are changed per VM and typically the VM will need to be powered off in order to make the change
      • I have successfully made advanced VM changes with VMs powered on using PowerCLI and then either powering the VM off/on or performing a vMotion. The vMotion is registering the VM on a new host, which means it goes through the .VMX file again
    • Configuring Advanced Virtual Machine Attributes
      • Log into the vSphere client
      • From the inventory, right-click a VM and select Edit Settings…
      • Click the Options tab > click General > click the Configuration Parameters button

configparam

      • Click the Add Row button
      • Enter in the Name of the attribute and the Value

newparam

  • Configure advanced cluster attributes
    • The only advanced cluster attributes that I know of are for vSphere HA. If there are others that can be configured for the cluster please let me know!
    • Configure Advanced Cluster Attributes
      • Log into the vSphere client
      • From the inventory, right-click on a cluster and click Edit Settings…
      • Click on vSphere HA > click the Advanced Options… button

advha

      • Here you can add different options and values. A list and explanation of advanced HA options can be found on Ducan Epping’s (blog / twitter) HA Deepdive post
      • Click OK when finished

Tools

Comments 8

  1. Thank you for all your hard work, this is helping me so much to prepare for the VCAP5DCA

  2. Post
    Author
  3. Hey mate this is all good stuff thanks.
    You wanted to know if there are any other advanced cluster attributes.
    We use for DRS:

    CostBenefit = 0
    MinGoodness = 0

    which keeps our DRS balanced more evenly.

    Keep up good work.

  4. Hi – in your overview of BIOS settings you state that you should “Disable node interleaving (enable this will essentially disable NUMA)” – the VMware document you referenced previously has this to say:

    Some NUMA-capable systems provide an option in the BIOS to disable NUMA by enabling node interleaving. In most cases you will get the best performance by disabling node interleaving (in other words, leaving NUMA enabled).

    You revisit this later in your note saying: “When dealing with NUMA systems, ensure that node interleaving is disabled in the BIOS. If node interleaving is set to enabled it essentially disables NUMA capability on that host”

    I suspect the first time you mention node interleaving you have the wrong setting advice, do you think so?

    1. Post
      Author

      Hi Charles,

      Thanks for the comment, and thanks for the catch. The BIOS setting can be confusing because of the word “Disable” in the setting name, and then the action of enabling or disabling. I’ve corrected the overview so it reads better. In the end ensure that the “Disable Node Interleaving” setting in the BIOS is set to Enabled for best performance of NUMA systems

      -Josh

  5. I noticed this while reading through, it says the swap file is created and the same size as the configured memory for the VM, isnt the swap file the size of the configured memory minus the memory reservation?

    Virtual machine swap files are created in the VM working directory by default (typically where the .vmx file is located)
    Ensure that location of those swap files have enough free disk space. The swap file is created dynamically during a power on operation and is the same size as the configured memory for that VM
    Don’t place swap files on thin-provisioned disks

  6. Post
    Author

    Hey Josh,

    You are correct. The swap file size is allocated memory – mem reservation. It will be the same size should the VM not have a reservation. The post has been updated to include VMs with memory reservations.

    Thanks for pointing it out!

  7. Pingback: The Unofficial Official VCAP5-DCA Study Guide. - RAVIITBLOG.COM

Leave a Reply to Josh Sims Cancel reply

Your email address will not be published. Required fields are marked *

*