VCAP5-DCA Objective 1.2 – Manage Storage Capacity in a vSphere Environment

For this objective I used the following documents:

  • Documents listed in the Tools section

Objective 1.2 – Manage Storage Capacity in a vSphere Environment

Knowledge

**ITEMS IN BOLD ARE TOPICS PULLED FROM THE BLUEPRINT**

  • Identify storage provisioning methods
    • There are two types of storage that can be provisioned through vSphere; block storage and NAS.
      • Block Storage
        • Local – any local storage attached to the host; uses VMFS
        • iSCSI – IP storage using a hardware or software iSCSI initiator; uses VMFS
        • FCoE – Fibre Channel over Ethernet using a hardware of software HBA; uses VMFS
        • FC – Fibre Channel using a hardware HBA; uses VMFNAS Storage
      • NAS Storage
        • NFS – currently using NFSv3 to mount NFS shares as datastores; uses NFS instead of VMFS
    • GUI Provisioning Method
      • The easiest way to provision storage is using the vSphere client. From the vSphere client you can create VMFS 3 or VMFS 5 datastores, you can create Raw Device Mappings or create a Network File System. You can do all this through the Add Storage wizard from within the client
        • Log into the vSphere client
        • Select a host > click the Configuration Tab
        • Click the Storage hyperlink
        • Click the Add Storage. . . hyperlink to launch the Add Storage wizard
      • From the Add Storage wizard you can provision block or NAS storage into the vSphere environment
    • Command-line Provisioning Methods
      • To provision storage through the command-line you can use vmkfstools
      • There aren’t a WHOLE lot of options for this command as it relates to creating file systems (you can also use vmkfstools to provision virtual disks. Here are the options:
        • You can specify whether it will be VMFS 3 or VMFS 5
        • You can set a block size (VMFS 3 ONLY)
        • You can set the volume name
      • You can also choose to span or grow an existing file system
      • Check out this example for creating a new VMFS 5 volume with a name of vmkfstools_vcap5_volume (a partition must exist on the LUN prior to creating a file system, which is what partedUtil is used for)  — VMware KB1009829 details this out as well
      • You can also add and remove new NAS volumes in the command-line using esxcli

 

  • Identify available storage monitoring tools, metrics and alarms
    • Two built-in monitoring tools that come with vSphere are Storage Reports and Storage Maps. Both of these can be found in the Storage Views tab within the vSphere client (this pertains to looking at host inventory objects)
      • In the hosts and clusters view click on a host
      • Click the Storage Views tab on the right
    • Different metrics exist to monitor storage performance and utilization. These metrics can be viewed within the vSphere client or by using esxtop/resxtop
    • There are also a number of pre-defined alarms that will assist your monitoring efforts, such as Datastore usage on disk and Thin-provisioned LUN capacity exceeded.  
    • Storage Reports
      • Storage reports will show you information on how different objects within your inventory map to storage entities. By default a storage report for a host inventory object includes:
        • VM Name
        • Multipathing Status
        • Space Used
        • Snapshot Space
        • Number of disks
      • Here is a screen shot detailing out the defaults (the items checked) as well as all available fields that can be displayed within storage reports (for host inventory objects)

image

      • The columns and information displayed will be dependent upon which inventory object you have selected. I’ll let you go through each one and see how these reports vary
      • Reports are updated every 30 minutes by default. You can manually update them by clicking the Update… hyperlink from within Storage Views > Reports located on the upper right of the screen
      • You can filter these reports by selecting which columns you want to search on, and then typing in the keyword(s)

image

      • You can export reports in the following formats
        • HTML
        • XLS
        • CSV
        • XML
        • Export Reports
          • Choose an inventory object
          • Click the Storage Views tab and select Reports
          • Choose which columns you want to view and any filtering
          • Right-click below the table and select Export List…
          • Enter in a name and choose the file format > click Save
    • Storage Maps
      • Storage maps give you a nice representation of storage resources (physical and virtual) as they pertain to a specific inventory object. Storage maps are also updated automatically every 30 minutes and you can manually update them by clicking the Update… hyperlink located near the top right of the inventory object > Storage Views > Maps screen
      • Just as with Storage reports, Storage maps have default views for each type of inventory object. Using the different checkbox within the Maps area you can filter out object relationships that you do not wish to see

image

      • By left-clicking on an object you can drag it to different parts of the screen
      • Storage maps can also be exported in the same fashion as Storage reports, although, as you can imagine, your file type selection will be different
        • .jpeg
        • .bmp
        • .png
        • .tiff
        • .gif
        • .emf
    • Storage Metrics (vSphere Client)
      • As with storage reports and storage maps, the types of metrics you will see as they relate to storage will vary depending upon which inventory object you select. For example, if you select a datastore inventory object you will by default be show space utilization views in a graph format (graphs based on file type and the top 5 virtual machines)
      • You can then change that default view from Space and change it to Performance, which will show you a slew of performance charts for that particular datastore
      • To see the real “meat and potatoes” of metrics as they relate to storage within the vSphere client you need to look at advanced performance charts
        • Select a host from the inventory
        • Click the Performance tab > click the Advanced button
        • From the drop down there are four related storage items
          • Datastore
          • Disk
          • Storage Adapter
          • Storage Path
        • If I went into every counter that you could see for the objects above you will be reading this post for the next 6 weeks. So know where these metrics are and at the very least familiarize yourself with defaults
    • Storage Metrics (esxtop/resxtop)
      • I decided not to go into a lot of detail for this section as there are already some great resources out there. For a good review of this tool check out Duncan Eppings blog post on esxtop. For a detailed review of all statistics for esxtop check out this VMware community post
      • For storage monitoring there are three panels within esxtop that you will want to be intimately familiar with (the letters at the end correspond the the esxtop hotkey for those panels)
        • Storage Adapter Panel (d)
        • Storage Device Panel (u)
        • Virtual Machine Storage Panel (v)
      • Some key metrics you want to look at for the panels above
        • MBREAD/s — megabytes read per second
        • MBWRTN/s — megabytes written per second
        • KAVG — latency generated by the ESXi kernel
        • DAVG — latency generated by the device driver
        • QAVG — latency generated from the queue
        • GAVG — latency as it appears to the guest VM (KAVG + DAVG)
        • AQLEN – storage adapter queue length (amount of I/Os the storage adapter can queue)
        • LQLEN – LUN queue depth (amount of I/Os the LUN can queue)
        • %USD – percentage of the queue depth being actively used by the ESXi kernel (ACTV / QLEN * 100%)
    • Alarms
      • There are a number of different pre-configured alarms related to storage that can be leveraged to alert you of impending storage doom. As with a lot of functions within vSphere, different alarms are pre-defined based on the inventory object that you select. Which means there are different storage related alarms for different inventory objects
        • If you are in the vSphere client and you select the top-most inventory object (the vCenter object) and you go to the Alarms tab, you can select Definitions and view ALL pre-configured alarms for all objects
      • Again, I won’t go into every single alarm and what they do, but here are a list of some I think are important to know, along with their default triggers
        • Cannot connect to storage – this alarm will alert you when a host has an issue connecting to a storage device The three defaukt triggers are:
          • Lost Storage Connectivity
          • Lost Storage Path Redundancy
          • Degraded Storage Path Redundancy
        • Datastore cluster is out of space – this alarm monitors disk space on datastore clusters. The default triggers are:
          • Send a Warning when utilization is above 75%
          • Send an Alert when utilization is above 85%
        • Datastore usage on disk – this alarm monitors disk space on a datastore. The default triggers are:
          • Send a Warning when utilization is above 75%
          • Send an Alert when utilization is above 85%
        • Thin-provisioned LUN capacity exceeded – this alarm monitors thin-provisioned LUNs using the vSphere Storage APIs. Triggers for these alarms must be modified through the vSphere API (VASA) and is implemented by your storage vendor

 

Skills and Abilities

  • Apply space utilization data to manage storage resources
    • I’m not 100% what VMware is looking for on this, but my best guess is to use some of the techniques above to determine current space utilization, and then manage your storage resources appropriately
    • Since we’ve already gone through the different metrics and alarms to monitor, let’s use the ESXi shell to determine VMFS disk usage. The command df, which in Linux speak stands for disk filesystem, is used to display the the filesystems that are mounted to that particular host.

Since I filtered the results you don’t see an explanation of each column. From left to right:                           

      Filesystem        Size       Used         Available    Use%        Mounted on

image

      • At the moment we are focused on space utilization, so we want to focus on the Use%. As you can see, none of my partitions are over 50%. If I had a highly used partition you would most likely get an alarm from the Datastore Usage alarm, and you could use df to see a summary of all your partitions
      • There are lots of way to rectify this, add more space/extents, delete unneeded virtual machines or remove unneeded virtual disks (you could accomplish this through the vSphere client or by using the vmkfstools -U command)
    • The bottom line is that you need to be aware of, not only how you can determine space utilization, but then to apply that data in an intelligent way in order to manage your storage resources effectively

 

  • Provision and manage storage resources according to Virtual Machine requirements
    • I’ve covered some of this in Objective 1.1 – Implement and Manage Complex Storage Solutions. Before you can provision, or manage, storage resources for a virtual machine, you first must know the virtual machine requirements, which includes, but is not limited to:
      • Space – how much space is needed
      • I/O workload – how many spindles are needed to satisfy the workload
      • Resiliency  — how protected does the data need to be
    • Looking at the above list you can look at the application requirements for the recommended amount of disk space. You can use tools such as vscsiStats or IOmeter to determine the workload characteristics and how many spindles you’ll need. Depending upon availability and resiliency requirements will determine RAID level, whether snapshots (array level) will be used, what level of backup and how often to backup and how long the data needs to remain in an off-site location
    • Once you’ve determined the virtual machine requirements you can start to provision and manage your storage based on those requirements. If you have a virtual machine that requires a certain level of service or, say it needs to be on super fast storage, you can leverage a few vSphere features to help you accomplish that goal
      • Profile Driven Storage – again, I covered this in Objective 1.1 on how to configure and implement profile driven storage. You can create a profile based on a virtual machine(s) requirement, such as fast disks, and assign that storage capability to one or more datastores. You can then create a storage profile and apply it to the virtual machine. Whenever that particular virtual machine is on a datastore that doesn’t meet that storage profile, it will be marked non-compliant
      • Datastore Cluster – you can group similar datastores into a construct known as a datastore cluster. This allows you to assign virtual machines to that datastore cluster, and, in conjunction with Storage DRS, the virtual machine will be placed on the least used datastore (in terms of I/O and space utilization)
    • You can provision storage for a virtual machine in a few different ways:
      • Adding a new disk through the vSphere Client
      • vmkfstools
    • Adding storage to a virtual machine through the vSphere client is pretty straight forward so lets go through how you would create an empty virtual disk using vmkfstools

image

        • Above you can see that the command was successful and that the vcap5-flat.vmdk and vcap5.vmdk files were created

 

  • Understand the interactions between virtual storage provisioning and physical storage provisioning
    • The virtual provisioning of physical storage can add benefit to your organization as long as you understand the implications of what you are doing. Virtual storage provisioning allows you to over-commit your storage resources as needed
    • If I had to pick one construct to understand when it comes to the interaction between virtual storage provisioning and physical storage provisioning it would be with Thin Provisioning. Thin provisioning allows you to create a virtual disk that is, for example, 40GB in size, but you’re actually only using 5GB. The guest operating system thinks its hard disk is physically 40GB, while the physical storage has only allocated 5GB
      • The biggest thing that you need to understand here is that by thin provisioning the actual size on the disk is less than what you’ve provisioned, which can get you into trouble if you aren’t paying attention to the physical storage
      • If you have a 100GB datastore, you can put 40 VMs with 5GB virtual hard disks that are thin provisioned. Even those those 40 VMs may only be using 2GB each, they have the potential to grow up to 5GBs, which at a certain point would cause you to physically run out of storage space; NOT GOOD!
      • In the section above we went over created an empty virtual disk, and we created it as a thin disk. Since it is a thin disk,the provisioned size will be different from the actual size. Here is what you’ll see when looking in the datastore browser

image

      • As you can see the Size and Provisioned Size are much different.
      • The same exists when you have a datastore full of thin disks, the Capacity and Provisioned Space will differ. Let’s have a look (Go to the Datastores and Datastore Cluster view > click on a datastore on the left > click the Summary tab on the right)

provisioned_space

      • The Capacity is 1.56 TB while the provisioned space is more than 1TB over the physical capacity. However, my physical free space is still ~600GB
    • The point I’m trying to get across is that you need to be intimately familiar with what your virtual storage environment is, and what it is doing, while keeping the physical storage in mind
    • If you have a thinly provisioned virtual disk that you want/need to physically consume all of its provisioned space AFTER you have created it then you can Inflate the disk. This can be done within the datastore browser by right-clicking on the VMDK file and selecting Inflate. You can also do this from the command line; here is how
      • This operation can take quite a long time to complete depending on how much physical space needs to be zeroed out
      • Now as you can see the Size shows what the Provisioned Size used to show, and now the Provisioned Size column is blank (which is expected as that field isn’t populated unless the virtual disk is thin)

image

 

 

  • Configure Datastore Alarms
    • There are five pre-configured datastore alarms that ship with vSphere 5, see the below screen shot for their names and descriptions

image

    • Aside from the five datastore alarms you see above, there are a lot more triggers we can use to create alarms for the Datastore monitor and whether you choose to monitor for a specific condition/state or for a specific event
      • Log into the vSphere client and navigate to the Datastores and Datastore Cluster view
      • Click on a datastore from the listing on the left > click the Alarms tab > click the Definitions button
      • Right-click anywhere under the pre-configured alarms and select New Alarm…
      • Enter in the following details:
        • Alarm Name: Datastore Over Provisiong Alarm
        • Description: Alarm to monitor the provisioned space on the datastore
        • Alarm Type: Datastore
        • Choose Monitor for specific conditions or state…
        • Enable this alarm: Check this box

image

      • Click on the Triggers tab > click Add to add a new trigger
      • Enter in the following details:
        • Trigger Type: Datastore Disk Provisioned (%)
        • Condition: Is above
        • Warning: 100
        • Alert: 200
        • Select the Trigger if any of the conditions are satisfied radial button

image

      • Click the Reporting tab
      • Choose if you want the alarm to repeat when the condition exceeds a certain range
      • Choose the frequency

image

    • Click the Actions tab > click Add to add an action
    • Enter in the following details
      • Action: Send a notification email
      • Configuration: josh.coen@valcolabs.com
      • You can choose when to perform this action based on the alarm transition state. By default this will perform the action one time when the alarm goes from warning to alert. Just leave the default

 

    • Click OK (you will get a warning message if your vCenter SMTP settings are not configured)

image

    • There are A LOT more triggers that relate to the Datastore monitor when you select the  Monitor for specific events occurring… radial button. Here is a list:

image

    • As you can see you have A LOT of options to choose from and you can use the instructions in the previous steps to create new alarms that can help you effectively monitor your datastores

 

  • Analyze Datastore Alarms and errors to determine space availability
    • Using datastore alarms and errors to determine your available space is pretty straight forward. The default alarm Datastore usage on disk is the perfect alarm to use, and it’s enabled by default
    • The Datastore usage on disk alarm is pre-configured to trigger a warning when its disk usage is over 75%. It will trigger an alert if it gets above 85%. Now again, these are the defaults for this alarm, you may want to edit the thresholds based on your organizations best practices as it relates to %free for storage
    • You can only edit alarms in the scope in which they are defined in. In this case, the Datastore usage on disk alarm is defined at the top level object, which is the vCenter object
    • I created an 8.6GB eagerzeroedthick virtual disk using vmkfstools on a datastore that had only 8.89GB free.
    • Once my view was updated (these are updated every 30 minutes) an alert was triggered

image

    • Now if I was seeing this alert for the first time the first thing I would do is check the space availability of my datastore. If it was in fact close to being at capacity I would either allocate more space, delete unneeded virtual disks/files or perform a storage vMotion to another datastore that had more capacity

  • Configure Datastore Clusters
    • Configuring datastore clusters is an easy enough process, but it is a process and can only be created from the vSphere client (can’t create in vSphere Web Client)
      • Log into the vSphere client and navigate to the Datastores and Datastore Clusters view
      • Right-click on your datacenter object and select New Datastore Cluster…
      • Enter in a name for the datastore cluster and choose whether or not to enable Storage DRS

image

      • Click Next
      • Choose either No Automation (Manual Mode) or Fully Automated
      • We aren’t adding any Advanced Options so click Next

image

      • Decide whether you want to enable the I/O metric for SDRS recommendations
      • Choose the thresholds you want SDRS recommendations to be triggered on
        • Utilized Space — default is 80%
        • I/O Latency — default is 15ms

image

      • Click the Show Advanced Options hyperlink to set the advanced options
        • Set the percentage for the minimum utilization difference between the source and destination datastore before SDRS will make a recommendation
          • Here is an example: If leave this at the default (5%), SDRS will not make a recommendation for a move unless the there is at least a 5% difference between the source datastore and the destination datastore in terms of utilization. So, the datastore first needs to exceed the utilization space threshold and then there needs to be at least 5% difference in terms of utilization before SDRS will make a recommendation
        • Set the frequency that SDRS should check for imbalances — default is 8 hours
        • Set the I/O imbalance threshold

image

      • Click Next
      • Select which cluster(s) you want to use > click Next

image

      • Select which datastores you want as part of the datastore cluster
        • Best practice is to use datastores that have similar capabilities, that way application owners and users should never experience a degradation of service due to an applied SDRS recommendation

image

      • Click Next > click Finish

 

Tools

Comments 8

  1. Pingback: VCAP5-DCA Objective 4.1–Implement and Maintain Complex VMware HA Solutions » ValCo Labs

  2. Pingback: Configuring and Analyzing Datastore Alarms - The Foglite

  3. Pingback: VCAP5-DCA-Objective 3.1–Tune and Optimize vSphere Performance » ValCo Labs

  4. Pingback: VCAP5-DCA–Objective 3.3 – Implement and Maintain Complex DRS Solutions » ValCo Labs

  5. Pingback: VCAP5-DCA – Objective 1.2 – Manage Storage Capacity in a vSphere Environment – Skills And Abilities | VCDX or Bust

  6. Pingback: The Unofficial Official VCAP5-DCA Study Guide. - RAVIITBLOG.COM

  7. Your blog is fantastic! if your command can used powervCli or powershell will good for everyone.

Leave a Reply to Hoo Cancel reply

Your email address will not be published. Required fields are marked *

*