VCAP5-DCA Objective 1.1 – Implement and Manage Complex Storage Solutions

For this objective I used the following documents:

Objective 1.1 – Implement and Manage Complex Storage Solutions

Knowledge

**ITEMS IN BOLD ARE TOPICS PULLED FROM THE BLUEPRINT**

  • Identify RAID Levels
    • If you are looking at this blueprint and contemplating taking this exam I’m going to assume that you know what RAID is. If you don’t, then you are possibly in for a LONG VCAP5-DCA preparation. I’m not going to list out every single RAID level, but I will go over the most commonly used ones; RAID 0, 1, 5, 6 and 1+0
      • RAID 0: Striping only, no redundancy. Data is striped over all disks in a RAID 0 set. Minimum of 2 disks.
        • Pros:
          • Very good performance
          • Allows for the maximum use of disk space
        • Cons
          • No redundancy
          • Any drive failure will destroy the entire dataset
      • RAID 1: Mirroring only, no striping. Data is mirrored across disks. If you have a two disk RAID 1 set then the same data is on both disks. Minimum of 2 disks.
        • Pros:
          • Redundant
          • Write performance degradation is minima
        • Cons:
          • You lose half of your disk capacity (two 1TB disks, 2TB total only nets you 1TB)
      • RAID 5: Striping with parity. Data is striped across the all disks in the RAID 5 set and parity bits are distributed across the disks. Minimum of 3 disks
        • Pros:
          • Can sustain a loss of 1 drive in the set
          • Very good read performance
        • Cons:
          • Write performance not as good as RAID 1 due to parity calculation
          • Throughput is degraded when a disk does fail
      • RAID 6: Striping with double parity. Data is striped across all disks in the RAID 6 set along with double parity. Minimum of 4 disks
        • Pros:
          • Can sustain a loss of 2 drives in the set
          • Useful in large RAID sets
          • Very good read performance
        • Cons:
          • Requires 4 disks
          • More disk space is utilized for the extra parity
          • Write performance not as good as RAID 1 or 5 due to double parity calculation
      • RAID 1+0 (RAID 10): Mirroring and Striping. Disks in a RAID 10 set are mirrored and then striped across more disks.Minimum of 4 drives and total drives must be an even number
        • Pros:
          • Great read/write performance
          • Can survive many drive failures as long as all drives in a mirror don’t fail
        • Cons:
          • Only 50% of disk capacity is available due to mirroring
          • Complex compared to RAID 0 and RAID 1

 

  • Identify Supported HBA types
    • The three types of Host Bus Adapters (HBA) that you can use on an ESXi host are Ethernet (iSCSI), Fibre Channel or Fibre Channel over Ethernet (FCoE). In addition to the hardware adapters there is software versions of the iSCSI and FCoE adapters (software FCoE is new with version 5) are available.
    • There are far too many adapters to list, but the usual suspects make them:
      • Broadcom
      • Brocade
      • Cisco
      • Emulex
      • QLogic
    • To see all the results search VMware’s compatibility guide

 

  • Identify virtual disk format types
    • There are three types of virtual disk formats:
      1. Thick Provision Lazy Zeroed – a thick disk is created and all space on the underlying storage is allocated upon creation. The blocks within the allocated space are zeroed out on demand (not at the time of virtual disk creation)
      2. Thick Provision Eager Zeroed – a thick disk is created and all space on the underlying storage is allocated upon creation. The blocks within the allocated space are zeroed out up front – it will take some time (considerable amount of time depending on disk size) to create this type of virtual disk
      3. Thin Provisioned – Only space that is needed is allocated to these types of disks. As the need for more physical space grows a thin provisioned disk will grow to meet that demand, but only up to its configured size
    • Using a Raw Device Mapping (RDM) may also be considered a virtual disk format type. While I don’t consider it a virtual disk format, I wanted to include it anyway. A RDM is a pointer to a physical LUN on a SAN. When you create a RDM a .vmdk file is created, but only contains pointer to the physical LUN

Skills and Abilities

  • Determine use cases for and configure VMware DirectPath I/O
    • DirectPath I/O allows a VM to access a device on the physical server without intervention from the hypervisor
    • The CPUs must have Intel Virtualization Technology for Directed I/O (Intel VT-d) feature or if using AMD processors, have AMD I/O Virtualization Technology (IOMMU). Once you verify your CPUs are capable, ensure the feature is enabled within the BIOS
    • According to test results done by VMware in a recent performance whitepaper, Network I/O Latency in vSphere 5, using DirectPath I/O lowered the round trip time by 10 microseconds. While 10 microseconds may seem miniscule, it can be the difference with very low latency applications
    • A few use cases:
      • Stock Market applications (an example used in the aforementioned white paper)
      • A legacy application that may be bound to the physical device
      • Can improve CPU performance for applications with a high packet rate
    • Configuring DirectPath I/O on the ESXi host (from VMware KB 1010789)
      • In the vSphere client select a host from the inventory > click the Configuration tab > click Advanced Settings  under the Hardware pane
      • Click Edit and select the device(s) you want to use > click OK
      • Reboot the host (once the reboot is complete the devices should now appear with a green icon)
    • Configuring a PCI Device (Direct Path I/O) on a Virtual Machine (from VMware KB 1010789)
      • In the vSphere client right-click the virtual machine you want to add the PCI device to and click Edit Settings…
      • Click the Hardware tab > click Add
      • Choose the PCI device > click Next

 

  • Determine requirements for and configure NPIV
    • N-Port ID Virtualization (NPIV) is used to present multiple World Wide Names (WWN) to a SAN network (fabric) through one physical adapter. NPIV is an extension of the Fibre Channel protocol and is used extensively on converged platforms (think Cisco UCS)

     

    • Here are a list of requirements you must meet in order to use NPIV
      • The Fibre Channel switches must support NPIV
      • The physical HBAs in your hosts must support NPIV
        • vMotioning a virtual machine configured with NPIV to a host whose physical HBA does not support NPIV will revert to using the WWN of the physical HBA
      • Heterogeneous HBAs across physical hosts is not supported
      • The physical HBAs must have access to the LUNs that will be accessed by the NPIV-enabled virtual machines
      • Ensure that the NPIV LUN ID at the storage layer is the same as the NPIV target ID
    • Guest NPIV only works with Fibre Channel switches
    • NPIV does not support Storage vMotion
    • Unfortunately I don’t have an environment that I can go through and document for you the step-by-step process. The steps below are from the vSphere 5 Storage Guide
    • Configuring NPIV
      • Open the New Virtual Machine wizard.
      • Select Custom, and click Next.
      • Follow all steps required to create a custom virtual machine.
      • On the Select a Disk page, select Raw Device Mapping, and click Next.
      • From a list of SAN disks or LUNs, select a raw LUN you want your virtual machine to  access directly.
      • Select a datastore for the RDM mapping file
      • Follow the steps required to create a virtual machine with the RDM.
      • On the Ready to Complete page, select the Edit the virtual machine settings before completion check box and click Continue. The Virtual Machine Properties dialog box opens.
      • Assign WWNs to the virtual machine.
        • Click the Options tab, and select Fibre Channel NPIV.
        • Select Generate new WWNs.
        • Specify the number of WWNNs and WWPNs.
          A minimum of 2 WWPNs are needed to support failover with NPIV. Typically only 1 WWNN is created for each virtual machine.
      • Click Finish.

 

  • Determine appropriate RAID level for various Virtual Machine workloads
    • Earlier in this objective I covered different RAID levels and their respective advantages/disadvantages. Now lets discuss where these RAID levels fit in best with different workloads
    • Typically when your workloads are read intensive it is best to use RAID 5 or RAID 6. When the workload is write intensive you want to use RAID 1 or RAID 1+0. Hopefully the application owner can give you the read/write percentages so that you can determine which RAID level is best.
    • Here’s an example:
          • Formula: (total required IOPs * read%) + (total required IOPs * write% * RAID penalty) = total IOPs required
        • 400 IOPs required
        • 35% read
        • 65% write
      • RAID1 = (400 * 0.35) + (400 * 0.65 * 2) = 660 IOPs
        • 15K disks required = 4
        • 10K disks required = 5
        • 7.2 disks required =  9
      • RAID5 = (400 * 0.35) + (400 * 0.65 * 4) =  1180 IOPs
        • 15K disks required = 7
        • 10K disks required = 9
        • 7.2 disks required =  16
      • RAID6 = (400 * 0.35) + (400 * 0.65 * 6) = 1700 IOPs
        • 15K disks required = 10
        • 10K disks required = 14
        • 7.2 disks required =  23
    • As you can see, the number of disks required depends on the RAID level you choose. So when determining which RAID level to choose, you need to factor in the number of disks you have against the level of protection you will provide. Each of the above RAID levels can meet the IOPs required for the workload, but some require more disks dependent upon the RAID level and type of disks.
    • In the above example I would go with RAID 5 on 15K disks. While RAID 1 would only require 4 disks to meet the IOPs requirement, it may actually require more disks because you lose 50% capacity in any give RAID 1 set.
    • A tool built-in to ESXi that can be VERY useful in determining the I/O characteristics  of a virtual machine workload is vscsiStats. I’m not going to go into real detail here as to how exactly to interpret the statistics is pulls, but will provide you with the basics and a super AWESOME blog that really goes into detail and even provides some templates
      • you can run vscsiStats from anywhere within the shell (console or SSH), but keep in mind that the first “S” in “Stats” is captilized
      • To get going, here is the commands you will run to start, along with an explanation of each paramter
[sourcecode language=”text”] # find the world ID for the VM you want to collect statistics on
vscsiStats -l

# this will start the collection. -s tells it to start and -w specifies the world ID
vscsiStats -s -w 466937

# here is what should be returned after entering the command above
# "vscsiStats: Starting Vscsi stats collection for worldGroup 466937, handleID 8207 (scsi0:0)"
# "Success."

# after this runs for a period of time you need to pull what’s been collected using the -w parameter
# for the world ID and -p <stat> for the stat you want to pull (-p can be ioLength, seekDistance, outstandingIOs,
# latency, interarrival and all. Use the -c parameter to specify a csv format
vscsiStats -w 466937 -p all -c

# once you’re done you want to stop the collection
vscsiStats -x
[/sourcecode]

      • If you want to learn how to interpret these results check out Erik Zandboer’s three-part series, it is definitely a useful resource

 

  • Apply VMware storage best practices
    • Best practices for storage and vSphere will always require a look at your storage vendor’s documentation as it will differ across platforms. However, from the vSphere side we can apply general best practices regardless of the underlying storage platform
    • Best Practices for Fibre Channel Storage
      • First and foremost you should document the environment
        • includes software versions, zoning LUN masking, etc…
      • Only one VMFS datastore per LUN
      • Disable automatic host registration
        • GUI – Modify Advanced Settings > Disk > Disk.EnableNaviReg = 0

image

        • the esxcli way
[sourcecode language=”text”] esxcli system settings advanced set -i=0 -o "/Disk/EnableNaviReg"
[/sourcecode]
      • Use read/write caching on the array
      • ensure non-ESXi hosts are not accessing the same LUNs or physical disks as your ESXi hosts
      • Ensure you have paths to all storage processors for proper load balancing and redundancy
      • Enable Storage I/O Control (SIOC)
      • Ensure you design your storage with proper IOPs in mind (see above section on identifying proper RAID levels)
      • use a dual redundant switching fabric
      • match all queue depths across the application, guest OS, ESXi host, HBA and storage array
    • Best Practices for iSCSI
      • Document the environment
      • Use on one VMFS datastore per LUN
      • Enable read/write cache on the array
      • only ESXi hosts should be accessing the LUN(s) and underlying physical disks
      • Ensure each ESXi hosts has the appropriate number of network adapters to handle throughput for iSCSI traffic
      • Bind multiple network adapters to the iSCSI software adapter for redundancy
      • match all queue depths across the application, guest OS, ESXi host and storage array
      • separate uplinks on the physical switch so they are not using the same buffers
      • Ensure you don’t have Ethernet bottle necks going to your storage (or anywhere for that matter)
      • Isolate storage traffic to its own VLAN if possible
      • Enable Storage I/O Control (SIOC)
    • Best Practices for NFS
      • Isolate storage traffic to its own VLAN if possible
      • Enable Storage I/O Control (SIOC)
      • Mount all NFS exports the same across all hosts
      • If you increase the max number of NFS mounts for a hosts, be sure to also increase the heap size accordingly
        • Increase Max NFS volumes through the GUI
          • Modify Advanced Settings > NFS > NFS.MaxVolumes
        • The esxcli way
[sourcecode language=”text”] esxcli system settings advanced set -i=32 -o "/NFS/MaxVolumes"
[/sourcecode]
        • Increase the TCP Heap Size through the GUI (changing the heap size requires a reboot of the ESXi host)
          • Modify Advanced Settings > Net > Net.TcpipHeapSize
        • The esxcli way
[sourcecode language=”text”] esxcli system settings advanced set -i=16 -o "/Net/TcpipHeapSize"
[/sourcecode]

 

  • Understand the use cases for Raw Device Mapping
    • In order to understand why you would use a Raw Device Mapping (RDM), we need to define it.  “An RDM is a mapping file in a separate VMFS volume that acts as a proxy for a raw physical storage device” – vSphere Storage Guide
    • RDMs come in two flavors; physical compatibility mode and virtual compatibility mode
      • Physical compatibility mode:
        • The VMkernel passes all SCSI commands to the mapped device with the exception of the REPORT LUNs command. This command is virtualized so that the VMkernel can isolate the mapped device to whichever virtual machine owns it
        • Can be greater than 2TB in size (assumes VMFS5)
      • Virtual compatibility mode:
        • Unlike physical compatibility mode, virtual mode will only pass the READ and WRITE command to the mapped device, all other SCSI commands are handled by the VMkernel
        • Cannot be greater than 2TB
    • There are certain scenarios in which you don’t have a choice but to use RDMs:
      • When using Microsoft Clustering Services across physical hosts. Any cluster data disks and quorum disks should be configured as a RDM
      • If at any point you want to use N-Port ID Virtualization (NPIV) within the guest you will need to use a RDM
      • If you need to run SAN management agents inside a virtual machine
    • To fully understand the use cases for RDMs you must also know their limitations
      • Virtual machine snapshots are only available when using a RDM in virtual compatibility mode
      • You can’t map to a certain partition on a device, you must map to the entire LUN
      • You cannot use direct attached storage devices to create a RDM (direct attached devices do not export the SCSI serial number, which is required for a RDM)
    • Now that you have read what a RDM is, the available modes, when you MUST use them and what some of their limiting factors are you can start to narrow down the use cases. To furthur assist you here is a table from the vSphere Storage Guide that outlines the feature sets when using VMFS, virtual RDM and physical RDM
ESXi Features Virtual Disk File Virtual Mode RDM Physical Mode RDM
SCSI Commands Passed Through No No YesREPORT LUNs is not passed through
vCenter Server Support Yes Yes Yes
Snapshots Yes Yes No
Distributed Locking Yes Yes Yes
Clustering Cluster-in-a-box only Cluster-in-a-boxcluster-across-boxes Physical-to-virtual clusteringcluster-across-boxes
SCSI Target-Based Software No No Yes

 

 

  • Configure vCenter Server storage filters
    • There are four different storage filters that can be configured; VMFS Filter, RDM Filter, Same Host and Transports Filter and the Host Rescan Filter. If you don’t know what these are, here is a quick explanation:
      • VMFS Filter: filters out storage devices or LUNs that are already used by a VMFS datastore
      • RDM Filter: filters out LUNs that are already mapped as a RDM
      • Same Host and Transports Filter: filters out LUNs that can’t be used as a VMFS datastore extend.
        • Prevents you from adding LUNs as an extent not exposed to all hosts that share the original VMFS datastore.
        • Prevents you from adding LUNs as an extent that use a storage type different from the original VMFS datastore
      • Host Rescan Filter: Automatically rescans and updates VMFS datastores after you perform datastore management operations
    • You create these filters from vCenter through Administration > vCenter Server Settings… > Advanced Settings. From here you enter in a new Key/Value pair and click the Add button
    • Once those settings are added there are a few different places you can view them:
      • within the Advanced Settings window of where you added them
      • The vpxd.cfgfile on your vCenter server (C:\ProgramData\VMware\VMware VirtualCenter)
        • located between the <filter></filter> tags
      • you can also view the vpxd.cfg file from the ESXi host itself (/etc/vmware/vpxa)
    • All storage filters are enabled by default. To disable them set the following keys to false
VMFS Filter config.vpxd.filter.vmfsFilter
RDM Filter config.vpxd.filter.rdmFilter
Same Hosts and Transports Filter config.vpxd.filter.SameHostAndTransportsFilter
Host Rescan Filter config.vpxd.filter.hostRescanFilter

 

    • Here is a short video of Configuring vCenter Server Storage Filters

 

  • Understand and apply VMFS resignaturing
    • VMFS resignaturing occurs when you you are trying to mount a new LUN to a host that already has a VMFS datastore on it. You have three options when mounting a LUN to an ESXi host with an existing VMFS partition; Keep the existing signature, Assign a new signature and Format the disk. Here is a brief description of each of those options
      • Keep the existing signature: Choosing this option will leave the VMFS partition unchanged. If you want to preserve the VMFS volume (keep the existing UUID), choose this option. This is useful when you are doing LUN replication to a DR site and need to mount the cloned LUN – MUST BE WRITABLE
      • Assign a new signature: Choosing this option will delete the existing disk signature and replace it with a new one. You MUST use this option (or the format option) if the original VMFS volume is still mounted (you can’t have two separate volumes with the same UUID mounted simultaneously). During resignaturing a new UUID and volume label are assigned, which consequently means that any virtual machines that are registered on this VMFS volume must have their configuration files updated to point to the new name/UUID or the virtual machines must be removed/re-added back to the inventory
      • Format the disk: Nothing much new here; choosing this option is the same as creating a new VMFS volume on a blank LUN – – ALL EXISTING DATA WILL BE LOST
    • There are two way that you can add an LUN with an existing VMFS volume to a host; through the GUI and through the command line. The following assumes your host has access to the LUN on the array side:
    • Adding a LUN with an Existing VMFS Volume using the GUI
      1. From within the vSphere client, either connect to vCenter or directly to a host, navigate to the Hosts and Clusters view: Home > Hosts and Clusters (or Ctrl + Shift + H)
      2. Select the host you want to add the LUN to on the right > select the Configuration tab
      3. Click on the Storage Hyperlink
      4. Click the Add Storage… hyperlink in the upper right
      5. Select Disk/LUN > click Next
      6. Select the appropriate LUN > click Next
      7. Select one of the aforementioned options (Keep the existing signature, Assign a new signature or Format the disk)
      8. Click Finish
      9. If you are connected to vCenter you may receive the following error during this process
          1. image
          2. Check out VMware KB1015986 for a workaround (connect directly to the host and add the LUN)

       

    • Adding a LUN with an Existing VMFS Volume using esxcli
      1. SSH or direct console to the ESXi host that you want to add the LUN with the existing VMFS volume to  — You can also connect to a vMA instance and run these commands
      2. Once connected you need to identify the ‘snapshots’ (which volumes have an existing VMFS volume on it)
[sourcecode language=”text” padlinenumbers=”true”] # This will list the snapshots that are available
esxcli storage vmfs snapshot list

# Mount a snapshot named ‘replicated_lun’ and keep the existing signature (find the snapshot you want
# to mount using the output from the previous command
esxcli storage vmfs snapshot mount -l ‘replicated_lun’

# Mount a snapshot named ‘replicated_lun’ and assign a new signature (find the snapshot you want
# to mount using the output from the first command
esxcli storage vmfs snapshot resignature -l ‘replicated_lun’
[/sourcecode]

    • Here is a video showing you how to mount a VMFS volumes that has an identical UUID as another volume. It will show you how to mount a volume while keeping the existing signature and by applying a new signature; all using esxcli Enjoy!

 

  • Understand and apply LUN masking using PSA-related commands
    • LUN masking gives you control over which hosts see which LUNs. This allows multiple hosts to be connected to a SAN with multiple LUNs while allowing only hosts that you specify to see a particular LUN(s). The most common place to do LUN masking is on the back-end storage array. For example, an EMC Clariion or VNX provides LUN masking by way of Storage Groups. You add hosts and LUNs to a storage group and you have then essentially “masked” that host to only seeing those LUNs.
    • Now that we have a better idea of what LUN masking is, let’s go into an example of how you would actually do this on an ESXi host.
    • The first thing we need to do is identify which LUN we want to mask. To do this:
      • esxcfg-scsidevs -m  — the -m will display only LUNs with VMFS volumes, along with the volume label. In this example we are using the “vmfs_vcap_masking” volume”
      • image
      • Now that we see the volume we want, we need to find the device ID and copy it (starts with “naa.” In this example our device ID is naa.5000144fd4b74168
    • We have the device ID and now we have to find the path(s) to that LUN
      • esxcfg-mpath -L | grep naa.5000144fd4b74168  — the -L parameter gives a compact list of paths
      • image
      • We now see there are two paths to my LUN, which are C0:T1:L0 and C2:T1:L0
    • Knowing what are paths are we can now create a new claim rule, but first we need to see what claim rules exist in order to not use an existing claim rule number
      • esxcli storage core claimrule list
      • image
    • We can use any rule numbers for our new claim rule that isn’t in the list above. We’ll use 500. Now lets create the new claim rule for the first path; C0:T1:L0 which is on adapter vmhba35
      • esxcli storage core claimrule add -r 500 -t location -A vmhba35 -C 0 -T 1 -L 0 -P MASK_PATH   — you know the command succeeded if you don’t get any errors.
    • Masking one path to a LUN that has two paths will still allow the LUN to be seen on the second path, so we need to mask the second path as well. This time we’ll use 501 for the rule number and C2:T1:L0 as the path. The adapter will still be vmhba35
      • esxcli storage core claimrule add -r 501 -t location -A vmhba35 -C 2 -T 1 -L 0 -P MASK_PATH — you know the command succeeded if you don’t get any errors.
    • Now if you run esxcli storage core claimrule list again you will see the new rules, 500 and 501 but you will notice the Class for those rules show as file which means that it is loaded in /etc/vmware/esx.confbut it isn’t yet loaded into runtime. Let’s load our new rules into runtime
      • esxcli storage core claimrule load
      • Now run esxcli storage core claimrule list and this time you will see those rules displayed twice, once as the file Class and once as the runtime Class
      • image
    • Only one more step left. Before those paths can be associated with the new plugin (MASK_PATH), they need to be disassociated from the plugin they are currently using. In this case those paths are claimed by the NMP plugin (rule 65535). This next command will unclaim all paths for that device and then reclaim them based on the claimrules in runtime. Again we’ll use naa.5000144fd4b74168to specify the device
      • esxcli storage core claiming reclaim -d naa.5000144fd4b74168
      • After about 30 seconds, if you are watching the storage area on your host within the vSphere client you will see that datastore disappear from the list
      • Running esxcfg-mpath -L | grep naa.5000144fd4b74168 again will now show 0 paths(before it showed 2)
    • Here is a quick list of commands you would need to run if you wanted to unmask those two paths to that LUN and get it to show up again in the vSphere client
[sourcecode language=”text”] esxcli storage core claimrule remove -r 500
esxcli storage core claimrule remove -r 501
esxcli storage core claimrule load
esxcli storage core claiming unclaim -t location -A vmhba35 -C 0 -T 1 -L 0
esxcli storage core claiming unclaim -t location -A vmhba35 -C 2 -T 1 -L 0
esxcli storage core adapter rescan -A vmhba35
[/sourcecode]
    • Here is a pretty awesome video of performing LUN masking using the all powerful OZ esxcli

 

  • Identify and tag SSD devices
    • There are a few ways that you can identify an SSD device. The easiest way is to look in the storage area (select host > click Configuration >  click the Storage hyperlink) and look at the Drive Type column of your existing datastores. This will either say Non-SSD or SSD
      • image
    • Now you can only use the previous method if you already have a datastore mounted on that LUN. If you don’t, SSH into your host and let’s use esxclito figure out which devices are SSDs
      • esxcli storage core device list
          • image
        • The Is SSD will show True or False
    • The PowerCLI Way
      [sourcecode language=”powershell” padlinenumbers=”true”] $esxcli = Get-EsxCli
      $esxcli.storage.core.device.list()

      #Here is the output (truncated)

      #AttachedFilters :
      #DevfsPath : /vmfs/devices/disks/na
      #Device : naa.5000144f60f4627a
      #DeviceType : Direct-Access
      #DisplayName : EMC iSCSI Disk (naa.50
      #IsPseudo : false
      #IsRDMCapable : true
      #IsRemovable : false
      #IsSSD : true
      #Model : LIFELINE-DISK

      [/sourcecode]
    • Identifying a SSD device is easy when they are detected automatically, but what if your SSD device isn’t tagged as a SSD by default? The answer is you can manually tag them. This has to be done with our good friend esxcli
      • First you need to identify which device is not being tagged automatically (there are multiple ways of tagging the device, in this example we will use the device name) Run the following command so you can get the Device Display Name and the Path Selection Policy
        • esxcli storage nmp device list
        • image
        • In this example the device name will be naa.5000144f60f4627a and the PSP will be VMW_SATP_DEFAULT_AA— now we must add a PSA claim rule specifying the device, the PSP and the option to enable SSD
          • esxcli storage nmp satp rule add -s VMW_SATP_DEFAULT_AA -d naa.5000144f60f4627a -o enable_ssd    — no result should be displayed
        • Just like our claimrules in the previous section, we need to unclaim the device and load the claimrules into runtime. An additional step is also needed to execute the claimrules (this step was not required when creating LUN Masking claim rules). Again, you will need the device ID for the next command (naa.5000144f60f4627a)
        [sourcecode language=”text”] # unclaim the device
        esxcli storage core claiming unclaim -t device -d naa.5000144f60f4627a

        # load the claim rules into runtime
        esxcli storage core claimrule load

        # execute the claim rules
        esxcli storage core claimrule run

        # if the device is already mounted you will see it disappear from the Datastore view
        # and then reappear with a Drive Type of SSD
        [/sourcecode]

 

  • Administer hardware acceleration for VAAI
    • Since I only have block storage in my lab I will not be showing examples hardware acceleration for NFS, but will list procedures and capabilities for it
    • Within the vSphere client you can see whether Hardware Acceleration is supported for your device (click on a host > click configuration > click the Storagehyperlink)
      • image
    • The hardware acceleration available for block devices are:
      • Full Copy
      • Block Zeroing
      • Hardware Assisted Locking (ATS)
      • Unmap
    • If your device is T10 compliant, it uses the the T10 based SCSI commands, therefore enabling hardware acceleration support without the use of the VAAI plugin. If your device is not T10 compliant (or is partially) the VAAI plugin is used to bridge the gap and enable hardware acceleration
    • Display Hardware Acceleration Plug-Ins and Filter
      • esxcli storage core plugin list -N VAAI  — displays plugins for VAAI
      • esxcli storage core plugin list -N Filter  — displays VAAI filter
      • image
    • Displaying whether the device supports VAAI and any attached filters (for this example I’m using naa.6006016014422a00683427125a61e011as the device)
      • esxcli storage core device list -d naa.6006016014422a00683427125a61e011
      • image
    • Display VAAI status of each primitive on a device (again using naa.6006016014422a00683427125a61e011)
      • esxcli storage core device vaai status get -d naa.6006016014422a00683427125a61e011
      • image
    • Before we move on to adding hardware acceleration claim rules, lets check out how to display the current claim rules for filters and for VAAI
      • Filter — esxcli storage core claimrule list –c Filter
      • VAAI — esxcli storage core claimrule list –c VAAI
    • Adding hardware acceleration claim rules is a 5 step process. The first two steps are creating two claim rules, one for the VAAI filter and another for the VAAI plugin. The third and fourth steps are loading the claim rules into runtime. The last step is executing the claim rules. Since you are doing this manually you would need to know the Type information, in our case is Vendor and the Vendor information which in this case will be vlabs. Let’s get to it:
[sourcecode language=”text”] # this will create a new claim rule for the VAAI_Filter with a type of "Vendor" and the vendor will be "vlabs"
# the -u parameter automatically assigns the rule number
esxcli storage core claimrule add -c Filter -P VAAI_FILTER -t vendor -V vlabs -u

# this will create a new claim rule for the VAAI Plugin with a plugin name of "VMW_VAAI_VLABS"
# the -f parameter is being used to force the command as the aforemention plugin name is not registered
esxcli storage core claimrule add -c VAAI -P VMW_VAAI_VLABS -t vendor -V vlabs -u -f

# load the filter plugin claim rule into runtime
esxcli storage core claimrule load -c Filter

# load the VAAI plugin claim rule into runtime
esxcli storage core claimrule load -c VAAI

# execute the new claim rules
esxcli storage core claimrule run -c Filter
[/sourcecode]

    • For NFS you will need to install the plug-in provided by your array vendor and then verify the hardware acceleration (use esxcli storage nfs list). To see the full procedure for installing and updating NAS plugins see pages 177-180 of the vSphere Storage Guide

 

  • Configure and administer profile-based storage
    • Before we can administer profile-based storage we first must configure it (I know “DUH”). Of course, before we can configure it we must have a basic understanding of the elemts of profile-based storage. Profile-based storage are profiles of certain storage features an array might have. Those features are added as a capability (if they are not already defined by the array). There are system-defined capabilities and user-defined capabilities. Here are a list of basic steps on the road to profile-based storage
      • Create user-defined capabilities (optional) to go along with any system-defined capabilities
      • Associate those capabilities with datastores that coincide with said capability
      • Enable virtual machine storage profiles (host or cluster level)
      • Create virtual machine storage profiles
      • Associate a virtual machine storage profile with virtual disks or virtual machine files
      • Check for compliance of the associated storage profile on the virtual machine
    • Let’s create some user-defined storage capabilities.
      • Log into vCenter using the vSphere client and click the Home button in the navigation bar
      • Under Management click the VM Storage Profiles button
      • Just under the navigation bar, click Manage Storage Capabilities
      • You’re now presented with a dialog box where you can add your own. Click the Add. . . button
      • Type the Name of the capability > give it a Description > click OK
      • I’ve created three user-defined capabilities; vcap5-dca, 7200 Disks and SSD
      • image
      • When you’re finished adding capabilities, click the Close button
    • We’ve created the capabilities, but now we need to associate them with a datastore(s)
      • Navigate to the Datastores and Datastore Cluster view (Home > Inventory > Datastores and Datastore Clusters or use the hot keys Ctrl + Shift + D)
      • Right-click on the datastore that you want to assign a capability to > click Assign User-Defined Storage Capability…
      • image
      • From the drop-down menu select an existing storage capability (you can also create a new capability from here should you need to by clicking the New…button)
            • image
      • Click OK
      • Repeat on all datastores in which you need to assign a user-defined storage capability. If you are assigning the same storage capability to multiple datastores you can select them all at once and then assign the capability
      • NOTE: You can only assign one storage capability per datastore
    • We need to create virtual machine storage profiles, but first we must enable this on either a host or a cluster
      • In the vSphere client and click the Home button in the navigation bar
      • Under Management click the VM Storage Profiles button
      • Under the navigation bar click Enable VM Storage Profiles
      • From here you can select a particular cluster
        • ALL hosts within the cluster must have a Licensing Status of Licensed. Any other status, such as Unknown and you will not be able to enable it
      • Once you’ve selected which cluster you want click the Enable hyperlink in the top right
      • image
      • Click the Close button once the VM Storage Profile Status changes to Enabled
    • Creating a new VM Storage Profile
      • In the vSphere client and click the Home button in the navigation bar
      • Under Management click the VM Storage Profiles button
      • Under the navigation bar click Create VM Storage Profile
      • Enter in a descriptive name (such as a defined SLA, e.g. Platinum)
      • Enter in a description for the new profile > click Next
      • Select which storage capabilities should be a part of this profile. For this example I’m selecting the vcap5-dcacapability)
        • BE CAREFUL HERE. If you select more capabilities than exist on a single datastore then a VM that has this particular storage profile applied to it will never show up as compliant
      • Click Next > click Finish
    • We have successfully created a VM Storage Profile, but it won’t do us any good until we associate it with a virtual machine
      • In the vSphere client navigate to the VMs and Templates view (Home > Inventory > VMs and Templates or press Ctrl + Shift + V)
      • Right-click on a virtual machine that you want to apply a VM Storage Profile to > click VM Storage Profile > Manage Profiles…
      • image
      • From the drop-down menu choose a profile. In our case it’s the Platinum profile
      •  From here you have two options. You can click Propagate to disks, which will associate all virtual disks for that VM to the Platinum profile. If you don’t want to propagate to all the disks you can manually set which disks you want to be associated with that profile
      • In this example I am forgoing the propagate option and only setting this on Hard disk 1
      • image
      • Click OK when you are finished
    • Lastly, we need to check the compliance of the VM Storage Profile as it relates to that particular VM
      • In the vSphere client navigate to the VMs and Templates view (Home > Inventory > VMs and Templates or press Ctrl + Shift + V)
      • Click on the virtual machine that you just associated the VM Storage Profile with and click the Summary tab (should be default)
      • Look at the VM Storage Profiles section and check the Profile Compliance
      • image
      • Here it will list whether it is compliant or not and the last time it checked (if you need to check it again for compliance you can initiate that by right-clicking the VM > click VM Storage Profile > Check Profiles Compliance

 

  • Prepare storage for maintenance (mounting/un-mounting)
    • Should you need to perform storage maintenance on disks that make up a VMFS volume you will want to unmount it from vSphere. Here are a list of prerequisites for a VMFS datastore before it can be unmounted
      • No virtual machine resides on the datastore
      • The datastore is not part of a Datastore Cluster
      • The datastore is not managed by storage DRS
      • Storage I/O control is disabled for this datastore
      • The datastore is not used for vSphere HA heartbeating
    • To un-mount a datastore perform the following steps:
      • In the vSphere client, navigate to the Hosts and Clusters view
      • Select a host on the left and click the Configuration tab on the right > click the Storage hyperlink
      • Right-click on the datastore you want to un-mount and click Unmount
      • Verify that all the aforementioned checks have passed validation > click OK
      • image
      • If any of the requirements fail to validate then you will not be able to unmount the datastore
    • Using esxcli (I’m using the vmfs_vcap_masking datastore)
      • esxcli storage filesystem unmount -l vmfs_vcap_masking
        • There are scenarios where the GUI won’t let you un-mount a volume, say for example the datastore has a virtual machine on it. In this instance, even if the VM is powered off the GUI won’t let you unmount the datastore. Using the esxcli command above however will let you unmount the datastore IF the VM is powered off
        • If you try to unmount a datastore via esxcli while a powered on VM resides on that datastore you will receive the following error
        • image
        • Here is more information from the vmkernel log (screenshot is broken up)
  • imageimage

 

    • Once you’re complete with you maintenance you want to mount the volume
      • In the vSphere client, navigate to the Hosts and Clusters view
      • Select a host on the left and click the Configuration tab on the right > click the Storage hyperlink
      • Right-click on the datastore you want to mount and click Mount
      • Monitor the Recent Tasks pane to see when the operation is complete. Once complete the datastore will be available
      • Using esxcli (I’m using the vmfs_vcap_masking datastore)
        • esxcli storage filesystem mount -l vmfs_vcap_masking

 

  • Upgrade VMware storage infrastructure
    • As with unmourning/mounting datastores, upgrading your VMware storage infrastructure, particularly upgrading to VMFS5, can be done through the GUI or using esxcli. Here are a few facts about upgrading from VMFS3 to VMFS5
      • VMFS5 has a 1MB block size regardless of disk file size
      • VMFS5 sub-blocks are now 8KB (VMFS3 is 64KB)
      • Block size you used on your VMFS3 partition will carry-over to the VMFS5 partition
      • The disk type of your newly upgraded VMFS5 partition will remain MBR until it exceeds the 2TB limit, at which it will automatically be converted to a GPT disk
      • The upgrade can be done online without disruption to running virtual machines
    • If you have any VMFS2 partitions you will need to first upgrade them to VMFS3 and then you can upgrade to VMFS5
    • If you prefer to build new VMFS5 partitions instead of upgrading, but don’t have space to create a new volume you can use the VM shuffle methodology to move VMs off one datastore to another, wipe the partition and create a new one and then continue with the shuffle until all VMFS datastores are complete. Conrad Ramos wrote a PowerCLI script to automate this, check it out here
    • Upgrade VMFS3 to VMFS5 via the vSphere Client
      • In the vSphere client, navigate to the Hosts and Clusters view
      • Select a host on the left and click the Configuration tab on the right > click the Storage hyperlink
      • Click on the datastore you want to upgrade > below the Datastore pane on the right, click the Upgrade to VMFS-5… hyperlink
      • image
      • Click OK to perform the upgrade
    • Upgrade VMFS3 to VMFS5 via esxcli (upgrading a volume with the name of vmfs3_upgrade)
      • esxcli storage vmfs upgrade -l vmfs3_upgrade
      • once the command completes you will see that volume reflected as VMFS5 under the Type column of the Datastore Views section within the vSphere client

Tools

Command-line Tools

  • vscsistats
  • esxcli
  • vifs
  • vmkfstools
  • esxtop/resxtop

Comments 11

  1. Post
    Author

    Thanks for the comment Gregg.

    I didn’t go through the VCAP4-DCA so I’m starting from scratch. It definitely helps to take a look and yours and Jason Langer’s VCAP4 write-ups whenever I’m not 100% confident that the content I’m writing is actually the intent of a particular objective :-).

    My goal is to have gone through the entire blueprint in this fashion by VMworld (which I hope VCAP5-DCA is GA by then).

  2. Pingback: VCAP5-DCA Objective 1.2 – Manage Storage Capacity in a vSphere Environment » ValCo Labs

  3. Pingback: VCAP5-DCA-Objective 3.2–Optimize Virtual Machine Resources » ValCo Labs

  4. Pingback: VCAP 5 DCA Useful Study Links | Electric Monk

  5. Pingback: VCAP5-DCA – Objective 6.4 – Troubleshoot Storage Performance and Connectivity » ValCo Labs

  6. Pingback: VCDA510 – Objective 1.1 – Implement and Manage Complex Storage Solutions – Skills And Abilities | VCDX or Bust

  7. Pingback: Yet Another VCAP-DCA post – Sección 1 – Objetivo 1.1 / Parte 3 | Chubascos Blog

  8. Pingback: Yet Another VCAP-DCA post – Sección 1 – Objetivo 1.1 / Parte 4 | Chubascos Blog

  9. Hi Josh,

    I don’t understand how you calculated the number of disks required based on being 10k, 15k or 7.2k rpm, I understand IOPS calculation but not the rpm one.
    What am I missing ?

    Many thanks.

    1. Post
      Author

      Hi Giuliano,

      Each of those types of disk (15K, 10K, 7.2K) have a limited amount of IOPs that they can provide. Given that, if you have a certain IOP requirement you can determine the number of disks required based on number of IOPs per disk type and RAID level. Hope that helps with your question. If not, please try and rephrase the question and I will answer in a much more timely fashion.

      -Josh

Leave a Reply to Giuliano Cancel reply

Your email address will not be published. Required fields are marked *

*