Setting Perennial Reservations for MSCS with EMC CX/VNX Arrays

A few months ago I was putting together a technical solution to support a vSphere upgrade (physical hosts and vSphere version) and a SAN upgrade, As part of the upgrade, the customer wanted to virtualize their file services as well. The customer wanted the same level of availability and redundancy in the virtual environment that they had in the physical environment; this meant Microsoft Clustering (MSCS). I thought, easy, vSphere 5.x supports MSCS across physical hosts (cluster across box) via physical RDMS. While the physical-to-virtual conversion process was relatively simple, the hosts in the cluster started experiencing odd behavior. Rescanning for new devices or VMFS volumes was taking 10 minutes or more. If I rebooted one of the hosts that would also take 10+ minutes. After a bit of research, and speaking with our VMware TAM, I figured out the problem; perennial reservations. Perennial reservations themselves was not the issue, it was more the lack of perennial reservations that was causing the problem.

What are perennial reservations and why are they needed? Whenever a LUN is participating in a MSCS cluster, the active node has ownership of that device(s) using permanent SCSI reservations. Now, whenever you rescan for devices on an ESXi host, or are booting an ESXi host, the host tries to query all devices that it can see, including the devices used for MSCS. Now, I’m not exactly sure what takes place during the query process, but because the MSCS device(s) are already have a permanent SCSI reservation, the ESXi query to the device fails, and will continue to fail until the host decides to move on (I would LOVE to know the actual process).

To solve the problem of of ESXi trying to query MSCS owned devices, a flag has been introduced on a device called Is Perennially Reserved. By default this flag is set to false. By setting this flag to true it lets the ESXi host know to essentially, NOT query this device during rescans/boot time.  Here’s VMware KB 1016106 that describes that problem/resolution

isPerennialyReserved Flag

There are a few different methods in which you can set this flag. If you are running vSphere 5.1 you can set this flag using Host Profiles and modifying the Device perennially reserved status. This is set under the PSA Device Setting under the Pluggable Storage Architecture (PSA) configuration portion of the Host Profile. The setting is set per device. You can also set this flag using the ever popular esxcli by running the following command

 

You can also use PowerCLI to invoke esxcli to set these commands using the get-esxcli cmdlet.

In this particular environment, there are 30 separate virtual environments, each with their own SAN, and each with their own set of Microsoft clusters. There were two challenges that presented itself here. One, the environment was being upgraded to vSphere 5.0, Update 2, so host profiles wasn’t an option. Two, each site had their own SAN, therefore the device IDs would be different for each environment.

Since the environment used all EMC VNX arrays, I decided to use a combination of Clint Kitson’s (blog / twitter) powershell scripts to gather CX/VNX data and PowerCLI to solve the problem.

Requirements:

  • exec_naviseccli_cmd.ps1
  • get_naviseccli.ps1
  • The set-perennialReservations powershell script below
  • Navisphere CLI (can be found on Powerlink)
  • PowerCLI
  • You must be using a CX or VNX branded storage array
  • You must know the LUN numbers of the LUNs you want to set (as shown in Unisphere)

The Script

At the beginning of the script you’ll see three variables named $lun1, $lun2 and $lun3. Make sure you change the numeric values inside each string to correspond to the LUN numbers you want to set. I’m also running this against hosts in a cluster, but you can remove that if you need to do it across all hosts in vCenter. Again, you’ll notice this script is based on setting perennial reservations on 3 LUNs. You can increase or decrease this as you wish, just copy or remove the relevant lines. If you need assistance modifying the script, or running it, please leave a comment below and I’ll help you where I can. Thanks again to Clint Kitson for the scripts he wrote to gather CX/VNX information. It has proven useful to me time and time again.

Leave a Reply

Your email address will not be published. Required fields are marked *

*