There are many resources that discuss vMSC. A few can be found here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007545

http://www.emc.com/collateral/software/white-papers/h11065-vplex-with-vmware-ft-ha.pdf

The purpose of this article is to discuss Uniform and Non-Uniform storage access, not recreate the work they have done. After reviewing the documents linked above, I was left scratching my head.

Ok, the way VMware defines it (see VMware link about)

• Uniform

Uniform Host Access (Cross-Connect) – This deployment involves establishing a front-end SAN across the two sites, so that the hosts at one site can see the storage cluster at the same site as well as the other site.

• Non-Uniform

Non-uniform Host Access – This type of deployment involves the hosts at either site seeing the storage volumes through the same site storage cluster only.

Now, the way EMC defines it (see EMC link above)

• Uniform

Uniform access is typically based on active/passive technology where all I/O is serviced by only 50% of the available storage controllers in the same physical location (i.e. 50% of the controllers are passive); therefore, all I/O is sent to or received from the same location where the active controller resides, hence the term “uniform”. Typically this involves “stretching” dual controller active/passive mid-range storage products, but can also be architected by using legacy active/passive replication. In both cases the use of an ISL is typically required so all hosts can access the active storage controllers at the remote location. These two types of uniform access are known as “split cluster” and “replication” uniform access respectively.

Figure 10 from the EMC Article located above.

• Non-Uniform

I/O can be serviced by any available storage controller (100%) at any given location; therefore I/O can be sent to or received from any storage target location, hence the term “nonuniform”. This is derived from “distributing” multiple active controllers/directors in each location and does not require an ISL (although an ISL can be optionally deployed).

Figure 11 from the EMC Article located above.

To me it appeared that VMware and EMC were describing the technology the same, but that their definitions of Uniform and Non-Uniform were reversed. Not a huge deal, but confusing when trying to explain why there were different.

Moment of Clarity

After reading (and re-reading) Duncan Epping’s blog post on the subject, (http://www.yellow-bricks.com/2012/11/13/vsphere-metro-storage-cluster-uniform-vs-non-uniform/) it started to make sense. The confusion came from VMware’s article mentioning “Uniform (cross-connect)” and it’s reference to VPlex.

Although VPlex can be configured to provide uniform access, where you see the most benefit is from non-uniform access. In fact, using uniform access would remove many of its benefits in a vMSC. Uniform access implies that we have an active/passive site configuration or storage replication technology. With VPlex, however, all controllers are active at both sites, and when configured for non-uniformed (cross-connect), all host paths from a host in site A, to its local VPlex front-end ports, as well as the paths to the remote sites VPlex front-end ports are active.

Non-Uniform (VPlex IO access pattern)

VPlex is ideally configured in (2) modes, non-uniform, and non-uniform (cross-connect).

• Non-Uniform
  • Hosts at SiteA are zoned ONLY to the SiteA VPlex cluster
  • Hosts at SiteB are zoned ONLY to the SiteB VPlex cluster
  • ALL write activity of a distributed volume (DVol) traverses the VPlex WANCOM
• Non-Uniform (cross-connect)
  • Host at SiteA are zoned to SiteA AND SiteB VPlex clusters
  • Host at SiteB are zoned to SiteA AND SiteB VPlex clusters
  • Write activity COULD be serviced by either VPlex cluster (See note below)

NOTE: Although technically the write activity in a VPlex non-uniform cross-connected configuration shows all paths as active, VMware/EMC recommended a ‘FIXED’ multipathing policy for these devices to prevent writes to the non-local VPlex cluster. This makes sense if you think about the write process of the host.

Host Write Penalty Using Non-local Path

From the diagram below you can see that if all paths are active, and a write is sent from a host in SiteA to the remote VPlex cluster (non-preferred), the write has to be mirrored back to the local cluster. (2) writes must be sent across the inter-site link. (1) write across the host WAN connectivity, and (1) write across the VPlex WANCom. If writes are only sent to the local (preferred) cluster, only (1) write needs to be sent to the remote cluster.

What about Uniform and VPlex

As I mentioned previously, VPlex is typically configured in a non-uniform configuration. There are two times where Uniform may be seen.

• Forced Uniform (Failure of WANCom/VPlex Cluster Partition)

During a WANCom partition, certain rules (or witness election) are enacted to determine which site is the authoritative source for write. This is necessary to avoid a split brain condition, and ultimately corruption. In this state, storage is only accessible at the surviving site, and hosts at both sites (assuming they are cross-connected) MUST access storage from a single site. If the hosts are NOT cross-connected, manual intervention may be required depending on the version of VMware ESXi. Check out the failure scenarios table on page 38 or the EMC article above.

Figure 13 from the EMC Article located above.

• Intentional (Uniform Configuration)

While technically it is possible, and there may be a specific use case, you could create a VPlex virtual volume at a single site, and forgo things like AccessAnywhere to make the volume available at the remote location. In this configuration, hosts at both sites could only access the storage at a single site.

Where is Uniform used?

I am not as familiar with other storage virtualization platforms, but can see the need for this configuration if the arrays at SiteA and SiteB are not truly active-active. In this case, hosts at SiteA, and SiteB would have to be ‘Uniform (cross-connected)’ to access the site’s storage that is read/write, while the other site is read-only until the site’s roles are switched. Check out this link for a better example. (http://www.yellow-bricks.com/2012/11/13/vsphere-metro-storage-cluster-uniform-vs-non-uniform/)

Questions?

- What happens to the remaining paths when selecting ‘FIXED’ as the multi-pathing policy? The remaining paths are all ‘active’ (just not active I/O). If there is a failure of the preferred path (local VPlex cluster), what stops a path to the remote VPlex cluster from being selected?

What is Flash Virtualization Platform (FVP)?

FVP promises to bring you, and the datacenter, write-through and write-back caching on a PER VIRTUAL MACHINE basis utilizing server-side PCIe SSD cards or standard SSD drives, while (now this is key), allowing you all the benefits of virtualization that you enjoy today. Now you might ask (assuming you know my name) “Josh, what about vMotion?”, YES, it even supports vMotion. FVP is all software, inserted into the ESXi vmkernel for seamless integration with ESXi and the current data plane, along with integration into the vSphere client for management.

FVP works with any SSD drive, PCIe card, server and storage array listed on VMware’s HCL. FVP works with any application.

How does FVP Work?

From the PernixData homepage, here is an illustration of the FVP concept

For those unaware the difference of write-through and write-back cache, it can be defined simply as:

• Write-through mode functions as read cache only
• Write-back mode functions as read and write cache

FVP, once installed, essentially becomes part of the hypervisor. As a virtual machine writes data to its hard disk, data is sent through FVP (presumably at the direction of the vmkernel) where it is either, cached for future access and passed to the device to perform the write (write-through), or, it is written to FVP and FVP sends the write acknowledgment to the host, and the data is flushed to disk at a later time (write-back).

As I stated earlier, FVP can be set per-virtual machine, on the fly. FVP can also be set on a per-datastore basis, so that any virtual machine that resides on that datastore will reap the benefits provided by FVP.

Write-back mode

Write-back mode has something called flash replicas that are SSD disks that sit on remote hosts, let’s call them remote devices. Writes that are cached on the flash device of the local host are then synchronously replicated to the remote devices, and then the write is acknowledged. In order to use write-back mode ALL hosts within the cluster must have PernixData FVP installed

Support for vMotion

Any company creating a server-side caching platform MUST support VMware vMotion out of the gate, or may not make it much past the start line. PernixData FVP does just that, so let me tell you how it works. When a virtual machine enabled with FVP is migrated to another ESXi host, whatever is in cache (FVP cache) at the time remains on the host. If the virtual machine needs to access data held within that cache, it utilizes the vMotion network to retrieve  it. Once retrieved, it stores the data it just fetched from the remote host, on the local host. Eventually all data resides on the local host which negates the need to pull from remote hosts.

Current Support

• Only supports VMware vSphere — this should be expanding to other platforms in the future
• Only supports block based storage
• Ships with Windows PowerShell support
• Heterogeneous flash devices and sizes across hosts within a cluster is supported
• All VMware vSphere features
• Installs via VMware Update Manager (via offline bundle)

Some unanswered questions…

In watching the PernixData SFD3 presentation (link below) by CTO and co-founder, Satyam Vaghani, I had some questions that weren’t addressed so I wanted to post them here:

1. When using write-back mode, what happens if you have a power outage and the UPS fails (or doesn’t exist); what happens to data that hasn’t been written to disk yet?
2. (I’m sure I know the answer to this one) Any support for RDMs? I know currently, in either compatibility mode, SCSI reads and writes bypass the vmkernel, but if someone could change that I imagine it’d be Mr. VMFS!
3. Are they any plans to enable FVP on a per-VM basis dynamically? — maybe based on trending data or I/O demand/queuing?

Final Thoughts

PernixData FVP is, at it’s core, an ingenious and brilliant scale-out model that just makes sense. It allows the application and storage to become intimate like never before, all while providing low latency and high IOPs, WITHOUT sacrificing features. As you scale out your compute, scale out FVP with some SSDs. I’m positive that FVP will become deeply integrated into any VMware design as it matures and comes out of beta (yes, it’s still in beta!). If you want to join the PernixData FVP beta, send an email to beta@pernixdata.com. I hope to get on the beta so I can take this for a test-drive and see what it can do, but if it can do even half of what it says, then it’s the best piece of software developed in the last 5 years.

Satyam Vaghani’s presentation at SFD3 – http://pernixdata.com/SFD3/

Session 5038 – Achieving High Availability with vSphere Storage Metro Clusters and EMC VPLEX

• The speakers are Joe Kelly and Martin Valencia from Varrow. Both great guys and should definitely have some serious knowledge to drop

Session 4757 – Root cause any storage issue in a vSphere environment

• The speaker is Nathan Small from VMware. The session looks good based on the title and abstract

Session 5279 – The 4-hour VCDX

• The speakers are Ben Lin and Rawlinson Rivera. Both guys are VCDXs, SUPER smart and Rawlinson might be giving away copies of his Exchange 2007 book

Session 4878 – vCloud Director: Designing for Real World Use Cases

• The speaker is Kendrick Coleman from VCE. Kendrick has posted many good vCloud Director articles on his blog, so I expect this session to be of the same high caliber

Session 4887 – The PowerCLI – vSphere API Rosetta Stone

• The speakers are Josh Atwell from VCE and Conrad Ramos from GDC Integration. I’ve known these guys both for over a year; top notch, which should translate to a top notch session

Session 5193 – Everything you need to know about DRS Resource Pools

• The speakers are Aashish Parikh from VMware and Frank Denneman from VMware PernixData. I’m sure this session doesn’t need my endorsement to get through, but you know it’s going to be good

Session 4518 – A Technical Deep Dive on NFS Network Design

• The speaker is Chris Wahl from Ahead. Chris is a certification maniac and posts a lot of good content on his blog; should be a good session

Session 5268 – Technical Deep Dive: esxtop tips and tricks to keep you on top of your vSphere environment

• The speakers are Wen Yu from Nimble Storage and Mostafa Khalil from VMware. ESXTOP is a great tool and Mostafa Khalil is wicked smart; should make for a great session

Session 5150 – Understanding Virtualized Memory Performance Management

• The speakers are Kit Colbert and Rajesh Venkatasubramanian from VMware. Topic and abstract sound good and I’ve seen Mr. Colbert present before (which was awesome).

Session 5420 – Software-Defined Storage: The Art of the Possible

• The speaker is Chad Sakacc from EMC. I don’t need to say anything else.

Session 5640 – The story behind designing and building a distributed automation framework for vCloud Hybrid Services

• The speaker is Nicholas Weaver. This session is probably going to be over my head, but if you know who Nick Weaver is, you’ll vote for it, and show up

Session 5508 – VMware Jeopardy for Support

• The speakers are Matt Mancini, Trey Gwaltney and Pat Thiel from VMware. This session looks fun.

There are also some panels listed that will surely get voted in with your typical list of virtualization/storage rock stars, so make sure you find and vote for those as well. Voting ends close of business on May 6th, happy voting everyone!

What are perennial reservations and why are they needed? Whenever a LUN is participating in a MSCS cluster, the active node has ownership of that device(s) using permanent SCSI reservations. Now, whenever you rescan for devices on an ESXi host, or are booting an ESXi host, the host tries to query all devices that it can see, including the devices used for MSCS. Now, I’m not exactly sure what takes place during the query process, but because the MSCS device(s) are already have a permanent SCSI reservation, the ESXi query to the device fails, and will continue to fail until the host decides to move on (I would LOVE to know the actual process).

To solve the problem of of ESXi trying to query MSCS owned devices, a flag has been introduced on a device called Is Perennially Reserved. By default this flag is set to false. By setting this flag to true it lets the ESXi host know to essentially, NOT query this device during rescans/boot time.  Here’s VMware KB 1016106 that describes that problem/resolution

There are a few different methods in which you can set this flag. If you are running vSphere 5.1 you can set this flag using Host Profiles and modifying the Device perennially reserved status. This is set under the PSA Device Setting under the Pluggable Storage Architecture (PSA) configuration portion of the Host Profile. The setting is set per device. You can also set this flag using the ever popular esxcli by running the following command

esxcli storage core device setconfig -d naa.id --perennially-reserved=true

You can also use PowerCLI to invoke esxcli to set these commands using the get-esxcli cmdlet.

In this particular environment, there are 30 separate virtual environments, each with their own SAN, and each with their own set of Microsoft clusters. There were two challenges that presented itself here. One, the environment was being upgraded to vSphere 5.0, Update 2, so host profiles wasn’t an option. Two, each site had their own SAN, therefore the device IDs would be different for each environment.

Since the environment used all EMC VNX arrays, I decided to use a combination of Clint Kitson’s (blog / twitter) powershell scripts to gather CX/VNX data and PowerCLI to solve the problem.

Requirements:

• exec_naviseccli_cmd.ps1
• get_naviseccli.ps1
• The set-perennialReservations powershell script below
• Navisphere CLI (can be found on Powerlink)
• PowerCLI
• You must be using a CX or VNX branded storage array
• You must know the LUN numbers of the LUNs you want to set (as shown in Unisphere)

The Script

#get user-defined variables 
$ip = read-host "Enter in the IP for SPA of the VNX"
$user = read-host "Enter in the username"
$password = read-host "Enter in the password"
$vcenter = read-host "Enter in the FQDN of the vCetner server"
$cluster = read-host "Enter in the Cluster name"

#set LUN numbers from EMC VNX or CX
$lun1 = "Logical Unit Number 200"
$lun2 = "Logical Unit Number 201"
$lun3 = "Logical Unit Number 202"
  

#get SAN data
$sandata = .\get_naviseccli.ps1 -spip $ip -username $user -password $password -scope 0 -get all

#get UID of SAN LUNs
$lun01 = $sandata."All Logical Units Information".$lun1.uid
$lun02 = $sandata."All Logical Units Information".$lun2.uid
$lun03 = $sandata."All Logical Units Information".$lun3.uid

#remove colon from UIDs and prepend "naa." to the UIDs
$lun01 = "naa."+$lun01.Replace(":","");$lun01 = $lun01.ToLower()
$lun02 = "naa."+$lun02.Replace(":","");$lun02 = $lun02.ToLower()
$lun03 = "naa."+$lun03.Replace(":","");$lun03 = $lun03.ToLower()

#connect to vcenter and get vmhosts
connect-viserver $vcenter
$vmhosts = get-cluster $cluster | get-vmhost

#set perennial reservations
Foreach ($vmhost in $vmhosts) {
$myesxcli = get-esxcli -VMhost $vmhost
$myesxcli.storage.core.device.setconfig($false, $lun01, $true)
$myesxcli.storage.core.device.setconfig($false, $lun02, $true)
$myesxcli.storage.core.device.setconfig($false, $lun03, $true)
}

At the beginning of the script you’ll see three variables named $lun1, $lun2 and $lun3. Make sure you change the numeric values inside each string to correspond to the LUN numbers you want to set. I’m also running this against hosts in a cluster, but you can remove that if you need to do it across all hosts in vCenter. Again, you’ll notice this script is based on setting perennial reservations on 3 LUNs. You can increase or decrease this as you wish, just copy or remove the relevant lines. If you need assistance modifying the script, or running it, please leave a comment below and I’ll help you where I can. Thanks again to Clint Kitson for the scripts he wrote to gather CX/VNX information. It has proven useful to me time and time again.

The tests below all run the same workload, and instead of three tests, five tests were performed. If you’d like to know more about the equipment or test setup/script, see my original post: Jumbo Frames and Multi-NIC vMotion Performance over 10Gbe

• The only difference in setup is all VMs are configured with 16GB of RAM instead of 24GB, as in the previous tests

The Tests

For testing, I used the same thing as Chris, prime95. All VMs were running at 100% CPU and using 13GB of memory; 12288MB set in prime95 and the rest was being used by the OS. The following 5 tests were performed utilizing a power CLI script (see previous post here for script):

• Test 1  — vMotion of 1 powered-on VM loaded with prime95
• Test 2  — vMotion of 2 powered-on VMs loaded with prime95
• Test 3  — vMotion of 4 powered-on VMs loaded with prime95
• Test 4  — vMotion of 6 powered-on VMs loaded with prime95
• Test 5  — vMotion of 8 powered-on VMs loaded with prime95

Each test was performed using two configurations with regards to the MTU:

• Configuration 1:
  • vmk interfaces:                 1500
  • 1000v uplink port profile    9000
• Configuration 2:
  • vmk interfaces:                 9000
  • 1000v uplink port profile    9000

The Results

Once all tests were complete I removed the highest and lowest times from each, and then came up with the average. The results are somewhat similar to previous testing:

Test 1 — 1 VM, 13GB mem workload

• 1500 MTU: 18.75
• 9000 MTU: 18.10

In test 1 jumbo frames are barely faster, not even noticeable; 3.46% faster

Test 2 — 2 VMs, 13GB mem workload

• 1500 MTU: 34.50
• 9000 MTU: 21.95

In test 2 jumbo frames are NOTICEABLY faster; 36.37% faster

Test 3 — 4 VMs, 13GB mem workload

• 1500 MTU: 56.94
• 9000 MTU: 47.19

In test 3 jumbo frames are faster, 50% less than test 2, but still faster; 17.12% faster

Test 4 — 6 VMs, 13GB mem workload

• 1500 MTU: 109.43
• 9000 MTU: 107.69

In test 4, just as in test 1 jumbo frames are barely faster, not even noticeable; 1.59% faster

Test 5 — 8 VMs, 13GB mem workload

• 1500 MTU: 203.38
• 9000 MTU: 207.84

In test 5 jumbo frames are actually slower, but just barely; 2.14% slower

Conclusion

So, what do you think? Are jumbo frames worth it? If you combine all 1500 MTU times and all 9000 MTU times and calculate a percentage, jumbo frames are ONLY 4.72% faster than the traditional MTU size. Scott Lowe made a comment in one of his recent technology short takes (based on my previous testing), and I’m paraphrasing “be sure to consider complexity of jumbo frames into your design and weigh pros/cons.” While I agree with Scott on this, I will say, if you’re already running jumbo frames as part of the design, why not use the 9000 MTU for vMotion? In this case I think it’s a no brainer, change the MTU size to 9000. Overall there is an increase in performance depending on the workload, and, based on my testing, you can see upwards of a 37% increase in performance.

The Equipment

• Two Dell R820s with 2, dual port Broadcom 57810MF CNAs
• Two Cisco Nexus 2248 Fabric Extenders
• Two Cisco Nexus 5596 UP switches
• Cisco Nexus 1000v switch

The Setup

Setup is pretty standard, two CNA ports, one from each CNA going to FEXA, which connects to Nexus5KA and the other two CNA ports going to FEXB, which connected to Nexus5KB. The Cisco Nexus 1000v is deployed as an HA pair. To see how the 1000v is configured for multi-NIC vMotion check out my recent blog post, Configuring Multi-NIC vMotion with Cisco 1000v.

• The 1000v is configured with two uplink port profiles, one with the MTU set to 1500 and another with the MTU set to 9000
• The Nexus5K ports aren’t configured explicitly for a 9000 MTU, but dynamically adjust as jumbo frames are sent/received via policy map/QOS
• There are 4 vmkernel interfaces on each host for vMotion with their MTUs set to either 1500 or 9000 depending on the test being performed
• Both R820s were configured identically
• Each VM used was configured for 2 vCPUs and 24GB of RAM, running Windows Server 2008 R2 SP1
• All VMs were ALWAYS on the same host at the start of each test.

The Tests

For testing, I used the same thing as Chris, prime95. The following 3 tests were performed utilizing a power CLI script (more on the script below):

• Test1
  • vMotion of 1 powered-on VM loaded with prime95 and running a 20GB memory workload
• Test 2
  • vMotion of 4 powered-on VMs loaded with prime95 and each running a 20GB memory workload
• Test 3
  • vMotion of 8 powered-on VMs loaded with prime95 and each running a 10GB memory workload

Each test was performed with 3 separate configurations with regards to the MTU:

• Configuration 1:
  • vmk interfaces:                 1500
  • 1000v uplink port profile    1500
• Configuration 2:
  • vmk interfaces:                 1500
  • 1000v uplink port profile    9000
• Configuration 3:
  • vmk interfaces:                 9000
  • 1000v uplink port profile    9000

The Script

Below is a copy of the script I used. For each test, lines were removed or added depending on the number of VMs I was testing for that particular configuration. The script iterates through the vMotion process 10 times. During tests that included multiple VMs, all vMotions were run asynchronously EXCEPT for the last one. Once the last vMotion was finished there is a wait period of 45 seconds to ensure all vMotions were complete. Then the time is measured between all vMotions for that specific iteration and the one that took the longest is the the recorded on screen and appended to a text file. Not very scientific (even though using async, all vMotions aren’t started at EXACTLY the same time), but since all tests are being performed in this manner, I figured it would do. At the end of each iteration the script pauses for a 30 second period. The script below is what i used for the test3 (the 8VM test).

For ($num=1; $num -le 10; $num++) { 
    $timearray = @()
    $date = get-date
    $vm = Get-VM -Name "Prime95" 
    $vm2 = Get-VM -Name "Prime95-02" 
    $vm3 = Get-VM -Name "Prime95-03" 
    $vm4 = Get-VM -Name "Prime95-04" 
    $vm5 = Get-VM -Name "Prime95-05" 
    $vm6 = Get-VM -Name "Prime95-06" 
    $vm7 = Get-VM -Name "Prime95-07" 
    $vm8 = Get-VM -Name "Prime95-08"
    If ($vm.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm -Destination $target -RunAsync -Confirm:$false
    If ($vm2.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm2 -Destination $target -RunAsync -Confirm:$false    
    If ($vm3.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm3 -Destination $target -RunAsync -Confirm:$false    
    If ($vm4.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm4 -Destination $target -RunAsync -Confirm:$false | out-null 
    If ($vm5.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm5 -Destination $target -RunAsync -Confirm:$false
    If ($vm6.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm6 -Destination $target -RunAsync -Confirm:$false    
    If ($vm7.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm7 -Destination $target -RunAsync -Confirm:$false    
    If ($vm8.Host -match "host01") {$target = (Get-VMHost -Name "host02")}  
    Else {$target = (Get-VMHost -Name "host01")}  
    Write-Host "Starting migration now..." 
    Move-VM -VM $vm8 -Destination $target -Confirm:$false | out-null
    start-sleep -seconds 45  
    $tasks = get-task | where { $_.name -eq "RelocateVM_Task" -and $_.starttime -gt $date }
    foreach ($task in $tasks) { $timearray+= new-timespan -start $task.starttime -end $task.finishtime | % { $_.totalseconds } }
    $totaltime = $timearray | measure-object -maximum | % { $_.maximum }
    $time = "Task completed in $totaltime seconds" 
    Write-Host $time
    $time | out-file -append vmotion_time_1500mtu_8vms.txt
    start-sleep -seconds 30
}

The Results

Once all tests were complete I removed the highest and lowest times from each, and then came up with the average. The results are interesting to say the least. All results are in seconds:

Test 1 — 1 VM, 20GB mem workload

• Configuration 1 (1500/1500): 30.24
• Configuration 2 (1500/9000): 28.81
• Configuration 3 (9000/9000): 26.31

In test 1 jumbo frames had better performance; 12.99% faster

Test 2 — 4 VMs, 20GB mem workload

• Configuration 1 (1500/1500): 89.69
• Configuration 2 (1500/9000): 92.54
• Configuration 3 (9000/9000): 75.71

In test 2 the results are much more concise, with jumbo frames blowing the competition our of water; 15.58% faster

Test 3 — 8 VMs, 10GB mem workload

• Configuration 1 (1500/1500): 203.82
• Configuration 2 (1500/9000): 204.71
• Configuration 3 (9000/9000): 252.14

Test 3, overwhelmingly has jumbo frames as the loser; 23.70% slower

Conclusions

First I want to start by saying, what in the world happened in test 3? After looking at the results from the first two tests I was shocked to see the results that test 3 produced. Back to that in a moment…

• The difference of the MTU setting on the Cisco Nexus 1000v uplink port profile is negligible and should be set to 9000
• Configuring jumbo frames for use with vMotion in a 10Gbe multi-NIC vMotion environment is workload dependent (whether it’s used or not)
• Jumbo frames are not the clear winner by any stretch of imagination.
• More testing needs to be performed to determine the pros and cons of utilizing jumbo frames for vMotion in a multi-NIC vMotion environment

The second conclusion in the list is loose and only based on multi-NIC vMotion with 4 NICs.

I’m really confused on why moving 4 VMs with a memory workload aggregate of 80GB is so much different than moving 8 VMs with the same memory workload aggregate. Possibly the way the algorithm balances packets across the physical NICs? Because moving 8 VMs generate more jumbo frames (which means more overhead) and there’s a tipping point on # of jumbo frames from a cost/benefit scale?

More testing and a better understanding of the algorithm may offer better conclusions then what I’ve come up with. Please feel free to share your thoughts in the comments section.

Here are the generic steps:

1. Create a vmkernel port for each physical NIC being used for multi-NIC vMotion per host
2. Create a vethernet port profile on the 1000v for each corresponding vmkernel interface you created in step 1 

   ### Example Port Profile Configuration ###
   
   port-profile type vethernet MULTI-NIC_VMOTION_1
   vmware port-group VMotion_1_VLAN1000
   switchport mode access
   switchport access vlan 1000
   pinning id 0
   no shutdown
   state enabled
   
   

3. Connect each vmkernel port to its matching port profile
4. Set channel group mode mac-pinning to auto or relative (more on this later)
5. Create class map on 1000v for vMotion traffic — this is important so you don’t saturate your uplinks with vMotion traffic

In my setup I’m using Dell R820s with two, dual port Broadcom 57810MF 10Gbe CNAs. This allowed me to do a 4 NIC, multi-NIC vMotion setup. Here is the logical design:

In this example, step 4 from the generic steps above, the ethernet port profile had the following command on it:

• channel-group auto mode on mac-pinning relative

The mac-pinning portion of the command can be set to auto or relative. What’s the difference?

• mac-pinning auto – when the channel sub-groups are created using mac-pinning auto, they are numbered relative to the vmnic number. In the example above, the channel sub-groups will be subgroups 4-7. This might work fine, however if the vmnic numbering is different on one of more of your hosts, it could be confusing.
• mac-pinning relative – when the channel groups are created using mac-pinning relative, they are numbered relative to the number of uplinks starting with 0. In the example above, the channel sub-groups will be subgroups 0-3. This command can be useful if you are connecting multiple hosts to the 1000v with different number of uplinks and using the same ethernet port profile

After reading Chris Wahl’s blog post on whether or not you get a performance increase by enabling jumbo frames on the vmkernel interfaces being used for multi-NIC vMotion, I decided to do some testing of my own. I ran the same test as he did and a few of my own using his basic test methodology, but with 10Gbe and the Cisco 1000v. I’ll be posting those results soon.

The book had a nice flow to it, starting from where the program came from and the different tracks that exist today, then going to the defense process and finally example scenarios. Here a list of the chapters along with my opinions of each:

• Chapter 1: Certification Overview
  • Nothing too fancy here. Chapter covers each of the VCDX tracks (there are 3) along with an overview of the VCDX process as a whole
• Chapter 2: Preparation
  • This chapter starts with the application and discusses the VMware side of things (application processing, review, etc…). The chapter also discusses how the defense panel/room is layed out and how to practice for the defense via study groups and mock defenses
• Chapter 3: The Design
  • Here the design blueprint is covered. What the panelists might be looking for in your design is also discussed; I find this information to be very useful as I’m beginning to prepare my design
• Chapter 4: Defense Overview
  • You’ll get an inside look at the panelists perspective in this chapter. You’ll also read about how the defense is timed and what you can expect when you arrive
• Chapter 5: Design Defense
  • Here you will read about how the design defense portion of the defense will go. You will find out how your presentation should be structured along with what content you should, and should NOT present (SPOILER ALERT: DON’T BRING GIFTS FOR THE PANELISTS!)
• Chapter 6: The Design Scenario
  • This was my favorite chapter, and also my least favorite. I do not do designs everyday and part of the defense is a 30 minute design scenario where you must take some loose requirements, coupled with panelists (the customer) feedback and create a base design. It’s my favorite because it truly gives me some focus points, it’s the worst because it makes me question whether I’m ready for this or not…and I definitely don’t like that
• Chapter 7: The Troubleshooting Scenario
  • This chapter is much like the previous two, but covers how the troubleshooting portion of the defense is laid out, and what the panelists are looking for

The book has tips from current VCDXs scattered through the chapters; a few useful tips. You’ll see a lot of scenarios for the the Design Scenario portion of the defense and the Troubleshooting Scenario portion of the defense. This may be useful and some of them you can see being discussed live on a past vBrownBag podcast on the VCDX boot camp in which John and Mostafa both were presenting. The 3-part series is posted on Vimeo (listed below) and also available on iTunes.

vBrownBag VCDX Boot Camp

• vBrownBag VCDX Boot Camp Part 1 (couldn’t find on vimeo, check iTunes…come on Damien!!!!)
• vBrownBag VCDX Boot Camp Part 2
• vBrownBag VCDX Boot Camp Part 3

You probably want to take this review with a grain of salt since it is merely a one-sided perspective. I have yet to submit an application or do a panel defense, which means I can’t honestly tell you that this book will help you pass the defense. However, I believe it will. The book gave me insight on how the entire VCDX defense process is structured and a glimpse into what the panelists might be looking for. The book also exposed one of my weaknesses, and I will surely do better in that area (design scenario) then I would have without having this information. I’m will definitely be picking up a copy of this once it is officially released to check out any new content added. I would love to hear some reviews of this book from current VCDXs.

This is an atypical post for me as normally my posts are purely technical. There are days where I start working in my lab and I’ll ask myself the question “why are you doing this?”…I’ll ask “why are you spending all this time in your lab constantly trying to learn new products"?”. Before I get too deep into this let me say this first; I love technology. I love the feeling I get when I ‘m able to solve a problem by using technology, whether it be a technical problem or a business problem. Although I must say, most of the times it is a technical problem, which leads me to the next part of this charade…

I clearly don’t have a solid understanding of solving real world business problems by utilizing technology. Sure, I understand the business benefits of virtualization, converged platforms, and blah blah blah, but a lot of that understanding is as a tactical minded person. I am so used to somebody asking me to fix a problem, or generate a technical solution based on new or existing requirements, but it’s ALL tactical. Where’s the strategy?

<tangent>Now, I do work, and have always worked for a company that holds a federal contract with the U.S. government, particularly within the DoD. While at these jobs, the business case and business problems have almost never been about dollars, and at times, efficiency, Some of you may question that, and at times I have too, but when I really think about these particular business cases, it makes sense. They aren’t in the business of making money, they are in the business of national security and fighting wars, and a major part of that involves technology. </tangent>

Without moving into the real business side of technology (meaning not the DoD), how can I change my mindset and incorporate strategic thinking with my tactical mind? Don’t get me wrong, there is definitely strategic planning that goes on in the particular sector I work in, but it isn’t at my level, or would I want it to be. The DoD ( and the federal government I imagine) has SO many IT systems and half of them never get deployed, or they are don’t work with each other, or a million other things I don’t care to get into. The point is, it’s a HUGE mess. I would LOVE to develop a framework that most/all fed agencies could use to build their information systems on which is scalable, integrated, interconnected, on-demand, and user friendly. It doesn’t seem attainable though. Even if I had that sort of mindset, I’m still not sure I would know where to start.

Sorry, I know, tangents everywhere. I think it’s time I start to try and understand the business side of technology, and how to solve real problems that can help businesses leverage technology to the fullest. My biggest problem is, I don’t know where to start. Is it learning a programming language like Ruby so that I can try and create something? Do I go for my MBA in hopes that it translates to how companies leverage IT today and in the future (I don’t think it will)? Do I go for my VCDX-DV certification with the hopes that the architecting the infrastructure layer will still be a much needed skill in 3-5 years, or do I change my path to VCDX cloud certification? Do I abandoned my quest for deep seeded knowledge in VMware’s core products and pursue open stack? Now I’m just rambling…I’ve went from tactical, to wanting to think strategically, to specific technologies and on and on and on.

This is what dominates my brain some nights…and I’m getting frustrated with it. While I know how to solve this problem for myself tactically, I have no clue how to solve it strategically….and now we’ve come full circle. Please go back to your sane lives,

Jason Langer (he blogs at www.virtuallanger.com) and I spent a bit of time last year and this year covering VCAP5-DCA blueprint on our blogs. Due to the sheer amount of information Jason and I split up the objectives so that we could complete it faster and we both used it to sit, and successfully pass our VCAP5-DCA exam. After dragging my feet for a few months I’ve finally got it all together into one document which you can download below. A special thanks goes out to Conrad Ramos who gave an assist by writing up Objective 8.1 – Execute VMware Cmdlets and Customize Scripts using PowerCLI.

I hope everyone gets as much out of it as Jason and I did. I also REALLY hope that the VCAP6 series doesn’t start for a while as I’m not ready to do this all over again. Enjoy, and happy studying!

The Unofficial Official VCAP5-DCA Study Guide