Over the past month or so I’ve been upgrading instances of vSphere 4.1 to vSphere 5.0. I am just about complete (and awesomely I get to do it all over again for vSphere 5.0 Update 1) and I wanted to share some things about my experience(s). My apologies in advance for not having any screenshots or exact error codes.
1. First and foremost you want to ensure that all of your virtual machines are vMotion ready. Josh Atwell (@josh_atwell) wrote a nice function that will check pretty much anything that will prevent a virtual machine from migrating, check that out here. I did run into a few issues where virtual machines were sitting on local storage instead of shared storage.
Bottom Line: Ensure your virtual machines are vMotion capable
2. In order to upgrade your hosts you need to be in maintenance mode. If you are going to leave your virtual machines powered on during the upgrade, which I assume you will, the virtual machines will need to be vMotioned to another vmhost before the host will go into maintenance mode. If your hosts are part of a cluster, and want to do this to be done for you, ensure DRS is enabled (why the hec wouldn’t you have this enabled?!?) and set to Fully Automated. If you notice the maintenance mode task stuck at 2% either DRS is off, not set to fully automated or a VM failed to vMotion properly
Bottom Line: Turn on DRS and set to Fully Automated if your hosts are in a DRS enabled cluster
3. If you are performing an upgrade you may get an error when update manager is trying to remediate your host, and it will most likely be a generic error. If you dig into the update manager logs, located on your vcenter server at C:\ProgramData\VMware\VMware Update Manager Logs (main log is vmware-vum-server-log4cpp.log) you could see a warning or error that, in some form or fashion, says “out of space”. To truly check your unreserved space run esxtop or resxtop on the host that received the error, press “M” for the memory page and look at VMKMEM at the top and over to the right you will see ursvd. Make sure that number is >310. As you can see below there is more than enough unreserved memory.
Bottom Line: Make sure you have at least >310MB of unreserved memory on your host
4. The very first issue I ran into was the upgrade script kept failing during remediation on all of my hosts. The error I saw in the update manger log was “Cannot execute upgrade script on host”; very informative. After a bit of research I discovered VMware KB 2006254. The gist of the problem is that if the VMkernel you are using for your management network has a label/name that is more than 20 characters then remediation will fail. I believe this is fixed in vSphere 5 Update 1, but I haven’t tested it.
Bottom Line: Ensure the name/label of your management network is less than 20 characters
5. Custom Roles: We have a few custom roles that we use, one of which is a full clone of the Administrator role. You might ask, “why make a clone of it if it’s an exact copy, why not just use the Administrator role?”—we have several other custom roles with particular naming standards, it only made sense for us to clone the administrator role and give it a name IAW (in accordance with) our naming standard. There are four new privileges that are added when you upgrade to vSphere 5; Datastore Cluster, ESX Agent Manager, Profile-driven storage and vService As you might have guessed, these new privileges are enabled on the built-in Administrator role, but not any roles that you have cloned (if this is your first time installing Update Manager be sure to check the VMware vSphere Update Manager privilege as well).
Bottom Line: As needed, update any roles that aren’t built-in with the new privileges
Here is a general guide that should get you through the upgrade process – vSphere 5 Upgrade.docx. If you have ESXi hosts external to your firewall or in a DMZ please see my post on Update Manager Firewall Considerations
All-in-all the upgrade process was smooth. The issues I did have I was able to diagnosis and resolve in a respectable amount of time. I get to now turnaround and do this process all over again with vSphere 5 update 1, but this time it should be a lot less work since some of the other components are now in place. Happy upgrading!