Recently I’ve been upgrading a vSphere 5.5 environment to vSphere 6.5U1. The vCenter upgrade process is pretty bulletproof by now: the installer is pretty much completely automated. I did run into some issues during the ESXi upgrade, one of which is the fact there were some conflicting VIBs present in the old installations. This would mean the ESXi 6.5U1 upgrade would not start. Time to start hacking in the CLI!

What’s a VIB?

VIB stands for vSphere Installation Bundle, and it’s conceptually similar to a collection of files grouped together in a ZIP file or tarball. VIBs make it easy to distribute drivers or software to/between ESXi hosts, and can be installed or removed with the “esxcli software” commands. VIBs are used for a number of reasons, for example to update drivers or install vendor software. The customer uses HPE customized ESXi installations for monitoring purposes, so some of the included VIBs are specific driver versions for BL460c blades and monitoring tool integration.

Conflicting VIBs during upgrade

While scanning the hosts against the HPE customized ESXi 6.5U1 image, I received a “Incompatible” result with a pretty self-explanatory error message:

It tells me that the current installation contains VIBs that conflict with the VIBs in the ESXi6.5U1 installation image. In this case it’s a bunch of NIC and HBA drivers. Most likely they are left-overs from previous upgrades and now deprecated. Hence I need to remove them manually.

Nice to know: if you get a bunch of conflicting VIB messages and they are about monitoring tools, double check you’ve uploaded the correct ESXi installation image to Update Manager. You might be trying to install a default, non-OEM modified VMware ESXi image over a HPE/Cisco customized ESXi image.

Verify in-use drivers and VIBs

Place your ESXi host in maintenance mode, enable SSH access and connect to the CLI. First of all we’re going to verify which drivers are in use, using the following two commands:

To verify NIC drivers, run: esxcli network nic list
To verify HBA drivers, run: esxcfg-scsidevs -a

This will result in output similar to the following screenshot:

Or in text:

~ # esxcli network nic list
Name PCI Device Driver Link Speed Duplex MAC Address MTU Description
------ ------------- ------ ---- ----- ------ ----------------- ---- -----------------------------------------------------------
vmnic0 0000:004:00.0 elxnet Up 10000 Full 00:17:a4:77:00:7c 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic1 0000:004:00.1 elxnet Up 10000 Full 00:17:a4:77:00:7e 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic2 0000:004:00.2 elxnet Up 10000 Full 00:17:a4:77:00:80 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic3 0000:004:00.3 elxnet Up 10000 Full 00:17:a4:77:00:82 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic4 0000:004:00.4 elxnet Up 10000 Full 00:17:a4:77:00:84 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic5 0000:004:00.5 elxnet Up 10000 Full 00:17:a4:77:00:86 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic6 0000:004:00.6 elxnet Down 0 Half 6c:3b:e5:b7:a4:43 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter
vmnic7 0000:004:00.7 elxnet Down 0 Half 6c:3b:e5:b7:a4:47 1500 Emulex Corporation HP FlexFabric 10Gb 2-port 554FLB Adapter

~ # esxcfg-scsidevs -a
vmhba0 hpsa link-n/a sas.5001438025019210 (0:3:0.0) Hewlett-Packard Company Smart Array P220i
vmhba1 lpfc link-up fc.20000090fa3db5ce:10000090fa3db5ce (0:5:0.0) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba2 lpfc link-up fc.20000090fa3db5cf:10000090fa3db5cf (0:5:0.1) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba32 usb-storage link-n/a usb.vmhba32 () USB

As you can see, the elxnet, lpfc and usb-storage drivers are in use. So we cannot remove any VIBs that touch these drivers.

Locate conflicting VIBs names

The names of the conflicting VIBs are the first hint to the driver names; I’ve marked the interesting part in red:

qlogic_bootbank_scsi-qla2xxx_934.5.6.0-1OEM.500.0.0.472560
Emulex_bootbank_scsi-lpfc820_8.2.4.141.55-1OEM.500.0.0.472560
Mellanox_bootbank_net-mst_2.0.0.0-1OEM.550.0.0.472560

Just to be sure, we’ll look up the conflicting VIBs with esxcli software vib list. This command spews a lot of output, so it makes sense to pipe it through grep and look for the specific driver brands. My conflicting VIBs are qlogic, Emulex and Mellanox based, so I filter accordingly:

Using the version information, identify the driver names that need to be removed. In this case it’s indeed scsi-qla2xxx, scsi-lpfc820 and net-mst (as the conflicting VIB names already gave away).

You now know which drivers/VIBs are in use, and which drivers/VIBs need to be removed. If there’s no overlap here, you’re ready for the next step. Notice for example that there’s another lpfc driver (running a newer 10.0.x version); that’s the one that’s in use by the HBA’s and shouldn’t be touched.

Removing the conflicting VIBs

Removing the conflicting VIBs is pretty straightforward:

esxcli software vib remove -n net-mst
esxcli software vib remove -n scsi-qla2xxx
esxcli software vib remove -n scsi-lpfc820

The output looks like this:

Finally, reboot the server and scan for updates with Update Manager. The Compliance Status should change from Incompatible (which is bad) to Non-Compliant (which just means you need to update).

There you go: no more incompatible VIBs and you’re ready to upgrade!

My thoughts on the upgrade to vSphere 6.5U1

The new vCenter 6.5 installation procedure is a very refreshing experience for me, after not doing any vSphere upgrades since 5.5. The installer asks you for a few fields of information and then does the majority of the installation/upgrade tasks fully automated. Definitely beats the “click 400 buttons” installations from vSphere 3 to 5.

For the vSphere Desktop Client lovers (myself included): there’s no vSphere C# Desktop client anymore! Rest assured though: vSphere 6.5U1 contains a pretty well stocked HTML5 client, which you can access over https://<vcenterserver>/ui/. It looks remarkably similar to the old desktop client, so you can probably find most options straight away. For example: opening SSH access is a MASSIVE improvement over both the old desktop client and the web client. Select ESXi host, click Configure, Services, right-click SSH, click Start. Refreshingly clean experience!

Not 100% of the Web Client functionality is in there yet: for example for licensing, Active Directory integration and Update Manager you still need to use the web client. VMware coders are working on closing the gap though, and there’s an active Fling for those people that want the newest HTML5 features first. As always: run experimental/beta code at your own risk. In general, I do the majority of tasks in the HTML5 client and just have the web client open in a different tab for the other tasks. Hopefully we can put Flash to bed soon as many storage and backup vendors are also switching to HTML5 user interfaces, but that’s for another post…

Upgrading to ESXi 6.5U1: Conflicting VIBs