Recently I’ve been busy with the upgrade of my vSphere 6.0U2 environment to 6.5U1. During and after the upgrade some of the VMs started getting rebooted by VMware HA. VMs were unresponsive at that stage and vSphere logged “OS crash” due to VMware Tools heartbeat loss. VMware HA VM monitoring naturally tried to reset the machines, however randomly for some it succeeded, for some it didn’t. All of the crashed VMs have one configuration in common: 3D Graphics enabled. At first I considered the reason was VMware Tools, but then I thought: “Is it possible to lose backward compatibility this much between vSphere versions?” Actually the running Tools versions were not updated since vSphere 5.5 and somehow it didn’t complain until this upgrade.
There were two different logs (screenshots below) on the VMs that HA failed to reset, first one recurring very frequently, almost every 20 seconds:
vSphere HA attempted to reset the virtual machine because of a heartbeat failure from VMware Tools or a guest application, depending on how vSphere HA was configured. However, the reset operation failed. The most likely reason for the reset failure is that the virtual machine was running another task at the time the reset was initiated. Action: Check to see whether the virtual machine requires attention and reset it manually if necessary.
This event is logged when vSphere HA did not reset a VM affected by an inaccessible datastore. It will attempt to reset the VM after storage failure is cleared. The VM is affected by an inaccessible datastore due to storage connectivity loss. Resetting such a VM might cause the VM to be powered off and not restarted by vSphere HA.
I couldn’t see any ongoing activity or an inaccessible datastore. These logs seemed misdirecting so I started updating VMware Tools and virtual HW as recommended. As suspected, it solved the problem: VMs stopped crashing and found some peace.
When you don’t have the luxury to restart the systems, you may choose to “postpone” the VMware Tools and virtual HW upgrade thinking that it is not compulsory and has an extended support. Here is a good old VMware blog entry about the subject: https://blogs.vmware.com/vsphere/2013/02/is-a-vmware-tools-upgrade-required-when-upgrading-vsphere.html. Although the VMware Product Interoperability Matrix shows Tools from 5.5 still works on ESXi 6.5U1, my case proved that there may be exceptions on rare configurations. The system needs to be kept as up-to-date as possible, as a whole.
Let’s finish with VMware’s statement:
Although upgrading VMware Tools is optional, it is still highly recommended. The extended support is meant to facilitate upgrades and should not be seen as an excuse to avoid upgrading VMware Tools.