VMware HA warnings on VMs with 3D Graphics and VMware Tools upgrades

Recently I’ve been busy with the upgrade of my vSphere 6.0U2 environment to 6.5U1. During and after the upgrade some of the VMs started getting rebooted by VMware HA. VMs were unresponsive at that stage and vSphere logged “OS crash” due to VMware Tools heartbeat loss. VMware HA VM monitoring naturally tried to reset the machines, however randomly for some it succeeded, for some it didn’t. All of the crashed VMs have one configuration in common: 3D Graphics enabled. At first I considered the reason was VMware Tools, but then I thought: “Is it possible to lose backward compatibility this much between vSphere versions?” Actually the running Tools versions were not updated since vSphere 5.5 and somehow it didn’t complain until this upgrade.

There were two different logs (screenshots below) on the VMs that HA failed to reset, first one recurring very frequently, almost every 20 seconds:

vSphere HA attempted to reset the virtual machine because of a heartbeat failure from VMware Tools or a guest application, depending on how vSphere HA was configured. However, the reset operation failed. The most likely reason for the reset failure is that the virtual machine was running another task at the time the reset was initiated. Action: Check to see whether the virtual machine requires attention and reset it manually if necessary.

&

This event is logged when vSphere HA did not reset a VM affected by an inaccessible datastore. It will attempt to reset the VM after storage failure is cleared. The VM is affected by an inaccessible datastore due to storage connectivity loss. Resetting such a VM might cause the VM to be powered off and not restarted by vSphere HA.

HA Error Type 2HA Error Type 1

I couldn’t see any ongoing activity or an inaccessible datastore. These logs seemed misdirecting so I started updating VMware Tools and virtual HW as recommended. As suspected, it solved the problem: VMs stopped crashing and found some peace.

When you don’t have the luxury to restart the systems, you may choose to “postpone” the VMware Tools and virtual HW upgrade thinking that it is not compulsory and has an extended support. Here is a good old VMware blog entry about the subject: https://blogs.vmware.com/vsphere/2013/02/is-a-vmware-tools-upgrade-required-when-upgrading-vsphere.html. Although the VMware Product Interoperability Matrix shows Tools from 5.5 still works on ESXi 6.5U1, my case proved that there may be exceptions on rare configurations. The system needs to be kept as up-to-date as possible, as a whole.

Let’s finish with VMware’s statement:

Although upgrading VMware Tools is optional, it is still highly recommended.   The extended support is meant to facilitate upgrades and should not be seen as an excuse to avoid upgrading VMware Tools.

VMware vCenter Server 6 installation low memory workaround

I was installing VMware vCenter Server 6 Update 2 with Embedded Platform Services Controller on a Windows lab server with 8GB RAM. I got the following error:

VMware vCenter Server detected 8160MB of memory. 8176MB of memory is required for the selected deployment type.

You can easily skip this error with initiating the following in command line:

VMware-vCenter-Server.exe SKIP_HARDWARE_CHECKS=1

HP Smart Array B140i SATA RAID controller OS boot problem

Recently I installed VMware ESXi on a HP DL160 Gen9 server and couldn’t boot afterwards. After a short investigation I found out that the default configuration of the controller prevents booting. The controller should be configured and enabled to support SATA AHCI mode. Actually it is necessary to configure it for all servers carrying HP Smart Array B110i, B120i, B140i and B320i RAID controllers.

To do that:

1) In BIOS/UEFI, go into “System Options”
2) Go to SATA Controller Options -> Embedded SATA Configuration
3) Select “Enable SATA AHCI Support”

For a similar case in RedHat: http://access.redhat.com/articles/118133

Installing VMware ESXi on IBM Lenovo x240 M5 with SD Media Adapter and SD Cards

Last week I installed VMware ESXi 6.0U1B on some (formerly IBM) Lenovo x240 M5 Compute Nodes in Flex System Chassis. The case was special because the nodes contain only SD cards and SD Media Adapters instead of SAS hard drives and standard RAID controllers. There are no interface other than IMM CLI of the node to configure this SD Media Adapter to build a RAID array. You should first connect to IMM of the node with SSH. Then you can get the initial information about your SD RAID and cards with these commands:

system> sdraid
SD Media Adapter for System x
Hardware Revision = 4.3
Firmware Version = 1.3.2.171
Serial Number = 58WXYZ
FRU Number = 00JY0XX
Mode = Operational
SDCard1
Status = Healthy
Capacity = 30436 MBytes
FRU Number = 00MLXYZ
SDCard2
Status = Healthy
Capacity = 30436 MBytes
FRU Number = 00MLXYZ

system> sdraid -driveList
SDCard 1
Index LUN Name Type Size(MB) Owner Access Removable
SDCard 2
Index LUN Name Type Size(MB) Owner Access Removable
ok

system> sdraid -getFreeSpaceInfo

I have two 32 GB SD cards which I intend to build a mirror RAID-1 array with. I switched to config mode, created the drive and switched back to operational mode with these commands:

system> sdraid -setMode Configuration -now
system> sdraid -create -driveName ESXiLocalDS -target mirror -sizeMB 30436 -removable 1 -owner system -systemReadOnly 0 -LUN 0
system> sdraid -setMode Operational -now

After these BIOS should be configured to boot from this drive:

Boot Manager > Add Boot Option > Generic Boot Option > Embedded Hypervisor
Boot Manager > Change Boot Order > move "Embedded Hypervisor" to the top
System Settings > Devices and I/O Ports > PCI 64-Bit Resource > Disabled
System Settings > Devices and I/O Ports > MM Config Base > 3 GB

When you save your changes and exit BIOS, you will be able to start ESXi installation aware of the SD card drive.

For reference: http://public.dhe.ibm.com/systems/support/system_x_pdf/dw2abms3.pdf

Converting Thin Provisioned Disks to Lazy Zeroed Thick in VMware vSphere

Lately I’ve dealt with a problem about thin-provisioned disks in my vSphere environment. Although there is the feature in VMware, I choose to leave thin provisioning to back-end, to storage administrators. The usage may go beyond your control in short time and you may not be able to receive disk resource during that time. It is just about to be on the safe side: “I do not use thin provisioning in VMware virtual disks.”

Unfortunately before me one datastore was over-provisioned (over 200%!!) and left unattended. It was only a short time after taking over and clients decided to use the resource which was promised to them. We received a call about a VM that went unresponsive and as we dig we understood the severity of the situation.

In short, we received new LUNs and moved VMs to new datastores. I took an action to convert all the thin provisioned disks to thick. It is officially done with right clicking the thin provisioned vmdk file in the datastore browser and choosing “Inflate”:

inflate vmdk

However this triggers a conversion to eager zeroed thick provisioned disk. It takes more time and creates unnecessary I/O on storage side. To convert the provisioning of disks to lazy zeroed thick I initiated Storage vMotion with the appropriate virtual disk format selection:

Storage vMotion

After the successful migration the virtual disks of VM is lazy zeroed thick provisioned. GUI may still show “Used Space 0.00B” about VM. A refresh on the Datastore Summary / Capacity Usage page should correct the glitch.

To automate the process you can use the following command in PowerCLI:

Move-VM -VM thinvm -Datastore differentdatastore -DiskStorageFormat Thick

My website uses cookies. Click for more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close