Meta

Kategoriler


Eskiden çok sevdiğim bir sitede vardı: Tamamen boş, fonksiyonu olmayan bir buton :) "stress tuşunda anlam arayan varsa, derhal siteyi terketsin..."


Olayı abartmamak lazım tabi:






View Ertürk İhsan Limon's profile on LinkedIn


Red Hat Certified Engineer


VCP5 Certified


VMware HA warnings on VMs with 3D Graphics and VMware Tools upgrades

Recently I’ve been busy with the upgrade of my vSphere 6.0U2 environment to 6.5U1. During and after the upgrade some of the VMs started getting rebooted by VMware HA. VMs were unresponsive at that stage and vSphere logged “OS crash” due to VMware Tools heartbeat loss. VMware HA VM monitoring naturally tried to reset the machines, however randomly for some it succeeded, for some it didn’t. All of the crashed VMs have one configuration in common: 3D Graphics enabled. At first I considered the reason was VMware Tools, but then I thought: “Is it possible to lose backward compatibility this much between vSphere versions?” Actually the running Tools versions were not updated since vSphere 5.5 and somehow it didn’t complain until this upgrade.

There were two different logs (screenshots below) on the VMs that HA failed to reset, first one recurring very frequently, almost every 20 seconds:

vSphere HA attempted to reset the virtual machine because of a heartbeat failure from VMware Tools or a guest application, depending on how vSphere HA was configured. However, the reset operation failed. The most likely reason for the reset failure is that the virtual machine was running another task at the time the reset was initiated. Action: Check to see whether the virtual machine requires attention and reset it manually if necessary.

&

This event is logged when vSphere HA did not reset a VM affected by an inaccessible datastore. It will attempt to reset the VM after storage failure is cleared. The VM is affected by an inaccessible datastore due to storage connectivity loss. Resetting such a VM might cause the VM to be powered off and not restarted by vSphere HA.

I couldn’t see any ongoing activity or an inaccessible datastore. These logs seemed misdirecting so I started updating VMware Tools and virtual HW as recommended. As suspected, it solved the problem: VMs stopped crashing and found some peace.

When you don’t have the luxury to restart the systems, you may choose to “postpone” the VMware Tools and virtual HW upgrade thinking that it is not compulsory and has an extended support. Here is a good old VMware blog entry about the subject: https://blogs.vmware.com/vsphere/2013/02/is-a-vmware-tools-upgrade-required-when-upgrading-vsphere.html. Although the VMware Product Interoperability Matrix shows Tools from 5.5 still works on ESXi 6.5U1, my case proved that there may be exceptions on rare configurations. The system needs to be kept as up-to-date as possible, as a whole.

Let’s finish with VMware’s statement:

Although upgrading VMware Tools is optional, it is still highly recommended.   The extended support is meant to facilitate upgrades and should not be seen as an excuse to avoid upgrading VMware Tools.

Installing VMware ESXi on IBM Lenovo x240 M5 with SD Media Adapter and SD Cards

Last week I installed VMware ESXi 6.0U1B on some (formerly IBM) Lenovo x240 M5 Compute Nodes in Flex System Chassis. The case was special because the nodes contain only SD cards and SD Media Adapters instead of SAS hard drives and standard RAID controllers. There are no interface other than IMM CLI of the node to configure this SD Media Adapter to build a RAID array. You should first connect to IMM of the node with SSH. Then you can get the initial information about your SD RAID and cards with these commands:

system> sdraid
SD Media Adapter for System x
Hardware Revision = 4.3
Firmware Version = 1.3.2.171
Serial Number = 58WXYZ
FRU Number = 00JY0XX
Mode = Operational
SDCard1
Status = Healthy
Capacity = 30436 MBytes
FRU Number = 00MLXYZ
SDCard2
Status = Healthy
Capacity = 30436 MBytes
FRU Number = 00MLXYZ

system> sdraid -driveList
SDCard 1
Index LUN Name Type Size(MB) Owner Access Removable
SDCard 2
Index LUN Name Type Size(MB) Owner Access Removable
ok

system> sdraid -getFreeSpaceInfo

I have two 32 GB SD cards which I intend to build a mirror RAID-1 array with. I switched to config mode, created the drive and switched back to operational mode with these commands:

system> sdraid -setMode Configuration -now
system> sdraid -create -driveName ESXiLocalDS -target mirror -sizeMB 30436 -removable 1 -owner system -systemReadOnly 0 -LUN 0
system> sdraid -setMode Operational -now

After these BIOS should be configured to boot from this drive:

Boot Manager > Add Boot Option > Generic Boot Option > Embedded Hypervisor
Boot Manager > Change Boot Order > move "Embedded Hypervisor" to the top
System Settings > Devices and I/O Ports > PCI 64-Bit Resource > Disabled
System Settings > Devices and I/O Ports > MM Config Base > 3 GB

When you save your changes and exit BIOS, you will be able to start ESXi installation aware of the SD card drive.

For reference: http://public.dhe.ibm.com/systems/support/system_x_pdf/dw2abms3.pdf

Converting Thin Provisioned Disks to Lazy Zeroed Thick in VMware vSphere

Lately I’ve dealt with a problem about thin-provisioned disks in my vSphere environment. Although there is the feature in VMware, I choose to leave thin provisioning to back-end, to storage administrators. The usage may go beyond your control in short time and you may not be able to receive disk resource during that time. It is just about to be on the safe side: “I do not use thin provisioning in VMware virtual disks.”

Unfortunately before me one datastore was over-provisioned (over 200%!!) and left unattended. It was only a short time after taking over and clients decided to use the resource which was promised to them. We received a call about a VM that went unresponsive and as we dig we understood the severity of the situation.

In short, we received new LUNs and moved VMs to new datastores. I took an action to convert all the thin provisioned disks to thick. It is officially done with right clicking the thin provisioned vmdk file in the datastore browser and choosing “Inflate”:

inflate vmdk

However this triggers a conversion to eager zeroed thick provisioned disk. It takes more time and creates unnecessary I/O on storage side. To convert the provisioning of disks to lazy zeroed thick I initiated Storage vMotion with the appropriate virtual disk format selection:

Storage vMotion

After the successful migration the virtual disks of VM is lazy zeroed thick provisioned. GUI may still show “Used Space 0.00B” about VM. A refresh on the Datastore Summary / Capacity Usage page should correct the glitch.

To automate the process you can use the following command in PowerCLI:

Move-VM -VM thinvm -Datastore differentdatastore -DiskStorageFormat Thick

VMware vSphere folder types and role permissions

A few days ago I had some trouble giving role permissions for specific operations in VMware vSphere environment that I administrate. I’ve found out an important property of vSphere folder structure while solving these problems:

  • I would give a set of datastores to our backup administrators for restore operations. They already have propagated read-only permissions from the root of vCenter server. I’ve reserved three datastores, put them in a folder in Datastore view and given necessary propagated permissions to this particular folder explicitly for provisioning of restored VMs. They said they couldn’t do a single restore. It was giving error about the permissions. First I thought built-in read-only permissions override my explicit setting.
  • For a second case; I would give permission to VMs under a specific folder for OS administators:
    • console control, power operations,
    • complete control on vmnic,
    • change between the VLANs (port groups) that I preconfigured in vSwitch.

They could do all except for the last one. They couldn’t see any vSwitch port groups inside the drop-down menu in vmnic tab. But why? They had the permissions required, mentioned in this article. After several tries, I made it work only when I defined these permissions in another role and assigned this propagated role to them from the root of vCenter server.

I wondered that there is an issue about the propagation and permissions from the root of vCenter server. Until one day, when I hit something while writing a script in PowerCLI:

Types!!! Should have known that! I had googled it but all I could find was general idea about vSphere permissions.

First of all I’ve created a folder in “VMs and Templates” view for the backup admins and assigned the same role to this folder with the “Datastore” RESTORE folder. Because they need permissions on VMs also. I’ve told them to try to restore only in this “VM” RESTORE folder. They’ve succeeded. Bingo, first problem solved!

Then I’ve created a “Network” folder in Network view and put the port groups I defined in. I’ve assigned the same permissions to this “Network” folder with the OS Admin “VM” folder. They could see the VLANs afterwards even I deleted the work-around permissions from the root of vCenter server.

You see, these folders have types according to the views they are created in. If you need any more complex permissions, you must take this into account. It becomes crystal clear after getting the main idea and with a little research: “Create a Folder

Sorry that I can’t share my environment’s screenshots :/ But happy to help with any questions through comments 🙂