Question: Do you know the recent security patches released for Intel Spectre and Meltdown? Have you patched ESXi servers (Cluster/stand alone/DMZ/Lab)? Can you share couple of issues you ran into while patching those hosts?
Answer: This is hot topic for many Administrators not limited to VMware to start the new year 2018. Some background for “Meltdown and Spectre” which are critical vulnerabilities existing in several modern CPU: these hardware bugs allow programs to steal data which is currently processed on the computer. Meltdown and Spectre can affect personal computers, mobile devices, server and several cloud services.
Actually, the only way to minimize those security risks is to patch your operating systems or the hypervisor level (if you are using virtual machines) and here is the latest VMware KB Article about these patches.
These kind of questions from the Interviewer to check your awareness to active issues in the IT Infrastructure space and followed by troubleshoot depth based on the scenario you picked to explain the answer. Let me share couple of good scenarios
Scenario 1: Standalone ESXi host in remote branch office can’t be patched via Update Manager
ESXUPDATE Log file
2018-01-11T16:24:39Z esxupdate: 76791: downloader: DEBUG: Downloading http://vumserver:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip to /tmp/tmp4m_ng4…
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: An esxupdate error exception was caught:
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: Traceback (most recent call last):
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: File “/usr/sbin/esxupdate”, line 238, in main
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: cmd.Run()
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: File “/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/Cmdline.py”, line 105, in Run
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: File “/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py”, line 73, in DownloadMetadatas
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: MetadataDownloadError: (‘http://vumserver:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip’, None, “(‘http://vumserver:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip’, ‘/tmp/tmp4m_ng4’, ‘[Errno 4] IOError: [Errno 104] Connection reset by peer’)”)
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: DEBUG: <<<
You can start the explanation with there is a remote branch office which needs to be patched by shutting down VM’s (as your design has single ESXi host) but failed to apply the required patches with the above error message. Log file and symptoms are giving the indication about communication failure between VMware update manager server and ESXi host. Then you tried to copy the patches manually to the data store and tried to apply them but failed. As this is problem seems to be complex, then you opened a case with VMware support to resolve it. VMware support requested for log bundle from the ESXi server and Update manager server to analyze this issue. They found issues with patch repository which is resolved by renaming the file – “D:\ProgramData\VMware\VMware Update Manager\Data\hostupdate\vmw\vmw-ESXi-6.0.0-metadata.zip” and followed by downloading metadata from the patch source.
Scenario 2: Failed to migrate VM’s from ESXi host while trying to keep it in Maintenance Mode
Migrate virtual machine:
The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct
This scenario is kind of V-Motion issue stopping your patching activity as this is only way to perform live migration of running virtual machines and keep the ESXi maintenance mode. You started checking the V-Motion related settings like VMKernel, IP address and port group settings but all of them looks good. You tried to perform vmkping command from source to destination host but it failed. It seems there is communication issue for the V-Motion network and VLAN associated with it. When you engaged Network team to validate the connectivity, they confirm no firewall rule changes and other IP’s are able to communicate in the same subnet. This scenario forced you to check other settings. Upon more investigation you found that these are HP Blade servers and V-Motion network is configured as Internal network which doesn’t have an uplink associated. HP confirmed it’s known issue with the current Openview/c7000/Flex backend network settings. After contacting HP support they suggested to move the V-Motion network to another port group where communication is not broken. This helped you to migrate all the VM’s to other ESXi servers in the cluster to complete the patching activity.
“Be social and share it with social media, if you feel worth sharing it”