Question: Have you ever fixed vSphere ESX Agent Manager (EAM) failure issues with NSX Solution? What do you think about below error message?
In the /var/log/netcpa.log file on the ESXi host, you see entries similar to:
Netcpa error: error netcpa[FFCFFB70] [Originator@6876 sub=Default] Failed to get host id, returned , len 0
Answer: NSX became core Network solution for most of Cloud Companies. IBM Soft Layer – VMware became Strategic Partners to deliver VMware solutions for Soft Layer customers. You can find more details here. Interviewer wants to check your Host preparation skills for NSX usage in this question. Host preparation involve couple of steps including installing VIB modules for esx-dvfilter-switch-security, esx-vsip & esx-vxlan.
Host preparation is the process in which the NSX Manager:
1) Installs NSX kernel modules on ESXi hosts that are members of vCenter clusters
2) Builds the NSX control-plane and management-plane fabric
NSX kernel modules packaged in VIB files run within the hypervisor kernel and provide services such as distributed routing, distributed firewall, and VXLAN bridging capabilities. To prepare your environment for network virtualization, you must install network infrastructure components on a per-cluster level for each vCenter server where needed. This deploys the required software on all hosts in the cluster. When a new host is added to this cluster, the required software is automatically installed on the newly added host. vSphere ESX Agent Manager automates the process of deploying and managing vSphere ESX agents. Once you are clear with basics then you can explain the possible reasons to troubleshoot VIB’s deployment failure using VMware ESX Agent Manager (EAM), Installation Status showing Not Ready on the Clusters and Hosts.
If you can speak or explain 4 or more steps to the Interviewer, then he/she will be happy to consider you for the applied position of NSX Engineer 🙂
- Ensure that the DNS is configured correctly on the ESXi hosts and the environment. The VMware ESX Agent Manager (EAM) may fail to deploy all required VIBs during installation of network virtualization components on vSphere hosts during Host Preparation in NSX for vSphere 6.x
- Verify that all required ports are open between ESXi hosts, vCenter Server and NSX Manager, and is not being blocked by a Firewall.
- If you are using VMware Update Manager, ensure that VUM is available and running.
- Verify that the URL http://VC_IP/eam/mob/ is accessible
- Ensure that the EAM is able to access the VIBs from NSX Manager. Review the eam.log file and look for VIB inaccessiblemessages. Also check the vCenter Server registration in NSX Manager.
- Ensure that the ESXi hosts are able to access VIBs from the vCenter Server. You see this error: esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known in the esxupdate.log file. This can occur if the ESXi host is unable to access the vCenter Server’s Fully Qualified Domain Name (FQDN).
- Verify that the cluster contains the correct agencies. If an EAM agency is missing on an already prepared cluster, new hosts can fail to prepare. To work around this issue, create a dummy cluster and prepare it. This forces NSX/EAM to update the configuration of all existing clusters, creating a new EAM Agency for the problematic cluster.
- Ensure that the ESXi hosts are rebooted after an upgrade. Failure to manually reboot the ESXi host marks them as red in theInstallation Status.Note: In a DRS enabled cluster, an ESXi host reboot can be initiated by clicking the Resolve option on the cluster.
- If you experience stale agencies with the error: Netcpa error: error netcpa[FFCFFB70]:
- Create a new cluster.
- Add the new cluster to the transport zone.
- Move the affected ESXi host to the new cluster.
- Re-attempt to install the VIBs on the affected ESXi host.
- If you see errorCode =99 on the eam.log file, this means that EAM incorrectly mark Host Preparation as successful. To resolve this issue, manually reboot the affected ESXi host.
- If you see Failed to complete VIB task on the eam.log file, this means that the vSphere Update Manager (VUM) are having issues installing NSX VIBs. To resolve this issue, restart the vSphere Update Manager service.
- If you see the error esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known) in the esxupdate.log file, this may happen if the ESXi host cannot reach the vCenter Server Fully Qualified Domain Name (FQDN). To resolve this issue, set the correct the vCenter Server Managed IP address.
- Sometimes after restarting NSX Manager, EAM or vCenter Server, you notice the cluster status reported Not Ready in Host Preparation tab, in Installation section of NSX Manager. This is a false-positive status that is a result of restarting one of the components. To get the state updated, click the Resolve All button.