VRA Agent Status Down in VRA 7.6, LDAPS Certificate Issue

Recently came across an issue in our Production environment that VRA Agent status was showing as Down in one of our Sites.

The screenshot is shown as below:

This screenshot has 2 clusters

On investigating, we checked the vSphereAgent.log file which is present on the server where this VRA agent was installed and configured. (In our case it was one one of the IWS (IAAS Web Server) Node)

The location of this log file is at C:\Program Files (x86)\VMware\vCAC\Agents\<VRA_Agent_Name>\logs\

In this log, you can find multiple lines with an error:

This exception was caught:
System.Web.Services.Protocols.SoapException: vCenter Error: Cannot complete login due to an incorrect user name or password.

if this is the case, check the LDAPS Certificate to your Domain Controllers of the domain you have added on the vCenter server Web UI.

Even though it doesn’t show you the certificate expiry in this UI, you can check the certificate status by logging into vcenter SSH and executing the following command:

openssl s_client -connect adds01.corp.test.local:636 -showcerts

Replace the Domain Controller hostname with your domain controller hostname after the -connect in the above command to get the valid cert from the domain controller.

In our case, we found that the cert on the domain controller has been recently renewed and we had to input the new cert to the Identity Source in the vcenter web UI.

Once the new cert is installed, you can login into your VRA Default Tenant (VRA 7.6), go to Infrastructure -> Endpoints -> Endpoints and go to your vcenter and click on edit and then re-validate the service account password (Test the connection) and once it is successful, the VRA Agent will come back UP.

Testing the connection to the vcenter using the service account which is already added and the test is successful.

Hope this article helps you if you see your VRA agents as down and can’t find anything else missing or even restarting the vra agent service doesn’t change the status.

Advertisement

Great VCF Troubleshooting Guide by my Fellow vExpert

I wanted to ping back one of the great article by one of my fellow vExpert Shank Mohan on his website about an unofficial VCF Troubleshooting guide. I have learned from this article and would like to remember this article and hence posting it back on my blog.

Great VCF Troubleshooting guide by Shank Mohan

LCM Directory Permission Error When pre-checking for SDDC Manager Upgrade with VCF 3.11 Patch

I was getting ready to patch our environment from VCF 3.10.2.2 to VCF 3.11 as VMware has officially released a complete Patch for VCF 3.10.x this month, when I was performing the VCF Upgrade Pre-Check for the Management Domain, I came across this issue

The LCM Pre-Check Failed due to a directory permission issue for one of the lcm directory

Issue is that the pre-check says that the directory “/var/log/vmare/vcf/lcm/upgrades/<long code directory>/lcmAbout” owner is root but the owner needs to be user vcf_lcm

This is how I resolved the issue:

Login into SDDC Manager as user vcf, do su and provide the root password

then go to the following directory “/var/log/vmware/vcf/lcm/upgrades/<long code directory as displayed in the lcm error on sddc manager>

chown vcf_lcm lcmAbout
chmod 750 lcmAbout

The above two commands will change the owner from root to vcf_lcm and also provide the required permissions to the folder so the pre-check can complete.

The full screenshot of what I performed is below:

Commands to change owner to vcf_lcm and to provide the required permissions for the folder lcmAbout

Once you perform the commands above, you can run the pre-check and this time it will proceed successfully as shown below

Hope this article helps if you come across this issue with sddc manager upgrade from VCF 3.10.2.2 to 3.11

VCF 3.x patch 3.11 for Log4J Vulnerability and Other Security Patches included

VMware has finally realeased an patch version for VCF 3.x and the version is 3.11. You can only download this as a patch form from the SDDC Manager. You can Upgrade to version 3.11 from 30.10.2.2 or VCF 3.5 or later.

VMSA-2021-0028.13 (vmware.com)

This Release VCF 3.11 includes the following:

  • Security fixes for Apache Log4j Remote Code Execution Vulnerability: This release fixes CVE-2021-44228 and CVE-2021-45046. See VMSA-2021-0028.
  • Security fixes for Apache HTTP Server: This release fixes CVE-2021-40438. See CVE-2021-40438.
  • Improvements to upgrade prechecks: Upgrade prechecks have been expanded to verify filesystem capacity, file permissions, and passwords. These improved prechecks help identify issues that you need to resolve to ensure a smooth upgrade.
  • This also resolves the following Security Advisory VMSA-2022-0004 which deals with several vulnerabilities in esxi 6.7 hosts
  • This also resolves the vulnerability in VCF SDDC Manager 3.x according to the security advisory VMSA-2022-0003
  • This version also addresses the heap-overflow vulnerability in esxi hosts according to the security advisory VMSA-2022-0001.2

The Updated product versions according to the BOM for VCF 3.11 are

Hope this post helps for the teams who have VCF 3.10.x and waiting for the long awaited log4j patch instead of an workaround.

VMware Cloud Foundation (VCF) API Reference Guides

Here is the direct link to the API Reference Guide for VMware Cloud Foundation (VCF)

https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_overview

This is the Generic API Reference Guide for VCF instead of being Version Centric.

Updated – 2/16/2022

From Version VCF 4.3.1 (Non-VxRail), VMware has moved their API Guides to a new location which is a better format than before and more user friendly.

VCF 4.3.1 New API Reference Guide

VCF 4.4.0 New API Reference Guide

Updated – 1/25/2022

For Version Centric API Guides

VCF 3.10 API Reference Guide

VCF 4.0 API Reference Guide

VCF 4.1 API Reference Guide

VCF 4.2 API Reference Guide

VCF 4.3 API Reference Guide

VCF 4.3.1 API Reference Guide

NOTE: These Reference Guides and their versions are for NON VXRAIL implementations. They are valid for Regular VCF Implementation with VSAN Ready nodes.