Recently I came across a few issues while preparing for an VCF Upgrade from 3.11.x version to 4.x in our environment.
Below are few of the issues and how to resolve them using commands on the SDDC Manager
Issue: SDDC Manager UI shows the message “Password Manager option failed in pre-validation stage” or when you go to the security tab for password management, it shows that one of the password tasks have failed.
Issue: Deployment locked by password manager when you try to rotate the passwords of PSC, VCENTER from the security -> password management tab
Find the Deployment Lock ID in the sddc manager by logging into sddc manager using SSH, login as VCF and then root user and use the following command
Recently I came across an issue trying to change the SSO account (email@example.com) password from the SDDC Manager using the Rotate password option under Security in VCF 3.11
I tried to Rotate the SSO password using the SDDC Manager, and got the following error:
However, Interesting thing is the sddc manager did change the SSO password in the backend
However, to check on this error, I dug a little deeper and saw the following error in the password rotate task:
I used the following command to check the operationsmanager.log to check the log in SDDC Manager
The log also shows that the sddc manager is trying to change the sso credential (firstname.lastname@example.org) on VRA endpoints
I had to open a VMware Support ticket and here is the answer I received:
“As per the Engineering team this issue is due to a misconfiguration of vRA endpoints. SDDC Manager is trying to change the email@example.com on the VRA endpoints but VRA endpoints are configured with a different user (firstname.lastname@example.org). This issue is addressed in VCF 4.x”
What the VMware Engineering team is saying is that in VCF 3.10.x, 3.11 there is an issue with VRA as it is typically configured using a different tenant admin instead of using email@example.com user to configure the endpoints in it. However, the SDDC manager is trying to change the firstname.lastname@example.org credential on VRA endpoints. Hence this issue. Looks like this issue has been fixed in VCF 4.x
This resolves the issue at this time as we will be working to upgrade our VCF to 4.x soon.
Recently I had to check the existing passwords in sddc manager in our VCF 3.11 environment and found out there is a simple way. Here it is.
SSH into your SDDC Manager using vcf user and go to the root prompt using su command and use the below command:
root@sddcmgr01 [home/vcf]# lookup_passwords
Select any product and then you will have to provide the sddc secured user credentials which you provided at the time of deploying SDDC manager in the VCF environment. This credential is also used for the backup of SDDC Manager and NSX Components.
This way, you can get all the passwords for all the components controlled by SDDC Manager in VCF 3.x
NOTE/Disclaimer: I had to Blur/Pixelate certain components in my screenshots as they are in a live environment.
VMware has finally realeased an patch version for VCF 3.x and the version is 3.11. You can only download this as a patch form from the SDDC Manager. You can Upgrade to version 3.11 from 220.127.116.11 or VCF 3.5 or later.
Security fixes for Apache Log4j Remote Code Execution Vulnerability: This release fixes CVE-2021-44228 and CVE-2021-45046. See VMSA-2021-0028.
Security fixes for Apache HTTP Server: This release fixes CVE-2021-40438. See CVE-2021-40438.
Improvements to upgrade prechecks: Upgrade prechecks have been expanded to verify filesystem capacity, file permissions, and passwords. These improved prechecks help identify issues that you need to resolve to ensure a smooth upgrade.
This also resolves the following Security Advisory VMSA-2022-0004 which deals with several vulnerabilities in esxi 6.7 hosts
This also resolves the vulnerability in VCF SDDC Manager 3.x according to the security advisory VMSA-2022-0003
This version also addresses the heap-overflow vulnerability in esxi hosts according to the security advisory VMSA-2022-0001.2
The Updated product versions according to the BOM for VCF 3.11 are
Hope this post helps for the teams who have VCF 3.10.x and waiting for the long awaited log4j patch instead of an workaround.