Recently, I have patched our Aria Suite Lifecycle Manager from 8.12.0 to 8.12.x Patch 2 from its web UI and encountered a strange issue where we were getting the following error when trying to get into any of the settings like ‘System Patches, System Upgrade, System Settings, DNS, NTP Servers, Binary Mapping’ etc., on the web UI.
Upon closer inspection, I couldn’t find anything wrong with the appliance itself or its services.
I even restarted the appliance with no success
I checked the vpostgres service, vrlcm-server service and they were active.
Resolution:
I stopped and started the vrlcm-service and this resolved the issue. I am not able to get into all the settings on the VRSLCM Web UI.
The above screenshot shows the commands to stop and start the service on the VRSLCM SSH Session.
Recently, I had to use the Asyncpatch tool in SDDC Manager to Patch our vcenter to 7.0U3o due to the Critical Security patch VMSA-2023-0023 and came across this issue when performing the precheck for Management Domain in SDDC Manager.
If you Expand “Sddc Security Configuration”, the error was on the option “VMware Cloud Foundation certificate validation check”
if you come across this issue, perform the following commands to enable the Certificate Validation Check in SDDC Manager
You might have come across an issue where the VCF REST API User account admin@local password is either lost or need to reset the password of this account.
NOTE:This is NOT an official way to do it according to VMware and is meant to be done under the supervision of VMware Support.
Login into the SDDC Manager as vcf and go to the root user prompt
sddcmanager#
type the following commands to reset the password of admin@local account on the sddc manager
mkdir -p /etc/security/local
chown root:vcf_services /etc/security/local
chmod 650 /etc/security/local
echo -n "" > /etc/security/local/.localuserpasswd
chown root:vcf_services /etc/security/local/.localuserpasswd
chmod 660 /etc/security/local/.localuserpasswd
type the following command to set a new password - in this example it is NewP@SSW0rd010
echo -n 'NewP@SSW0rd010' | openssl dgst -sha512 -binary | openssl enc -base64 | tr -d '\n' > /etc/security/local/.localuserpasswd
Once you have changed the password, I would recommend to restart the sddc manager services using the following command
Recently, after upgrading our SDDC Manager from 3.11.1 to 4.5, I came across an UI message on the SDDC Password Management tab as follows:
Issue Description: The error message shown in the screenshot above is “Failed to get tasks data. Something went wrong. Please retry or contact the service provider and provide the reference token.“
Root Cause: As I was digging deeper into this, I found that the functionality of the password manager itself was fine, but it was somehow not able to get the complete task list data and hence throwing this error out.
2023-08-08T16:02:57.768+0000 ERROR [vcf_om,0000000000000000,0000] [o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7300-exec-6] Servlet.service() for servlet [dispatcherServlet] in context wit h path [/operationsmanager] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: No enum constant com.vmware.vcf.passwordmanager.exception.PasswordManag erErrorCode.PASSWORD_MANAGER_VRA_ENDPOINT_FAILED] with root cause java.lang.IllegalArgumentException: No enum constant com.vmware.vcf.passwordmanager.exception.PasswordManagerErrorCode.PASSWORD_MANAGER_VRA_ENDPOINT_FAILED
This means that after the SDDC Manager Upgrade, it was not able to determine the error with “PASSWORD_MANAGER_VRA_ENDPOINT_FAILED” in its internal database
Solution: After consulting with VMware Engineering, the issue was resolved by going into the operationsmanager db on the sddc manager and executing the following command to replace the message with something which the SDDC Manager could understand.
Disclaimer: Do this at your own risk, I would highly recommend to contact VMware GSS if you have the same issue to get an official resolution.
operationsmanager=# update passwordmanager.master_password_transaction set diagnostic_message = replace (diagnostic_message, 'PASSWORD_MANAGER_VRA_ENDPOINT_FAILED','PASSWORD_UPDATE_VRA_ENDPOINT_FAILED');
Once this is executed, you will get an message of UPDATE XX (where XX is the update number if it has been updated successfully)
Conclusion: Now, you can refresh the SDDC Manager UI and you will not see the error message on the UI in Password Management as before.
Hope this helps if you see similar message after your VCF Upgrade from 3.11.x to 4.5
Recently I came across a few issues while preparing for an VCF Upgrade from 3.11.x version to 4.x in our environment.
Below are few of the issues and how to resolve them using commands on the SDDC Manager
Issue: SDDC Manager UI shows the message “Password Manager option failed in pre-validation stage” or when you go to the security tab for password management, it shows that one of the password tasks have failed.
Issue: Deployment locked by password manager when you try to rotate the passwords of PSC, VCENTER from the security -> password management tab
Solution/s:
Find the Deployment Lock ID in the sddc manager by logging into sddc manager using SSH, login as VCF and then root user and use the following command
Recently I came across an issue trying to change the SSO account (administrator@vsphere.local) password from the SDDC Manager using the Rotate password option under Security in VCF 3.11
I tried to Rotate the SSO password using the SDDC Manager, and got the following error:
However, Interesting thing is the sddc manager did change the SSO password in the backend
However, to check on this error, I dug a little deeper and saw the following error in the password rotate task:
I used the following command to check the operationsmanager.log to check the log in SDDC Manager
less /var/log/vmware/vcf/operationsmanager/operationsmanager.log
The log also shows that the sddc manager is trying to change the sso credential (administrator@vsphere.local) on VRA endpoints
I had to open a VMware Support ticket and here is the answer I received:
“As per the Engineering team this issue is due to a misconfiguration of vRA endpoints. SDDC Manager is trying to change the administrator@vsphere.local on the VRA endpoints but VRA endpoints are configured with a different user (vcf-secured-user@vsphere.local). This issue is addressed in VCF 4.x”
What the VMware Engineering team is saying is that in VCF 3.10.x, 3.11 there is an issue with VRA as it is typically configured using a different tenant admin instead of using administrator@vsphere.local user to configure the endpoints in it. However, the SDDC manager is trying to change the administrator@vsphere.local credential on VRA endpoints. Hence this issue. Looks like this issue has been fixed in VCF 4.x
This resolves the issue at this time as we will be working to upgrade our VCF to 4.x soon.
Recently I had to check the existing passwords in sddc manager in our VCF 3.11 environment and found out there is a simple way. Here it is.
SSH into your SDDC Manager using vcf user and go to the root prompt using su command and use the below command:
root@sddcmgr01 [home/vcf]# lookup_passwords
Screenshot:
This will bring up all the products which sddc manager keeps track of
Select any product and then you will have to provide the sddc secured user credentials which you provided at the time of deploying SDDC manager in the VCF environment. This credential is also used for the backup of SDDC Manager and NSX Components.
in this case ESXI was selected to display the esxi hosts credentials
This way, you can get all the passwords for all the components controlled by SDDC Manager in VCF 3.x
NOTE/Disclaimer: I had to Blur/Pixelate certain components in my screenshots as they are in a live environment.
I wanted to ping back one of the great article by one of my fellow vExpert Shank Mohan on his website about an unofficial VCF Troubleshooting guide. I have learned from this article and would like to remember this article and hence posting it back on my blog.
I was getting ready to patch our environment from VCF 3.10.2.2 to VCF 3.11 as VMware has officially released a complete Patch for VCF 3.10.x this month, when I was performing the VCF Upgrade Pre-Check for the Management Domain, I came across this issue
The LCM Pre-Check Failed due to a directory permission issue for one of the lcm directory
Issue is that the pre-check says that the directory “/var/log/vmare/vcf/lcm/upgrades/<long code directory>/lcmAbout” owner is root but the owner needs to be user vcf_lcm
This is how I resolved the issue:
Login into SDDC Manager as user vcf, do su and provide the root password
then go to the following directory “/var/log/vmware/vcf/lcm/upgrades/<long code directory as displayed in the lcm error on sddc manager>
chown vcf_lcm lcmAbout
chmod 750 lcmAbout
The above two commands will change the owner from root to vcf_lcm and also provide the required permissions to the folder so the pre-check can complete.
The full screenshot of what I performed is below:
Commands to change owner to vcf_lcm and to provide the required permissions for the folder lcmAbout
Once you perform the commands above, you can run the pre-check and this time it will proceed successfully as shown below
Hope this article helps if you come across this issue with sddc manager upgrade from VCF 3.10.2.2 to 3.11
VMware has finally realeased an patch version for VCF 3.x and the version is 3.11. You can only download this as a patch form from the SDDC Manager. You can Upgrade to version 3.11 from 30.10.2.2 or VCF 3.5 or later.
Security fixes for Apache Log4j Remote Code Execution Vulnerability: This release fixes CVE-2021-44228 and CVE-2021-45046. See VMSA-2021-0028.
Security fixes for Apache HTTP Server: This release fixes CVE-2021-40438. See CVE-2021-40438.
Improvements to upgrade prechecks: Upgrade prechecks have been expanded to verify filesystem capacity, file permissions, and passwords. These improved prechecks help identify issues that you need to resolve to ensure a smooth upgrade.
This also resolves the following Security Advisory VMSA-2022-0004 which deals with several vulnerabilities in esxi 6.7 hosts
This also resolves the vulnerability in VCF SDDC Manager 3.x according to the security advisory VMSA-2022-0003
This version also addresses the heap-overflow vulnerability in esxi hosts according to the security advisory VMSA-2022-0001.2
The Updated product versions according to the BOM for VCF 3.11 are
Hope this post helps for the teams who have VCF 3.10.x and waiting for the long awaited log4j patch instead of an workaround.
Hyperconverged Infrastructure & Virtualization 