VCF 3.11 – Converged, Hyperconverged Infrastructure & Virtualization

SDDC Manager 3.x [3.11.x] Issues & Solutions

January 12, 2023Pradeep AdapaLeave a comment

Hello All,

Recently I came across a few issues while preparing for an VCF Upgrade from 3.11.x version to 4.x in our environment.

Below are few of the issues and how to resolve them using commands on the SDDC Manager

Issue: SDDC Manager UI shows the message “Password Manager option failed in pre-validation stage” or when you go to the security tab for password management, it shows that one of the password tasks have failed.

Issue: Deployment locked by password manager when you try to rotate the passwords of PSC, VCENTER from the security -> password management tab

Solution/s:

Find the Deployment Lock ID in the sddc manager by logging into sddc manager using SSH, login as VCF and then root user and use the following command

psql --host=localhost -U postgres -d platform -c "select * from lock"

This will display the ID’s of the locked tasks in the SDDC Manager

Delete the locked task by using the following command

psql --host=localhost -U postgres -d platform -c "delete from lock where id=<ID displayed above>"

Once the task is deleted, the lock is released. You can now refresh the SDDC Manager UI and then continue with the password update or rotate options

Issue: Remove Failed Tasks in SDDC Manager

Solution:

Reference – https://www.martingustafsson.com/removing-failed-tasks-in-sddc-manager/

Multiple Useful Commands for SDDC 3.x

The Unofficial VCF Troubleshooting Guide v2 – https://www.lab2prod.com.au/2021/03/the-unofficial-vcf-troubleshooting-guide.html

Reference

MANAGING VMWARE CLOUD FOUNDATION – FIRST LOOK

Password Operation Failed to Change the SSO password on an external PSC in VCF 3.11

October 28, 2022October 28, 2022Pradeep AdapaLeave a comment

Recently I came across an issue trying to change the SSO account (administrator@vsphere.local) password from the SDDC Manager using the Rotate password option under Security in VCF 3.11

I tried to Rotate the SSO password using the SDDC Manager, and got the following error:

However, Interesting thing is the sddc manager did change the SSO password in the backend

However, to check on this error, I dug a little deeper and saw the following error in the password rotate task:

I used the following command to check the operationsmanager.log to check the log in SDDC Manager

less /var/log/vmware/vcf/operationsmanager/operationsmanager.log

The log also shows that the sddc manager is trying to change the sso credential (administrator@vsphere.local) on VRA endpoints

I had to open a VMware Support ticket and here is the answer I received:

“As per the Engineering team this issue is due to a misconfiguration of vRA endpoints. SDDC Manager is trying to change the administrator@vsphere.local on the VRA endpoints but VRA endpoints are configured with a different user (vcf-secured-user@vsphere.local). This issue is addressed in VCF 4.x”

What the VMware Engineering team is saying is that in VCF 3.10.x, 3.11 there is an issue with VRA as it is typically configured using a different tenant admin instead of using administrator@vsphere.local user to configure the endpoints in it. However, the SDDC manager is trying to change the administrator@vsphere.local credential on VRA endpoints. Hence this issue. Looks like this issue has been fixed in VCF 4.x

This resolves the issue at this time as we will be working to upgrade our VCF to 4.x soon.

Check for Passwords in SDDC Manager in VCF 3.x

July 22, 2022Pradeep AdapaLeave a comment

Recently I had to check the existing passwords in sddc manager in our VCF 3.11 environment and found out there is a simple way. Here it is.

SSH into your SDDC Manager using vcf user and go to the root prompt using su command and use the below command:

root@sddcmgr01 [home/vcf]# lookup_passwords

Screenshot:

This will bring up all the products which sddc manager keeps track of

Select any product and then you will have to provide the sddc secured user credentials which you provided at the time of deploying SDDC manager in the VCF environment. This credential is also used for the backup of SDDC Manager and NSX Components.

in this case ESXI was selected to display the esxi hosts credentials

This way, you can get all the passwords for all the components controlled by SDDC Manager in VCF 3.x

NOTE/Disclaimer: I had to Blur/Pixelate certain components in my screenshots as they are in a live environment.

LCM Directory Permission Error When pre-checking for SDDC Manager Upgrade with VCF 3.11 Patch

February 19, 2022Pradeep AdapaLeave a comment

I was getting ready to patch our environment from VCF 3.10.2.2 to VCF 3.11 as VMware has officially released a complete Patch for VCF 3.10.x this month, when I was performing the VCF Upgrade Pre-Check for the Management Domain, I came across this issue

The LCM Pre-Check Failed due to a directory permission issue for one of the lcm directory

Issue is that the pre-check says that the directory “/var/log/vmare/vcf/lcm/upgrades/<long code directory>/lcmAbout” owner is root but the owner needs to be user vcf_lcm

This is how I resolved the issue:

then go to the following directory “/var/log/vmware/vcf/lcm/upgrades/<long code directory as displayed in the lcm error on sddc manager>

chown vcf_lcm lcmAbout
chmod 750 lcmAbout

The above two commands will change the owner from root to vcf_lcm and also provide the required permissions to the folder so the pre-check can complete.

The full screenshot of what I performed is below: