vRA 7.6

Password Operation Failed to Change the SSO password on an external PSC in VCF 3.11

Pradeep AdapaLeave a comment

Recently I came across an issue trying to change the SSO account (administrator@vsphere.local) password from the SDDC Manager using the Rotate password option under Security in VCF 3.11

I tried to Rotate the SSO password using the SDDC Manager, and got the following error:

However, Interesting thing is the sddc manager did change the SSO password in the backend

However, to check on this error, I dug a little deeper and saw the following error in the password rotate task:

I used the following command to check the operationsmanager.log to check the log in SDDC Manager

less /var/log/vmware/vcf/operationsmanager/operationsmanager.log

The log also shows that the sddc manager is trying to change the sso credential (administrator@vsphere.local) on VRA endpoints

I had to open a VMware Support ticket and here is the answer I received:

“As per the Engineering team this issue is due to a misconfiguration of vRA endpoints. SDDC Manager is trying to change the administrator@vsphere.local on the VRA endpoints but VRA endpoints are configured with a different user (vcf-secured-user@vsphere.local).  This issue is addressed in VCF 4.x”

What the VMware Engineering team is saying is that in VCF 3.10.x, 3.11 there is an issue with VRA as it is typically configured using a different tenant admin instead of using administrator@vsphere.local user to configure the endpoints in it. However, the SDDC manager is trying to change the administrator@vsphere.local credential on VRA endpoints. Hence this issue. Looks like this issue has been fixed in VCF 4.x

This resolves the issue at this time as we will be working to upgrade our VCF to 4.x soon.

VRA Agent Status Down in VRA 7.6, LDAPS Certificate Issue

Pradeep Adapa1 Comment

Recently came across an issue in our Production environment that VRA Agent status was showing as Down in one of our Sites.

The screenshot is shown as below:

This screenshot has 2 clusters

On investigating, we checked the vSphereAgent.log file which is present on the server where this VRA agent was installed and configured. (In our case it was one one of the IWS (IAAS Web Server) Node)

The location of this log file is at C:\Program Files (x86)\VMware\vCAC\Agents\<VRA_Agent_Name>\logs\

In this log, you can find multiple lines with an error:

This exception was caught:
System.Web.Services.Protocols.SoapException: vCenter Error: Cannot complete login due to an incorrect user name or password.

if this is the case, check the LDAPS Certificate to your Domain Controllers of the domain you have added on the vCenter server Web UI.

Even though it doesn’t show you the certificate expiry in this UI, you can check the certificate status by logging into vcenter SSH and executing the following command:

openssl s_client -connect adds01.corp.test.local:636 -showcerts

Replace the Domain Controller hostname with your domain controller hostname after the -connect in the above command to get the valid cert from the domain controller.

In our case, we found that the cert on the domain controller has been recently renewed and we had to input the new cert to the Identity Source in the vcenter web UI.

Once the new cert is installed, you can login into your VRA Default Tenant (VRA 7.6), go to Infrastructure -> Endpoints -> Endpoints and go to your vcenter and click on edit and then re-validate the service account password (Test the connection) and once it is successful, the VRA Agent will come back UP.

Testing the connection to the vcenter using the service account which is already added and the test is successful.

Hope this article helps you if you see your VRA agents as down and can’t find anything else missing or even restarting the vra agent service doesn’t change the status.

VRA Proxy Agent Down and Inventory Data Collection stuck ‘in progress’ – VRA 7.6

Pradeep AdapaLeave a comment

Recently we had an issue where in one of our Sites (We have multiple sites in VCF), the VRA Proxy Agent was showing as Down and restarting the services (VRA Agent) on the ims (Infrastructure Manager Service) did not bring the agent up.

Here is the process to check if the ims load balancer address is entered in the VRMAgent.exe.config file on the ims server.

Issue: In our case, the VRM Agent was installed on the active Infrastructure Manager Service server (ims01a), However, the vrm agent config only had this entry instead of the load balancer entry (imslb) in its configuration. So, when the ims01a became passive and node ims01b became active node, this broke the VRM agent and the agent status became down.

Solution: Edit the vrmagent.config file and update the lines 83 and 104 pointing this file to the ims load balancer hostname so that when the ims servers change active-passive state, the VRM agent will not go down.

Before we continue, stop the service “VMware vCloud Automation Center Agent – agent_name” (Here in my example the agent name is dc2)

Pictures of the issue are below:

VRM Agent status showing as Down
Data Collection status showing as in progress but not changing state to successful

Solution Screenshots are as below:

Location of the VRMAgent.exe.config file on the iws (Infrastructure Web Server) node
Line 83 where you will need to change the hostname to the ims loadbalancer. (In this screenshot, the load balancer hostname is https://dc1vraimslb.domain.local)
line 104 where you need to edit the endpoint address to be the load balancer hostname

Once these modifications are done in this config file, you save it and then start the service “VMware vCloud Automation Center Agent – dc2 (where dc2 is the agent name configured when the agent was installed on this server)

Disclaimer: As this Environment is Property of my Company, The Original names have either been modified or pixelated for Privacy.

Once the agent service is started, you can go back to VRA and check the Agent Status and it will be up and the in progress data collection will actually complete in few minutes (For my environment it took atleast 15-20 minutes for the inventory to complete).

Hope this article helps if you face the same issue in VRA 7.6!

NSX Plugin 1.2 in VRA 7.6 Not Generating NSX Security Groups in a Page (NOT SOLVED YET)

Pradeep AdapaLeave a comment

Recently, we have an ongoing issue where the NSX Plugin in VRO is not populating one page out of 4 pages and this is messing up our VRO Code to create and put Security Tags (NSX-V) on our VMs.

Below is a screenshot of the issue

This shows that the other pages have security groups in them but page-1 under one of the NSX Manager (NSX-V version 6.4.x) are not populated.

I have already deleted and re-installed the NSX-V Plugin using the VRO Control Center to no resolution.

The issue is not resolved yet and I will update this post with the resolution soon.

How to Unregister a VM which is missing in VRA 7.6

Pradeep AdapaLeave a comment

Recently I had to get rid of multiple vms through VRA, However, I found that some of the vms status was showing as missing. This happens if the VM has already been deleted through the vCenter and VRA can’t find that VM in the vCenter.

The way you can see the missing status is you go to the deployments tab, check the Status if its ON, OFF or Missing (?) as the screenshot shows below:

The missing status is displayed next to the VM Name

Some of the info in the screenshot has been removed to protect my Organization Data and the VM Names have also been changed for the same purpose.

In VRA 7.6, you can unregister it easily using the GUI, You click on the Deployment Name

Then click on the VM Name itself (in this case its DC1Test001), then click on the small gear icon and then click on the option “Unregister” in the drop down menu as in the screenshot below:
The unregister option will remove this VM from the VRA internal DB so that it doesn’t show up in VRA.

Hope this post helps, as I was not able to see any blog posts regarding this simple unregister procedure in VRA 7.6

VRA 7.6 with VCF 3.10.x SDDC Manager AD Error

Pradeep AdapaLeave a comment

I have recently come across an issue in our new VCF 3.10.x build that when we try to deploy the VRA using SDDC Manager, we get an error that the AD Account we have provided can’t validate with the Domain.

The warning is as shown in the picture below:

Note That I had to change a few details and also blur some details from my environment due to privacy reasons.

The Error basically states that VRA is not able to communicate to my domain lab.com with the service account lab\svc_vra_adm because it is trying to contact test.lab.com instead of lab.com Domain

test.lab.com is a DNS Zone in our actual root Domain lab.com and all our VRA Appliances have the host records added to test.lab.com instead of the root domain.

After multiple tries and VMware support, we got to know that VRA (7.x and 8.x) doesn’t support explicit identification of the Active Directory domain name. The kb article which mentions this issue is

https://kb.vmware.com/s/article/59128

The Solution is to make sure that the host records of your VRA is the same as your ‘Actual‘ Domain, in this case lab.com and then retry the validation using the SDDC Manager with the same service account lab\svc_vra_adm

This time, the validation should pass.

Install & Configure VRLCM 2.1 Part-2

Pradeep AdapaLeave a comment

Next, We Create a New Environment and then create an New VRA environment using vRLCM

Go to Home and Click on Create Environment to get started

Click on Create Environment
The Default password is used for all the products being deployed using this instance
In this case, we selected the vRA deployment with deployment type as Small for the lab

Agree to the EULA, click Next

Enter the License

Select the NTP Servers and then click Next
Input all the Network Details and click Next

Select the Certificate which we have generated before and click Next

This is where things have gotten tricky in this version as we have multiple options to define the VRA environment including the windows template to create new vms themseleves.

let us go step by step process

Under Product Properties, provide the windows server username and password which you want to access after the box has been provisioned using the windows template.

Scroll down for further options

In the above configuration, We have only 3 VMs being deployed in VRA Simple Configuration.

  • VRA Primary Appliance
  • VRA DB server (Database server)
  • VRA IAAS web server (this contains iaas-web server, iaas manager, iaas DEM Worker and proxy-agent-vsphere )

Once all the Product details of VRA are put in, we will proceed to the precheck phase.

Click on RUN PRECHECK option to continue

Next, we click on Validate & Deploy option to deploy the vms

Make sure you disable UAC in the windows template and then click on Validate & Deploy option to continue.

The Validation process will start
Looks like my test failed with 2 Items, which I will be rectifying before trying to Validate again before Deployment

NOTE: The re-validation took more than 30 mins in my lab to complete. Not sure why it took a lot of time, but I suggest you all to be patient during this process as there is no way to speed it up.

The validation is successful and now we can go ahead and run the PRECHECK to continue

NOTE that at this point, I haven’t installed SQL Software on the SQL Server, but VRSLM has created an windows server for both the db and iaas install. I will have to install SQL Server on the db windows VM and see how it goes.

This Post is pending and I will be updating it soon once I have some clarification on if I need to install and configure the SQL software in the vRA SQL server windows machine or will the scripts do it if I provide the SQL ISO file. Stay Tuned …….