How to Unregister a VM which is missing in VRA 7.6

Recently I had to get rid of multiple vms through VRA, However, I found that some of the vms status was showing as missing. This happens if the VM has already been deleted through the vCenter and VRA can’t find that VM in the vCenter.

The way you can see the missing status is you go to the deployments tab, check the Status if its ON, OFF or Missing (?) as the screenshot shows below:

The missing status is displayed next to the VM Name

Some of the info in the screenshot has been removed to protect my Organization Data and the VM Names have also been changed for the same purpose.

In VRA 7.6, you can unregister it easily using the GUI, You click on the Deployment Name

Then click on the VM Name itself (in this case its DC1Test001), then click on the small gear icon and then click on the option “Unregister” in the drop down menu as in the screenshot below:
The unregister option will remove this VM from the VRA internal DB so that it doesn’t show up in VRA.

Hope this post helps, as I was not able to see any blog posts regarding this simple unregister procedure in VRA 7.6

VMKPING and its uses in ESXI

I have recently been working with esxi hosts and to decommission them and recommission them into new projects and had to use the command vmkping to test the MTU of certain types of vmkernel ports like VMOTION, VSAN, VTEPs etc.

Here is a refresher for the vmkping commands which are very useful for a day to day Virtual Administrator

Command to check the MTU of 9000 with a certain amount of packets and with a certain interval and using a certain vmkernel port

vmkping -I vmk3 -d -s 8972 -c 1000 -i 0.005

vmkping -d -s 1472 <IP_Address>

In one of the above command vmkernel port is vmk3, for MTU 9000, we will be using 8972 as the packet size , -c is the count of packets and -i is the interval for which the ping will work (In the above example it is 0.005 seconds)

The second command is to test the MTU 1500 and the IP to test. You can also add -I (Interface) and vmkernel port through which you want to ping the IP

Command to check the communication of an IP address through an vmkernel port

vmkping -I vmk# IP address of the host

Command to get all the network adapters and the type of tcp/ip stack assigned to the nics

esxcfg-vmknic -l

Using the above command you can check the netstack which will be used in the below command to ping a vmotion vmkernel port

vmkping -S vmotion -I vmk1 <IP_Address_to_ping>

The -S is for netstack name like vmotion and this is the only command to be used if we use a NetStack

List of arguments:

vmkping [args] [host/IP_Address]

args:

  -4                            use IPv4 (default)

  -6                            use IPv6

  -c <count>            set packet count

  -d                           set DF bit (IPv4) or disable fragmentation (IPv6)

  -D                           vmkernel TCP stack debug mode

  -i <interval>           set interval (secs)

  -I <interface>         outgoing interface – for IPv6 scope or IPv4 bypasses routing lookup

  -N <next_hop>       set IP*_NEXTHOP – bypasses routing lookup

                                  for IPv4, -I option is required

  -s <size>                 set the number of ICMP data bytes to be sent.

                                  The default is 56, which translates to a 64 byte

                                  ICMP frame when added to the 8 byte ICMP header.

                                 (Note: these sizes does not include the IP header).

  -t <ttl>                   set IPv4 Time To Live or IPv6 Hop Limit

  -v                            verbose

  -W <timeout>        set timeout to wait if no responses are received (secs)

  -X                            XML output format for esxcli framework.

  -S                           The network stack instance name. If unspecified the default netstack instance is used.

New Product Lifecycle Matrix from VMware

VMware has released a new Product Lifecycle Matrix website so that we can check the validity of all the software from VMware like General Availability, End of General Support, End of Availability etc in one page.

Previously, this was a tedious process to check the end of support cycle for some of the VMware products, but now its all in one place !!

The link is —

https://lifecycle.vmware.com/#/

You can even filter by Product or Filter on any of the columns on the site.

This website can definitely help the Virtual Administrator to check for End of Life/End of mainstream support in their environments.

Visio Diagrams for VMware Validated Design for SDDC 6.0

Looks like the new Visio diagrams and Stencils for VMware SDDC 6.0 are out at communities.vmware.com and created by the author Gary JBlake.

The link to download the diagrams is below:

https://communities.vmware.com/t5/Documents/Visio-Diagrams-for-VMware-Validated-Design-for-SDDC-6-0/ta-p/2782683

This has been referenced from communities.vmware.com forum under VMTN > SDDC > VMware Validated Designs > Validated Designs for SDDC 6.x > Documents

https://communities.vmware.com/t5/Documents/Visio-Diagrams-for-VMware-Validated-Design-for-SDDC-6-0/ta-p/2782683

VMware Cloud Foundation (VCF) API Reference Guides

Here is the direct link to the API Reference Guide for VMware Cloud Foundation (VCF)

https://vdc-download.vmware.com/vmwb-repository/dcr-public/2d4955d7-fb6f-4a61-be78-64d95b951ccd/c6e26ae1-9438-4da0-bfc7-2e21d9046820/index.html#_overview

This is the Generic API Reference Guide for VCF instead of being Version Centric.

For Version Centric API Guides

VCF 3.10 API Reference Guide

VCF 4.0 API Reference Guide

VCF 4.1 API Reference Guide

NOTE: These Reference Guides and their versions are for NON VXRAIL implementations. They are valid for Regular VCF Implementation with VSAN Ready nodes.

VRSLCM 2.1 Deploy Option Greyed Out in VCF 3.10.x

I recently came across an issue where the vRealize Life Cycle Manager 2.1 has the Deploy option greyed out in SDDC Manager in VCF 3.10.x and the issue looks like the screenshot below:

The issue happened as we were using VLAN Backed Network for vRealize products instead of AVN in this version of VCF

The solution is as follows:

Log in to SDDC Manager by using a secure shell (SSH) client, use the account vcf to login into SSH session
Type su to elevate to root and enter the root_password.
Enter the following and press enter.

cd /home/vcf
echo "feature.vrealize.enable.non.avn.deployments=true" >> feature.properties
chown vcf:vcf feature.properties
chmod 644 feature.properties
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

When prompted enter Y to confirm.​​​​​​​ vRealize Suite deployments using SDDC Manager will now be deployed to VLAN backed networks

This is taken from the VMware KB Article https://kb.vmware.com/s/article/79267

Hope this helps if you are using VLAN Backed Network for vRealize Components when deploying a new VCF 3.10.x version in your environment.

VRA 7.6 with VCF 3.10.x SDDC Manager AD Error

I have recently come across an issue in our new VCF 3.10.x build that when we try to deploy the VRA using SDDC Manager, we get an error that the AD Account we have provided can’t validate with the Domain.

The warning is as shown in the picture below:

Note That I had to change a few details and also blur some details from my environment due to privacy reasons.

The Error basically states that VRA is not able to communicate to my domain lab.com with the service account lab\svc_vra_adm because it is trying to contact test.lab.com instead of lab.com Domain

test.lab.com is a DNS Zone in our actual root Domain lab.com and all our VRA Appliances have the host records added to test.lab.com instead of the root domain.

After multiple tries and VMware support, we got to know that VRA (7.x and 8.x) doesn’t support explicit identification of the Active Directory domain name. The kb article which mentions this issue is

https://kb.vmware.com/s/article/59128

The Solution is to make sure that the host records of your VRA is the same as your ‘ActualDomain, in this case lab.com and then retry the validation using the SDDC Manager with the same service account lab\svc_vra_adm

This time, the validation should pass.

Install & Configure VRLCM 2.1 Part-2

Next, We Create a New Environment and then create an New VRA environment using vRLCM

Go to Home and Click on Create Environment to get started

Click on Create Environment
The Default password is used for all the products being deployed using this instance
In this case, we selected the vRA deployment with deployment type as Small for the lab

Agree to the EULA, click Next

Enter the License

Select the NTP Servers and then click Next
Input all the Network Details and click Next

Select the Certificate which we have generated before and click Next

This is where things have gotten tricky in this version as we have multiple options to define the VRA environment including the windows template to create new vms themseleves.

let us go step by step process

Under Product Properties, provide the windows server username and password which you want to access after the box has been provisioned using the windows template.

Scroll down for further options

In the above configuration, We have only 3 VMs being deployed in VRA Simple Configuration.

  • VRA Primary Appliance
  • VRA DB server (Database server)
  • VRA IAAS web server (this contains iaas-web server, iaas manager, iaas DEM Worker and proxy-agent-vsphere )

Once all the Product details of VRA are put in, we will proceed to the precheck phase.

Click on RUN PRECHECK option to continue

Next, we click on Validate & Deploy option to deploy the vms

Make sure you disable UAC in the windows template and then click on Validate & Deploy option to continue.

The Validation process will start
Looks like my test failed with 2 Items, which I will be rectifying before trying to Validate again before Deployment

NOTE: The re-validation took more than 30 mins in my lab to complete. Not sure why it took a lot of time, but I suggest you all to be patient during this process as there is no way to speed it up.

The validation is successful and now we can go ahead and run the PRECHECK to continue

NOTE that at this point, I haven’t installed SQL Software on the SQL Server, but VRSLM has created an windows server for both the db and iaas install. I will have to install SQL Server on the db windows VM and see how it goes.

This Post is pending and I will be updating it soon once I have some clarification on if I need to install and configure the SQL software in the vRA SQL server windows machine or will the scripts do it if I provide the SQL ISO file. Stay Tuned …….

Install & Configure VRLCM 2.1 Part-1

This Blog Post is to Install & Configure vRealize Life Cycle Manager 2.1 in my lab environment.

The reason why I had to install vRLCM 2.1 is to install and configure vRA 7.4/7.5 in my environment.

Deploy the vRLCM OVF file in the lab and the below screenshots will show the configuration after deploying the appliance.

After you login into the vRLCM appliance, the self help starts and below are the screenshots.

Main Page after login using the local user. Click on Start to get started.
Click on Next to configure product binaries
Click Next to continue
Click Next to continue
Click Next to Continue
Click Next to Continue
Click on Got it to complete the Self Help.

Now, Let us change the root password of the appliance from the settings option -> System Administration and click on SAVE to move ahead with the configuration.

Change the root account password and click save to save the appliance credential

Next, we configure NTP Servers and DNS Servers from Servers and Protocol option

Configure the NTP Servers
Configure DNS Servers

Next, we configure the Product Binaries (Where we download the vRA 7.x version using the my vmware account or using the Product Binaries option)

Provide my vmware credentials and then select the product to download

In my case, I have downloaded vRealize Automation 7.6 version to install and configure in my lab.

We will continue with the vRA 7.6 configuration below.

First, Let us configure the Certificates in vRLCM so that the certificate can be used to Deploy the components through vRLCM.

Click on Certificate Management on the left hand side, I will be Generating an CSR as I would like to use my AD CA Signing Authority to generate an Certificate for this instance.

Some of the fields have been blurred for privacy .. Click Generate

Once the CSR is generated, use it to create an Cert and then download the Cert chain which will be in the .p7b format. Use this cert chain to create an pem file.

In my case, I used Cygwin in windows to create an pem file, but with an .cer extension. I had to open the csr file generated which contained the key certificate and then open the generated .cer file by using Cygwin to input the Domain Certificate (in this case, its the vrlcm certificate from the CA, Intermediate CA and Root CA into the Import field and imported it.

NOTE: These are the links which helped me to use Cygwin on my Windows machine to generate the PEM file from existing cert.p7b file

https://www.youtube.com/watch?reload=9&v=3wywzRvfLHo

https://support.delphix.com/Delphix_Virtualization_Engine/Sysadmin/Extracting_PEM_Certificates_from_a_.p7b_Bundle_File_(KBA1859)

Use the Command ” openssl pkcs7 -inform PEM -outform PEM -in certnew.p7b -print_certs > certificate.cer ” after copying the certnew.p7b file into the C:\cygwin64\home\username directory to generate a new .cer file

Next, We create a Data Center

Next, We Add vCenter Server to this Data Center we created

In this case, I had to select the option ‘Consolidated Management and Workload’ as this is a lab environment. In a production environment, you would typically select either an Management or Workload Domain.

Since, This Post was getting too big, I have decided to split it into 2 parts. The Installation of vRA and its configuration is explained in the next part.

The link to Next Part — https://wordpress.com/block-editor/post/hypervc.com/943

Issue with AD Sync in vRA 7.3.1 and 7.4

I have recently come across an issue in our vRA 7.3.1 environment where the AD sync started failing all of a sudden.

The error message looks as in the screenshot below:

AD Sync Error

This error basically means that vRA is not able to communicate with the Active Directory (Lets say my Domain is dallas.com and my vRA appliance hostname is dc1-vcf-vra-01.dallas.com) to update the AD groups and Users for authentication.

The error also means that the vRA is complaining that the connector hostname (in this case it is dc1-vcf-vra-01) doesn’t match the Common Name (CN) in the certificate which is the FQDN (dc1-vcf-vra-01.dallas.com).

Opened a ticket with VMware support and here are the troubleshooting steps recommended so far by them:

1.	     /usr/java/jre-vmware/bin/keytool -v -list -keystore /opt/vmware/horizon/workspace/conf/tcserver.keystore 
                 Check the The Common Name  in the self signed cert. It will be set to node hostname.
2.	     mkdir /root/tmp-bkp
3.	     mv /usr/local/horizon/conf/flags/fips* /root/tmp-bkp		( No file named fips or starting with fips in the flags directory as FIPS is not enabled in our environment)
4.	     /usr/local/horizon/scripts/secure/wizardssl.hzn
                 Install Self Signed Cert and update the keystore
5.	     mv /root/tmp-bkp/fips* /usr/local/horizon/conf/flags		(had to skip it as I was not able to execute the above fips* command)
6.	     service horizon-workspace restart

Will update this post with more steps once VMware support comes back to resolve this issue.

UPDATE

VMware support confirmed that the Common Name (CN) in the self signed Certificate has the FQDN and to follow the steps in the KB article https://kb.vmware.com/s/article/2145268 to check the postgres database for the connector and there we found the issue and rectified it.

From the KB 2145268, I followed the below steps:

Log in to each appliance and type hostname.
If the hostname is shortname and not FQDN, update it from VAMI.

Ensure that the following tables display all the appliances with the FQDN.
Connect to the database by running this command:

su - postgres /opt/vmware/vpostgres/current/bin/psql vcac

Set schema as SaaS by running this command:

set schema 'saas';

Verify the appliances hostnames in the ServiceInstance table by running this command:

select * from "ServiceInstance";

If the hostnames in the table are short, update the hostnames to FQDN by running this command:

update "ServiceInstance" set "hostName"='<new_hostname>' where "id"='<row_id>';

Verify the appliances hostnames in the Connector table by running this command:

select * from "Connector";

If the hostnames in the table are short, update the hostnames to FQDN by running this command:

update "Connector" set "host"='<new_hostname>' where "id"='<row_id>';

I had to substitute new_hostname as the FQDN of my vRA appliance (my case dc1-vcf-vra-01.dallas.com) and the row_id is the ID of the row in which the host name is displayed.

Once I made the modifications in the ‘ServiceInstance’ and ‘Connector’ and restarted the vRA appliance, my AD Sync started to Sync.