vSphere 6.0

All Products under vSphere 6.0

Workaround instructions to address CVE-2021-44228 in vCenter Server 6.7.x – For VCF 3.10.x

Pradeep AdapaLeave a comment

UPDATE: VMware has Updated the KB 87081 to Include the Script to remove log4j_class

I have taken these Workaround Instructions from the KB article 87081 and KB article 87095

For vCenter 6.7.x appliance in an VCF 3.10.x setup, some of the instructions in article 87081 don’t work and also in VCF 3.10.x since there are external PSC’s and the order to execute the instructions is as follows.

I am calling out VMware team to amend the steps for vCenter 6.7.x appliance in an non-HA configuration in the article 87081, especially for VCF 3.10.x installations.

For vCenter 6.7.x ; Steps to execute

vMON Service

  1. Backup the existing java-wrapper-vmon file

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

  1. Update the java-wrapper-vmon file with a text editor such as vi

vi /usr/lib/vmware-vmon/java-wrapper-vmon

  1. At the very bottom of the file, replace the very last line with 2 new lines
    • Originalexec $java_start_bin $jvm_dynargs “$@”Updated
      log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
      exec $java_start_bin $jvm_dynargs $log4j_arg “$@” 
  2. Restart vCenter Services

service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

  • chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
  • chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

Analytics Service

NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only. vCenter Server Appliance 6.7 Update 3p is by default covered by vMON Service workaround. 

  1. Back up the log4j-core-2.8.2.jar file

cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak

  1. Run the zip command to disable the class

zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  1. Restart the Analytics service

service-control –restart vmware-analytics 

CM Service

  1. Back up the log4j-core.jar file

cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak

  1. Run the zip command to disable the class

zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  1. Restart the CM service

service-control –restart vmware-cm

Run the remove_log4j_class.py script

1. Download the script attached to this KB (remove_log4j_class.py)

2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)

3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work

4. Execute the script copied in step 1:

python remove_log4j_class.py

The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the Analytics Service changes:

grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
 This should return 0 lines

  1. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

The remaining steps for Secure Token Service, Identity Management Service don’t work for vcenter 6.7.x in VCF 3.10.x (3.10.2.1) environment

——– So, after this Step, we will have to SSH into the External PSC and follow the below steps ———-

CM Service

  1. Back up the log4j-core.jar file

cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak

  1. Run the zip command to disable the class

zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  1. Restart the CM service

service-control –restart vmware-cm


Secure Token Service

  1. Back up and edit the the vmware-stsd file

cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bakvi /etc/rc.d/init.d/vmware-stsd

  1. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

start_service()
{
  perform_pre_startup_actions

  local retval
  JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
  $JSVC_BIN -procname $SERVICE_NAME \
            -home $JAVA_HOME \
            -server \
            <snip>
            -Dauditlog.dir=/var/log/audit/sso-events  \
            -Dlog4j2.formatMsgNoLookups=true \
            $DAEMON_CLASS start

  1. Restart the vmware-stsd service

service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

  1. Back up and edit the the vmware-sts-idmd file

cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bakvi /etc/rc.d/init.d/vmware-sts-idmd

  1. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
          -wait 120 \
          -server \
          <snip>
          -Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
          -Dlog4j2.formatMsgNoLookups=true \
          $DEBUG_OPTS \
          $DAEMON_CLASS

  1. Restart the vmware-sts-idmd service

service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

The steps in VMware KB Article 87081 is for vCenter with Embedded PSC and the above steps are for the vCenter server 6.7 with an External PSC

Hope this article helps the Engineers who are working on this log4j Vulnerability and if they have VCF 3.10.x you can follow the above steps with an external PSC Configuration.

Advertisement

Reset Root Password for vRA 7.x Appliance

Pradeep AdapaLeave a comment

Recently I had to reset the root password to my vRA 7.3 appliance and I had to follow the VMware kb article https://kb.vmware.com/s/article/2150647

Only thing I would change in the kb article instructions would be the 6th step.

6th step — look for the work ‘vmlinuz’ in the second line and then hit e on that line to edit the line, then add the command init=/bin/bash to the end of the line and hit Enter

7th step — hit b to boot from that line to get to the root prompt

10th step — After you reset the password and it says that the new password has been accepted, type reboot to reboot the appliance

Hope this helps!

Add External PSC 6.0 U2 to Active Directory Domain

Pradeep AdapaLeave a comment

I have been thinking of writing this post for a while and here you go…

In vSphere 6.0 U2, you can have an External PSC or an Embedded PSC. The below process is to add an External PSC to the Active Directory Domain.

Login into the vCenter server, go to Administration tab, go to System Configuration –> Nodes and click on the PSC node you want to add to the domain.

psc01_Domain_setting
Click on the node, go to Manage tab, click on Active Directory and click on Join button

psc01_domain_join01
Provide the Domain name and its credentials (It could be a read-only service account in the domain) to join the PSC to the domain

psc01_domain_join02

once credentials are provided, click OK to proceed.

Note that the only way for you to know that this process is complete is that you get no error and there is no entry in the recent tasks tab in the vSphere web client. If that is the case then the domain add is successful.

Now, you will need to reboot the PSC

psc01_domain_after_psc01_reboot
Once the PSC is rebooted, you can go back into the vCenter server and to the node and you can see that this node (PSC) is now added to the domain.

In a similar way, you can add the remaining PSC’s to the domain and finally, you will need to add the Identity source to the vCenter server itself under single sign-on

 

SSO Server test failed when Configuring VDP 6.1.8 during vCenter Registration

Pradeep AdapaLeave a comment

Recently, I came across an issue while configuring a new instance of VDP 6.1.8 appliance while performing vCenter Registration to the vCenter appliance 6.5 with an external Platform Services Controller.

below is the error message I have been getting

vdp_error_vcenter_registration

I have provided the administrator account user credentials to the VCSA (vCenter server) with the default ports but still received the error.

Upon some deep troubleshooting, found out that the SSO server is the Platform Services Controller (PSC) since my environment had an external PSC and here is how you resolve this issue:

De-select the checkbox “Use vCenter for SSO authentication”, and add the Platform Services Controller hostname/IP in the new SSO entry line.

vdp_vcenter_registration_01

Now, you can test the connection and it will be a success

vdp_vcenter_registration_02

This is how the issue was resolved. Hope it helps someone out there.

This is with VDP version 6.1.8 connecting to VCSA 6.5 with External PSC

PSC’s fail to join Domain after Upgrade from vSphere 6.0 U2 to 6.0 U2b or 6.0U3b

Pradeep AdapaLeave a comment

Recently, I have come across an issue with the PSC’s not joining to the domain (They disconnected from the domain automatically) after upgrading the vCenter components (PSC01, PSC02 and vCenter windows server) from 6.0 Update 2 build 3634791 to 6.0 Update 2a build 4632154 or to 6.0 Update 3b build 5326079. This issue occurred as the windows domain controller was 2012 R2 and SMB 2 was the communication protocol to the domain controller. we have to enable SMB 2 on the PSC’s for them to communicate to the domain after the Upgrade.

here is the process to enable SMB2 on the PSC’s —

login to PSC01 and run the following command to check the values

/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]'

check the value of “Smb2Enabled”, this value will be 0, we will need to change it to 1 to enable it.

change SMB2 to be Enabled

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]' Smb2Enabled 1

Once enabled, we need to restart the service lwio

/opt/likewise/bin/lwsm restart lwio

Check the values again —

/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]'

This time the value of “Smb2Enabled” will be 1.

This process is repeated for PSC02 as well and once this is done on both the PSC’s, you can go ahead and add the PSC’s to the domain and the function works !!

The process to change the SMB value on ESXi 6.0 is a little different and here are the commands —

Check values —

/usr/lib/vmware/likewise/bin/lwregshell list_values ‘[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]’

Change SMB2 to be enabled

/usr/lib/vmware/likewise/bin/lwregshell set_value ‘[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]’ SMB2Enabled 1

Restart lwio service

/usr/lib/vmware/likewise/bin/lwsm restart lwio

 

Reset root password on vCenter PSC 6.0 U2

Pradeep AdapaLeave a comment

Resetting the root password on a Platform Services Controller (PSC) 6.0 U2 is taken from the following link —

Link to Reset PSC root Password

The above website clearly mentions on how to use the SUSE Linux Rescue CD to create a new root password and update it in the /etc/shadow file on the PSC itself and after reboot you will be able to get into the PSC with the new password.

Hope this helps !!

Disable vSphere Managed Object Browser (MOB)

Pradeep AdapaLeave a comment

To harden your ESXi 6.0 hosts, we disable the MOB service so that any attacker can’t get to the web browser and access the MOB of the ESXi host (ex: https://esxi01.lab.com/mob), this setting will disable one of the attack vectors of theESXi hosts in the environment.

to do this, you SSH into the ESXi host where you want to disable the mob service and perform the following commands

esxi01# vim-cmd proxysvc/remove_service "/mob" "httpsWithRedirect"

to verify if the mob service has been removed from the ESXi host, use the following command

esxi01# vim-cmd proxysvc/service_list

the above command will list all the services on the ESXi host, look for the service “/mob”, if you don’t see this service, then it has been removed. if it is still there, then you will have to perform the first command and reboot the ESXi host to disable the mob service from the host.

 

 

Can’t fork error on ESi 6 host on UCS Blade

Pradeep AdapaLeave a comment

Recently, I was working on an UCS blade firmware upgrade along with esxi upgrade from esxi 5.5 to 6.0 and came across this error where the esxi host became unresponsive with an error “can’t fork” on its DCUI.

here is a little background on this story, this particular blade was B240 blade which was being used as SAP HANA blade by the customer and the firmware upgrade and esxi upgrade went fine and two days later the host became unresponsive and we couldn’t connect to it using SSH, DCUI, etc, connecting to the kvm console revealed the below screen when we went to its Alt+F1 command interface

Esxi_Cant_fork_error

we had to bounce the box and we had to reduce the linux vm memory which was hosting SAP HANA on it to be 10% less than the memory of the esxi host.

Conclusion: The HANA VM (linux) on the esxi host should have 10% less memory than the overall memory of the esxi host to avoid this problem.

How to Install Cisco VEM vib on an ESXi 6 host

Pradeep AdapaLeave a comment

Recently, I had to install the Cisco vem module onto an esxi 6 host as it was not installed and i couldn’t join the esxi host to the cisco nexus 1000v distributed switch. here is the process on how to first check if the vem module is installed on the esxi host.

SSH into the esxi host and run the following commands to check if the VEM module is installed

host# esxcli software vib list | grep -i vem

the above command will display the cisco vem module installed on the esxi host, if nothing is displayed then you will have to install the vem module by downloading the vem vib from the nexus 1kv in the environment.

i did it by going to https://nexus1kv_hostname    in a web browser which will display you the vibs which you can download from nexus 1000v, download the vem vib associated with your environment and run the following command to install the vib onto the esxi host

upload the vem vib file onto a datastore on the host

SSH into the esxi host where you want to install the vem module

host# esxcli software vib install -v /vmfs/volumes/<directory_path>/cross_cisco-vem-version_x.x.x.x.x.vib

NOTE: directory_path in the above command is the place where the cisco vem vib is stored. (name of the datastore/volume)

Once the vib is installed you can check the status of the vem by using the command

host# vem status

The above command will display that the VEM agent (vemdpa) is running

Restart the services in VCSA 6.0

Pradeep Adapa2 Comments

Recently I had to restart the web-client service in vcsa 6.0 U1 appliance and found out that the web client service is called differently than in the windows vCenter. the web client service in the vcsa is called vSphere-client

here are the commands to start, stop and restart any services in the vcsa appliance.

To restart a vCenter Server and/or Platform Services Controller service using the command-line:
Log in as root through an SSH or console session on the vCenter Server Appliance.
Run this command to enable the shell:

shell.set --enabled true

Run this command to launch the shell:

shell

Run this command to change directories to /bin:

cd /bin

Run this command to list the vCenter Server Appliance services:

service-control --list

Run this command to stop a specific service:

service-control --stop servicename

You may also stop all services by typing the command:

service-control --stop --all

Run this command to start a specific service:

service-control --start servicename

You may also start all services by typing the command:

service-control --start --all

Here are all the services in the vCenter server appliance –>

vCenter Server Appliance services:

Service Name Description
applmgmt VMware Appliance Management Service
vmware-cis-license VMware License Service
vmware-cm VMware Component Manager
vmware-eam VMware ESX Agent Manager
vmware-sts-idmd VMware Identity Management Service
vmware-invsvc VMware Inventory Service
vmware-mbcs VMware Message Bus Configuration Service
vmware-netdumper VMware vSphere ESXi Dump Collector
vmware-perfcharts VMware Performance Charts
vmware-rbd-watchdog VMware vSphere Auto Deploy Waiter
vmware-rhttpproxy VMware HTTP Reverse Proxy
vmware-sca VMware Service Control Agent
vmware-sps VMware vSphere Profile-Driven Storage Service
vmware-stsd VMware Security Token Service
vmware-syslog VMware Common Logging Service
vmware-syslog-health VMware Syslog Health Service
vmware-vapi-endpoint VMware vAPI Endpoint
vmware-vdcs VMware Content Library Service
vmafdd VMware Authentication Framework
vmcad VMware Certificate Service
vmdird VMware Directory Service
vmware-vpostgres VMware Postgres
vmware-vpx-workflow VMware vCenter Workflow Manager
vmware-vpxd VMware vCenter Server
vmware-vsm VMware vService Manager
vsphere-client vSphere Web Client
vmware-vws VMware System and Hardware Health Manager
vmware-vsan-health VMware Virtual SAN Health Service

Reference: VMware_KB_Article