vCenter

Workaround instructions to address CVE-2021-44228 in vCenter Server 6.7.x – For VCF 3.10.x

Pradeep AdapaLeave a comment

UPDATE: VMware has Updated the KB 87081 to Include the Script to remove log4j_class

I have taken these Workaround Instructions from the KB article 87081 and KB article 87095

For vCenter 6.7.x appliance in an VCF 3.10.x setup, some of the instructions in article 87081 don’t work and also in VCF 3.10.x since there are external PSC’s and the order to execute the instructions is as follows.

I am calling out VMware team to amend the steps for vCenter 6.7.x appliance in an non-HA configuration in the article 87081, especially for VCF 3.10.x installations.

For vCenter 6.7.x ; Steps to execute

vMON Service

  1. Backup the existing java-wrapper-vmon file

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

  1. Update the java-wrapper-vmon file with a text editor such as vi

vi /usr/lib/vmware-vmon/java-wrapper-vmon

  1. At the very bottom of the file, replace the very last line with 2 new lines
    • Originalexec $java_start_bin $jvm_dynargs “$@”Updated
      log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
      exec $java_start_bin $jvm_dynargs $log4j_arg “$@” 
  2. Restart vCenter Services

service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

  • chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
  • chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

Analytics Service

NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only. vCenter Server Appliance 6.7 Update 3p is by default covered by vMON Service workaround. 

  1. Back up the log4j-core-2.8.2.jar file

cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak

  1. Run the zip command to disable the class

zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  1. Restart the Analytics service

service-control –restart vmware-analytics 

CM Service

  1. Back up the log4j-core.jar file

cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak

  1. Run the zip command to disable the class

zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  1. Restart the CM service

service-control –restart vmware-cm

Run the remove_log4j_class.py script

1. Download the script attached to this KB (remove_log4j_class.py)

2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)

3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work

4. Execute the script copied in step 1:

python remove_log4j_class.py

The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the Analytics Service changes:

grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
 This should return 0 lines

  1. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

The remaining steps for Secure Token Service, Identity Management Service don’t work for vcenter 6.7.x in VCF 3.10.x (3.10.2.1) environment

——– So, after this Step, we will have to SSH into the External PSC and follow the below steps ———-

CM Service

  1. Back up the log4j-core.jar file

cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak

  1. Run the zip command to disable the class

zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  1. Restart the CM service

service-control –restart vmware-cm


Secure Token Service

  1. Back up and edit the the vmware-stsd file

cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bakvi /etc/rc.d/init.d/vmware-stsd

  1. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

start_service()
{
  perform_pre_startup_actions

  local retval
  JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
  $JSVC_BIN -procname $SERVICE_NAME \
            -home $JAVA_HOME \
            -server \
            <snip>
            -Dauditlog.dir=/var/log/audit/sso-events  \
            -Dlog4j2.formatMsgNoLookups=true \
            $DAEMON_CLASS start

  1. Restart the vmware-stsd service

service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

  1. Back up and edit the the vmware-sts-idmd file

cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bakvi /etc/rc.d/init.d/vmware-sts-idmd

  1. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
          -wait 120 \
          -server \
          <snip>
          -Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
          -Dlog4j2.formatMsgNoLookups=true \
          $DEBUG_OPTS \
          $DAEMON_CLASS

  1. Restart the vmware-sts-idmd service

service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

  1. Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

  1. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

The steps in VMware KB Article 87081 is for vCenter with Embedded PSC and the above steps are for the vCenter server 6.7 with an External PSC

Hope this article helps the Engineers who are working on this log4j Vulnerability and if they have VCF 3.10.x you can follow the above steps with an external PSC Configuration.

Advertisement

PSC’s fail to join Domain after Upgrade from vSphere 6.0 U2 to 6.0 U2b or 6.0U3b

Pradeep AdapaLeave a comment

Recently, I have come across an issue with the PSC’s not joining to the domain (They disconnected from the domain automatically) after upgrading the vCenter components (PSC01, PSC02 and vCenter windows server) from 6.0 Update 2 build 3634791 to 6.0 Update 2a build 4632154 or to 6.0 Update 3b build 5326079. This issue occurred as the windows domain controller was 2012 R2 and SMB 2 was the communication protocol to the domain controller. we have to enable SMB 2 on the PSC’s for them to communicate to the domain after the Upgrade.

here is the process to enable SMB2 on the PSC’s —

login to PSC01 and run the following command to check the values

/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]'

check the value of “Smb2Enabled”, this value will be 0, we will need to change it to 1 to enable it.

change SMB2 to be Enabled

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]' Smb2Enabled 1

Once enabled, we need to restart the service lwio

/opt/likewise/bin/lwsm restart lwio

Check the values again —

/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]'

This time the value of “Smb2Enabled” will be 1.

This process is repeated for PSC02 as well and once this is done on both the PSC’s, you can go ahead and add the PSC’s to the domain and the function works !!

The process to change the SMB value on ESXi 6.0 is a little different and here are the commands —

Check values —

/usr/lib/vmware/likewise/bin/lwregshell list_values ‘[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]’

Change SMB2 to be enabled

/usr/lib/vmware/likewise/bin/lwregshell set_value ‘[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]’ SMB2Enabled 1

Restart lwio service

/usr/lib/vmware/likewise/bin/lwsm restart lwio

 

Reset root password on vCenter PSC 6.0 U2

Pradeep AdapaLeave a comment

Resetting the root password on a Platform Services Controller (PSC) 6.0 U2 is taken from the following link —

Link to Reset PSC root Password

The above website clearly mentions on how to use the SUSE Linux Rescue CD to create a new root password and update it in the /etc/shadow file on the PSC itself and after reboot you will be able to get into the PSC with the new password.

Hope this helps !!

Restart the services in VCSA 6.0

Pradeep Adapa2 Comments

Recently I had to restart the web-client service in vcsa 6.0 U1 appliance and found out that the web client service is called differently than in the windows vCenter. the web client service in the vcsa is called vSphere-client

here are the commands to start, stop and restart any services in the vcsa appliance.

To restart a vCenter Server and/or Platform Services Controller service using the command-line:
Log in as root through an SSH or console session on the vCenter Server Appliance.
Run this command to enable the shell:

shell.set --enabled true

Run this command to launch the shell:

shell

Run this command to change directories to /bin:

cd /bin

Run this command to list the vCenter Server Appliance services:

service-control --list

Run this command to stop a specific service:

service-control --stop servicename

You may also stop all services by typing the command:

service-control --stop --all

Run this command to start a specific service:

service-control --start servicename

You may also start all services by typing the command:

service-control --start --all

Here are all the services in the vCenter server appliance –>

vCenter Server Appliance services:

Service Name Description
applmgmt VMware Appliance Management Service
vmware-cis-license VMware License Service
vmware-cm VMware Component Manager
vmware-eam VMware ESX Agent Manager
vmware-sts-idmd VMware Identity Management Service
vmware-invsvc VMware Inventory Service
vmware-mbcs VMware Message Bus Configuration Service
vmware-netdumper VMware vSphere ESXi Dump Collector
vmware-perfcharts VMware Performance Charts
vmware-rbd-watchdog VMware vSphere Auto Deploy Waiter
vmware-rhttpproxy VMware HTTP Reverse Proxy
vmware-sca VMware Service Control Agent
vmware-sps VMware vSphere Profile-Driven Storage Service
vmware-stsd VMware Security Token Service
vmware-syslog VMware Common Logging Service
vmware-syslog-health VMware Syslog Health Service
vmware-vapi-endpoint VMware vAPI Endpoint
vmware-vdcs VMware Content Library Service
vmafdd VMware Authentication Framework
vmcad VMware Certificate Service
vmdird VMware Directory Service
vmware-vpostgres VMware Postgres
vmware-vpx-workflow VMware vCenter Workflow Manager
vmware-vpxd VMware vCenter Server
vmware-vsm VMware vService Manager
vsphere-client vSphere Web Client
vmware-vws VMware System and Hardware Health Manager
vmware-vsan-health VMware Virtual SAN Health Service

Reference: VMware_KB_Article

Upgrade of vCenter 5.5 to vCenter 6.0 with External PSC

Pradeep AdapaLeave a comment

I was pretty surprised to see that there are very few posts on the internet detailing the process of upgrading vCenter 5.5 to vCenter 6.0 with an External PSC. so here is my recent experience on how I did it

Environment:

vCenter server 5.5 (Windows)

vCenter SSO 5.5 (Windows)

SQL Server 2012 (Windows)

  1. Take snapshots of the vCenter server, vCenter SSO server and SQL server
  2. take a backup of the vCenter database in the SQL server
  3. mount the “VMware-VIMSetup-all-6.0.0-2562643.iso” to the Windows vCenter SSO 5.5 server
  4. Go ahead and run the autorun program and run the vCenter setup program
  5. The program will automatically detect the SSO 5.5 in the server and let you know that it will be upgraded to PSC 6.0
  6. Click next to continue to go ahead and upgrade the SSO 5.5 to PSC 6.0

This will complete the upgrade process of SSO 5.5 to PSC 6.0

Once this process is complete, we can go ahead and upgrade the vCenter 5.5 to vCenter 6.0 as the SSO 5.5 is upgraded to PSC 6.0

This concludes the way in which we can upgrade the vCenter 5.5 to 6.0 using external PSC if there is already an SSO 5.5 server available in the environment.