I wanted to ping back one of the great article by one of my fellow vExpert Shank Mohan on his website about an unofficial VCF Troubleshooting guide. I have learned from this article and would like to remember this article and hence posting it back on my blog.
VMware Cloud Foundation (VCF)
VMware Cloud Foundation
LCM Directory Permission Error When pre-checking for SDDC Manager Upgrade with VCF 3.11 Patch
I was getting ready to patch our environment from VCF 3.10.2.2 to VCF 3.11 as VMware has officially released a complete Patch for VCF 3.10.x this month, when I was performing the VCF Upgrade Pre-Check for the Management Domain, I came across this issue
Issue is that the pre-check says that the directory “/var/log/vmare/vcf/lcm/upgrades/<long code directory>/lcmAbout” owner is root but the owner needs to be user vcf_lcm
This is how I resolved the issue:
Login into SDDC Manager as user vcf, do su and provide the root password
then go to the following directory “/var/log/vmware/vcf/lcm/upgrades/<long code directory as displayed in the lcm error on sddc manager>
chown vcf_lcm lcmAbout
chmod 750 lcmAboutThe above two commands will change the owner from root to vcf_lcm and also provide the required permissions to the folder so the pre-check can complete.
The full screenshot of what I performed is below:
Once you perform the commands above, you can run the pre-check and this time it will proceed successfully as shown below
Hope this article helps if you come across this issue with sddc manager upgrade from VCF 3.10.2.2 to 3.11
VCF 3.x patch 3.11 for Log4J Vulnerability and Other Security Patches included
VMware has finally realeased an patch version for VCF 3.x and the version is 3.11. You can only download this as a patch form from the SDDC Manager. You can Upgrade to version 3.11 from 30.10.2.2 or VCF 3.5 or later.
This Release VCF 3.11 includes the following:
- Security fixes for Apache Log4j Remote Code Execution Vulnerability: This release fixes CVE-2021-44228 and CVE-2021-45046. See VMSA-2021-0028.
- Security fixes for Apache HTTP Server: This release fixes CVE-2021-40438. See CVE-2021-40438.
- Improvements to upgrade prechecks: Upgrade prechecks have been expanded to verify filesystem capacity, file permissions, and passwords. These improved prechecks help identify issues that you need to resolve to ensure a smooth upgrade.
- This also resolves the following Security Advisory VMSA-2022-0004 which deals with several vulnerabilities in esxi 6.7 hosts
- This also resolves the vulnerability in VCF SDDC Manager 3.x according to the security advisory VMSA-2022-0003
- This version also addresses the heap-overflow vulnerability in esxi hosts according to the security advisory VMSA-2022-0001.2
The Updated product versions according to the BOM for VCF 3.11 are
Hope this post helps for the teams who have VCF 3.10.x and waiting for the long awaited log4j patch instead of an workaround.
Workaround instructions to address CVE-2021-44228 in vCenter Server 6.7.x – For VCF 3.10.x
UPDATE: VMware has Updated the KB 87081 to Include the Script to remove log4j_class
I have taken these Workaround Instructions from the KB article 87081 and KB article 87095
For vCenter 6.7.x appliance in an VCF 3.10.x setup, some of the instructions in article 87081 don’t work and also in VCF 3.10.x since there are external PSC’s and the order to execute the instructions is as follows.
I am calling out VMware team to amend the steps for vCenter 6.7.x appliance in an non-HA configuration in the article 87081, especially for VCF 3.10.x installations.
For vCenter 6.7.x ; Steps to execute
vMON Service
- Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
- Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
- At the very bottom of the file, replace the very last line with 2 new lines
- Originalexec $java_start_bin $jvm_dynargs “$@”Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
- Originalexec $java_start_bin $jvm_dynargs “$@”Updated
- Restart vCenter Services
service-control –stop –all
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Analytics Service
NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only. vCenter Server Appliance 6.7 Update 3p is by default covered by vMON Service workaround.
- Back up the log4j-core-2.8.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the Analytics service
service-control –restart vmware-analytics
CM Service
- Back up the log4j-core.jar file
cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the CM service
service-control –restart vmware-cm
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)
2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)
3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work
4. Execute the script copied in step 1:
python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
This should return 0 lines
- Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
This should return 0 lines
The remaining steps for Secure Token Service, Identity Management Service don’t work for vcenter 6.7.x in VCF 3.10.x (3.10.2.1) environment
——– So, after this Step, we will have to SSH into the External PSC and follow the below steps ———-
CM Service
- Back up the log4j-core.jar file
cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak
- Run the zip command to disable the class
zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
- Restart the CM service
service-control –restart vmware-cm
Secure Token Service
- Back up and edit the the vmware-stsd file
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bakvi /etc/rc.d/init.d/vmware-stsd
- Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actions
local retval
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
- Restart the vmware-stsd service
service-control –stop vmware-stsd
service-control –start vmware-stsd
Identity Management Service
- Back up and edit the the vmware-sts-idmd file
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bakvi /etc/rc.d/init.d/vmware-sts-idmd
- Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
- Restart the vmware-sts-idmd service
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
- Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
- Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
This should return 0 lines
The steps in VMware KB Article 87081 is for vCenter with Embedded PSC and the above steps are for the vCenter server 6.7 with an External PSC
Hope this article helps the Engineers who are working on this log4j Vulnerability and if they have VCF 3.10.x you can follow the above steps with an external PSC Configuration.
Visio Diagrams for VMware Validated Design for SDDC 6.0
Looks like the new Visio diagrams and Stencils for VMware SDDC 6.0 are out at communities.vmware.com and created by the author Gary JBlake.
The link to download the diagrams is below:
This has been referenced from communities.vmware.com forum under VMTN > SDDC > VMware Validated Designs > Validated Designs for SDDC 6.x > Documents
https://communities.vmware.com/t5/Documents/Visio-Diagrams-for-VMware-Validated-Design-for-SDDC-6-0/ta-p/2782683
VMware Cloud Foundation (VCF) API Reference Guides
Here is the direct link to the API Reference Guide for VMware Cloud Foundation (VCF)
This is the Generic API Reference Guide for VCF instead of being Version Centric.
Updated – 2/16/2022
From Version VCF 4.3.1 (Non-VxRail), VMware has moved their API Guides to a new location which is a better format than before and more user friendly.
VCF 4.3.1 New API Reference Guide
VCF 4.4.0 New API Reference Guide
Updated – 1/25/2022
For Version Centric API Guides
NOTE: These Reference Guides and their versions are for NON VXRAIL implementations. They are valid for Regular VCF Implementation with VSAN Ready nodes.
VMware Cloud Foundation Upgrade
Hello All,
This post shows how to Upgrade VMware Cloud Foundation v2.2.x to 2.2.x.
NOTE: This is a pretty old version of VMware Cloud Foundation we have in our Test environment as we are preparing it for an upgrade before we do it on our production environment.
The current version of VMware Cloud foundation is 3.5 (As of 03/11/2019)
First, we will login into the SDDC Manager Web UI
The above screen is the main page once you login into SDDC Manager
On the left hand side, you need to click on Life Cycle Manager to check for your upgrade options and to actually perform the upgrade of all the components in the SDDC.
Select the Upgrade available and you want to go to and click “UPDATE”
Select the Target (In this case its the DataCenter Work Domain) and click Next
Select the Date and Time you want the schedule to start and click Next
Select the option “SCHEDULE UPDATE” to schedule the update of all the components.
In the next window, you can see the update progress. You can expand the Update option to see the actual status of the update.
After clicking on View Update Details option, you expand on the next screen to see the hosts/objects you are upgrading using the SDDC Manager (In this case I was upgrading the esxi hosts)
Final Screenshot shows that the upgrade is complete. It generally takes a few hours depending on how many nodes are present in the cluster.
This concludes on how to Upgrade VMware Cloud Foundation. More posts to follow soon.
Converged, Hyperconverged Infrastructure & Virtualization 