VxVerify – Pre-upgrade Health Check Tool
- David Ring
- Apr 14, 2021
- 3 min read
VxVerify is an incredibly useful tool to have at your disposal while working with VxRail. Complimentary to the native VxRail manager checks VxVerify performs a health check analysis on a VxRail cluster and it is highly recommended to run VxVerify in advance of upgrades, expansions or for general maintenance operations. VxVerify when run on VxRail Manager deploys what is referred to as ‘Minions’ (small python programs) which are sent to each of the VxRail nodes in the cluster, these minions in turn perform host checks on each of the nodes. In addition to ESXi host specific tests VxVerify also performs checks on VxRail Manager, VMs, vCSA and at a cluster level. Here are just a sample of what gets tested:
Maintenance mode status
Check hostd service
IPMI tool for VxRail hardware
NTP status and time delta
Secure Boot status
vSAN Health
Service Datastore available capacity
ESXi version consistency
vSAN Health
Free space
Password for unsupported characters
Check free space in root and /tmp
Host Lockdown Mode
Reboot required for ESXi host
Check for RecoverPoint
Check for external vCenter
vMotion compatibility
Mounted ISO on VM
VM-to-host affinity rules
Docker Servers Running
VCF installation type, if present
Check time difference between VxRM & VC
It is highly recommended that you run the VxVerify tool before proceeding with a VxRail upgrade; more details on how to download and run VxVerify can be found here:
There is a VxVerify download available for each of the following VxRail versions:
VxVerify 1.xx.xxx is for VxRail 4.0 only
VxVerify 2.xx.xxx is for VxRail 4.5, 4.7 & 7.0.000
VxVerify 3.xx.xxx is for VxRail 7.0.010+
Note: Always download the very latest edition as new checks are being added regularly. In addition each version of VxVerify is timelimited to prevent outdated versions.
Running VxVerify
In this example VxRail is running code version 7.0.100 so we begin by downloading the VxVerify 3.x edition:
Once download completes extract the .zip and review the readme.txt to review all the recent updates:
Before uploading VxVerify to VxRail Manager we need to create a directory with the required permission set. Login to VxRail manager via SSH elevating to root user and execute the following set of commands to create a ‘vxv’ dir in /tmp and set the permissions:
Upload the ‘.pyc’ file to the newly created directory (in this example I am leveraging ‘winscp’ to perform the upload to VxRail manager):
Once uploaded navigate to /tmp/vxv/ and use --help to expose all the VxVerify associated command line arguments:
Simply running the .pyc script as root will kick off VxVerify:
As can be seen from the results table below; all hosts and cluster tests completed successfully. You can also see the status 3 critical alert notifying that ‘VC login failed’ the VC management credentials get decrypted automatically from the VxRail Manager database and runtime (although they can also be specified with -u & -p), so if those are failing, you’ll need to check the vxv.log
Inserting the vCenter root credentials performs the respective vCSA tests such as free space tests for the VCSA partitions:
Results table below now inclusive of vCenter health checks:
VxVerify creates a number of files in ‘/tmp/vxv/’ these include .log files which can be used for troubleshooting purposes etc. also the summary report seen on the previous screen can be recalled by viewing the ‘/tmp/vxv/vxverify.txt’ file:
Another really useful output here is the ‘vxtii.txt’ file which includes detailed information on each ESXi host in the VxRail cluster providing a nice overview of the hardware:
Creating a dummy failure by attaching an .iso to the vCenter VM using local storage results in the following failure scenario. You will note for each warning or failure there is an associated support.dell.com KB referenced in the results table which will assist with resolving any issue highlighted by VxVerify (do not attempt an upgrade until these yellow or red event codes are investigated):
Special mention to our Escalation Engineering team for maintaining such a valuable tool! Thank you!
Comments