Organizations use VMware virtual machines (VMs) to optimize hardware utilization and improve operational efficiency, so any degradation in VM performance is unwanted. In this post, we will explain how to fix common problems and improve VMware VM performance.
Ensure complete data protection for VMware environments with NAKIVO Backup & Replication. Incremental backups, instant recovery, secure backup targets, anti-ransomware features and IT monitoring capabilities.
When VMware Performance Requires Improvement
The main reasons to diagnose your VMware VMs and try to improve their performance are:
- A VM’s performance is reduced compared to that of an identical machine running on another host.
- Apps are running slower than intended. The apps’ performance is okay when a VM is moved to a different host.
- Apps take too long to launch.
- Applications suddenly stop responding or fail to launch.
- CPU, RAM or disk load reaches 100% and stays at that level.
VMware Performance Issues and How to Fix them
VMware VM performance may decrease due to either software or hardware problems. Here, we will review VM performance problems connected to CPU and RAM, storage, snapshots, network and software. The explanation on solving those problems will be provided as well.
CPU and RAM
One of the most common reasons for VMware VM performance to degrade is the lack of hardware resources. If a VM does not have enough CPU resources, then that VM is unable to run apps smoothly. To solve that issue, enable the VM to utilize additional CPU cores or more virtual processors. Additionally, if your processor has fewer than 4 cores, not enough cache, and low clock speed, consider the hardware upgrade for your host.
The same goes regarding the amount of RAM allocated to the VM. However, when setting RAM for a VM, don’t forget to leave enough memory resources for the host to operate and run the VM environment. If a VM does not have enough RAM, the guest operating system (OS) has to utilize a swap file intensively, which leads to overall VM performance reduction. With a lot of RAM dedicated to a VM, the host OS is forced to use a swap, which causes major problems with the performance of both the host OS and all the VMs running on that host. To avoid all this, don’t provide VMs with huge RAM pools and make sure to maintain a proper balance between RAM volumes for VMs and the host.
When the total RAM allocated to VMs exceeds the actual physical hardware capacity, a memory overcommitment situation arises. For example, if your host has 16 GB of RAM and runs 6 VMs that are configured to use 3 GB of RAM per machine, the memory is overcommitted. If overcommitment happens, the performance of the host and VMs can decrease when all VMs need to use memory intensively. When some VMs are lightly loaded while running, ESXi can improve memory usage by allocating more RAM to the VMs that are heavily loaded.
The most appropriate solution is to entirely prevent host and guest OSs from using swap files. In case you don’t have enough RAM to provide the required level of production performance, you might want to consider upgrading the host hardware. Also, make sure that the memory overcommitment is not the case.
Sometimes, a memory ballooning feature can cause VM performance to degrade if a VM uses a swap file. Make sure to check the feature and configuration to reclaim the unused memory from VMs.
A CPU overcommitment issue is another reason for VMware VM performance to drop off. With VMware, you can use hardware resources rationally and provide VMs with more virtual processors than the number of cores that your physical host CPU has. The normal ratio between the number of virtual processors assigned to VMs and the number of physical host CPU cores is 3:1. When that ratio hits 5:1, the performance of VMs may decrease. At 6:1, many production troubles can appear due to the drastic performance degradation of the VMs.
Sometimes, CPU power saving and energy efficiency features can drag VM performance down. Check the BIOS and try disabling those features. Also, check the sufficiency of CPU and memory on the host.
Storage
Storage system performance is critical to ensure the high performance of the VMs storing virtual disks on that system. VM performance heavily depends on storage latency. Using a Solid State Drive (SSD) is recommended when storing VMs. When it is not possible to use SSD disks, 7200 RPM or 10000 RPM HDDs (Hard Disk Drives) should be used. A 5400 RPM HDD is too slow for VMs to function properly. Disks with SAS interfaces are preferred in VM environments.
Those who use VMware Workstation can store virtual machines on external USB 3.0 or SATA drives. The disk type and speed recommendations remain the same for external drives. Be advised that disconnecting an external drive without powering off the VM first may cause data loss.
Preallocated or non-growable disks are preferred for VMs in production environments. Eager-zeroed thick provisioned disks perform first-write operations faster.
For VMs stored on an HDD, defragmenting both the physical drive and virtual disk can help to improve VM performance. To reduce disk fragmentation, use partitions. Install and run an OS on one partition, and store app files (databases, for instance) on another partition.
Check the relevance of the Host Bus Adapter (HBA) firmware. If necessary, update the HBA firmware on your server hardware.
Regularly monitor the health of your disks and file systems, and inspect the interface cables (SATA, SAS). A corrupted disk or a mechanically damaged cable should be replaced without any delay.
Overheads caused by disk encryption can reduce VM performance. If data encryption is not critical for a VM, move that VM to non-encrypted storage.
Use independent hardware RAID controllers for more reliability and performance boost.
Check if enough free space is available on the virtual disk and allocate more space if necessary. Consider moving the VM to another disk or array if the current storage is overloaded.
Snapshots
Virtual disk snapshots reduce virtual disk performance. A new delta VMDK file of a virtual disk is required for each snapshot. Each data VMDK contains the change log with data differences from a parent VMDK file. The VM reads that data when interacting with the VM disk. That means the VM reads data from multiple files on the same physical disk simultaneously, and thus VM performance decreases.
Avoid using split VMDK files for VMs in VMware Workstation, as that is only an option added to use on systems not supporting files exceeding 4 GB in size. Also, remember that as the quantity of snapshots grows, the load put on a disk by the VM increases. Whenever possible, commit or delete snapshots, and use them only for temporary purposes. For example, a snapshot is perfect for when you copy VM data for backups or run an internal app test.
VMware Tools
VMware Tools is a set of utilities and drivers used to improve user experience and boost VM performance. Therefore, the guest OS should have VMware Tools installed to increase overall graphics performance, synchronize a mouse, use VMs without lags in a fullscreen mode, among other improvements.
To check if VMware tools are installed in Windows, find the VMware Tools Icon in the Tray, or go to C:\Program Files\VMware\VMware Tools\ (this is a default directory) and run the command:
VMwareToolboxCmd.exe -v
In Linux, a single console command is enough:
vmware-toolbox-cmd -v
With VMware vSphere Client, you can check if VMware Tools are installed on the guest OS directly in the VM’s Summary tab.
Network
With an ESXi host connected to a Storage Area Network (SAN) or Network Attached Storage (NAS), the network overload or network speed deficiency can reduce VM performance. For high performance, separate the networks in your environment:
- vMotion network
- ESXi management network
- Storage network
In case your current network does not meet the needs of your VM environment, you might want to think about upgrading. For instance, consider installing equipment such as 5-gigabit or 10-gigabit adapters and routers to replace current 1-gigabit and 2,5 gigabit hardware.
Antivirus Software
The host’s antivirus scanning virtual disk files can reduce VM performance. To solve the issue, exclude VMDK files in VM directories from the antivirus scanning list.
Also, keep in mind that viruses, ransomware, and other malware can significantly reduce the performance of hosts and VMs. Use an antivirus solution to monitor the health of the host OS and guest OS and to deal with malicious agents if they infiltrate your IT environment.
Hyper-V Troubles
VMware VM performance can be reduced when you use VMware Workstation versions 15.5 or newer, which support running VMware VMs when the Hyper-V feature is installed on a Windows host. Hyper-V then uses Intel VT-x and AMD-V hardware virtualization features directly. For VMware Workstation with a Virtual Machine Monitor (VMM) that must run in privileged mode, those features are not available when Hyper-V is installed. VMM runs in the user mode instead to ensure compatibility, and cannot utilize the CPU’s hardware virtualization features.
To enable VMM to run in privileged mode and work directly with CPU virtualization, uninstall Hyper-V and features such as Virtualization Based Security from the Windows host. This uninstall should fix the VMware VM slow performance issue.
Performance Monitoring: Avoid Bottlenecks
By regularly monitoring the performance of your VMs, you can solve issues proactively and maintain production stability. With VM monitoring, troubleshooting the exact cause of a performance problem can be done quickly. However, the key is to monitor VMs on the host level instead of a guest OS. Guest operating systems might show incorrect data as they are not virtualization-aware.
Use VMware vSphere client to monitor hardware running VMs on the host. View the Monitor tab, then go to Performance > Overview, or Performance > Advanced to see graphs and more. The Overview tab shows the real-time and historical graphs of CPU, RAM, network and storage use. Navigate through tabs to see the monitoring data by VMs or hosts.
Summary
In this post, we covered the main reasons for VMware VM performance degradation and quick ways to troubleshoot them. To ensure the maximum performance of your VMs and host, you need hardware with an appropriate performance envelope, balanced RAM distribution, and proper software configuration.
This is the basic guide. Read more in our blog to find in-depth instructions on solving VMware VM performance issues.
Download the 15-day Free Trial of NAKIVO Backup & Replication to explore the solution’s data protection capabilities including incremental backups, instant recovery, ransomware protection and more. No capacity limitations and no credit card required.
