VMware ESXi Server Storage Incident

Assistance Request – VMware ESXi / H_VM VM Incident

Hello,

We are requesting your assistance regarding the virtual machine named H_VM , hosted on our VMware ESXi server, an HPE ProLiant DL380 Gen10.
We experienced a critical incident related to the VM storage and a long snapshot chain. Below is a summary of the situation, the actions already performed, and the remaining issue.

Initial Context

The H_VM VM stopped with the following error message:
There is no more space for virtual disk H_VM -000014.vmdk.
You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry.
Although the H_VM VM uses only 160 GB, the main datastore, datastore1, was almost full, with less than 1 GB available.
The VM had a long snapshot chain, including files from H_VM -000001.vmdk through H_VM -000014.vmdk.
Actions Performed to Free Up Space
We first checked the main datastore:
• datastore1
• Capacity: approximately 988.75 GB
• Initial usage: close to 100%
• Affected VM: H_VM , 160 GB
We then added two new SAS disks to the ESXi server:
• 2 × 1.2 TB 10K Enterprise SAS disks
• Form factor: 2.5-inch SFF
• Interface: SAS 12 Gb/s
• Compatible with HPE ProLiant DL380 Gen10
The two disks were detected correctly by ESXi and used to create two new datastores:
• datastore2: approximately 1.09 TB free
• datastore3: approximately 1.09 TB free

Snapshot Removal

We removed old snapshots in order to recover space. From the ESXi interface, we launched:
VM H_VM -> Snapshots -> Manage Snapshots -> Delete All
The operation progressed and released a significant amount of space on datastore1.
After this operation:
• The snapshots are no longer visible in the Snapshot Manager.
• The list only shows: H_VM -> You are here.
• The free space on datastore1 increased to approximately 306 GB.
• However, ESXi then displayed an alert indicating that the VM disks still need to be consolidated.

Current Issue: Consolidation Error

We attempted the following action:
Actions -> Snapshots -> Consolidate disks
However, the operation fails with the following error:
Failed – An error occurred while consolidating disks: 5 Input/output error.
Despite the removal of the visible snapshots, several snapshot/delta files are still present in the VM folder, including:
• H_VM -000001.vmdk
• H_VM -000002.vmdk
• ...
• H_VM -000014.vmdk
The logs also indicate that the VM still appears to be pointing to the latest snapshot disk:
scsi0:0.fileName = "H_VM -000014.vmdk"
This suggests that the snapshot chain may still be active at the disk level, even though the snapshots are no longer visible in the Snapshot Manager.
We have not manually deleted any .vmdk files.

Attempt to Clone the Active Disk

To work around the consolidation failure from the ESXi interface, we attempted to clone the active disk with vmkfstools to datastore2.
Command executed:
vmkfstools -i /vmfs/volumes/datastore1/H_VM /H_VM -000014.vmdk /vmfs/volumes/datastore2/H_VM _consolide/H_VM .vmdk -d thin
The cloning process started, but failed at approximately 13% with the following error:
Failed to clone disk: Input/output error (327689)
This seems to confirm that there is probably a read issue in the VMDK chain or on a block/file related to the snapshots.

Current Status of the VM

The H_VM VM can still boot.
We still have access to the guest operating system, which gives us a window to back up the application and business-critical data.
We have also prevented users from connecting during maintenance, in particular by temporarily disconnecting the local network adapter of the VM on the ESXi side.

Available Technical Information
Environment :
• Server: HPE ProLiant DL380 Gen10
• Hypervisor: VMware ESXi
• VM: H_VM
• Guest OS: Ubuntu Linux 64-bit
• VM CPU: 8 vCPU
• VM RAM: 24 GB
• Displayed virtual disk size: 160 GB
• Main datastore: datastore1
• New available datastores:
• datastore2: approximately 1.09 TB
• datastore3: approximately 1.09 TB
Errors Encountered :
There is no more space for virtual disk 'H_VM -000014.vmdk'
Failed – An error occurred while consolidating disks: 5 Input/output error.
Failed to clone disk: Input/output error (327689)

Logs and Evidence Available:
We are available to provide the following logs and information:
• vmware.log
• vmware-91.log
• hostd.log
• vmkernel.log
• Screenshots of the ESXi interface
• List of files present in the VM folder


Assistance Requested

We need your assistance with the following points:

1. Analyze the current situation of the VM and provide your technical assessment of the best approach to adopt.
2. Determine whether it is still possible to resolve the disk consolidation issue without worsening the situation or compromising the existing data.
3. Propose the safest intervention method to address the consolidation error and bring the VM back to a stable state.
4. Assist us in defining and executing a secure intervention plan aimed at preserving the data, minimizing the risk of data loss, and restoring the VM to a stable and usable state.
5. Define a reliable backup procedure for the future, including ESXi backup best practices, snapshot management, external storage, and periodic restore testing.
6. Assist us in properly backing up the business-critical data from the current VM.
7. Help us secure the VM and avoid any data loss before performing further infrastructure-level operations.
8. Advise us on whether a clean VM reconstruction is required, and if so, define the safest reconstruction approach.

--
Best regards,