help recovering inaccesible lvm thin volumes
Budget: $10 – $30 USD
Need lvm expert to bring back to life a Proxmox Thin-Pool
My ProxMox VMs was not starting afer a power outage. So I ran:
# lvconvert --repair pve/data
The command finished well, without errors, but a lot more faster as usual. Then I deleted the "data_meta0" generated by the previous command. I think this action made the problem bigger. I didn't know what i was really doing, just following a guide.
# lvremove /dev/pve/data_meta0
I executed the reapair again, to get a data_meta0 again:
# lvconvert --repair pve/data
Tried to enable one vm disk, but all 4 vm disks was failing with the same error:
# lvchange -ay pve/vm-100-disk-1
device-mapper: reload ioctl on (253:7) failed: No data available
Tried creating another volume to repair meta into that vol and swapped it to pool meta:
lvcreate -an -Zn -L4G --name repaired_01 pve
./thin_repair -i /dev/mapper/pve-data-meta0 -o /dev/mapper/pve-repaired_01
lvchange -an pve/data
lvconvert --thinpool pve/data --poolmetadata /dev/mapper/pve-repaired_01
Then restarted the computer and noticed that pve/data was inactive too now. So I tested again, getting with a new error (all 4 vm disks and pve/data volumes was failing with the same different error):
# lvchange -ay pve/data
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
# lvchange -ay pve
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Googled about that and find a possible solution (backing up and restoring vg):
# vgcfgbackup pve -f /root/pvebak
Volume group "pve" successfully backed up.
# nano /root/pvebak
(modified transaction_id, changin 7 for 0 on pve/data)
# vgcfgrestore pve --force -f /root/pvebak
Restored volume group pve.
Then tried to activate volumes again, but getting same results as the beggining, now pve/data gets activated, but 4 vms not and showing the first error message again
# lvchange -ay pve
device-mapper: reload ioctl on (253:5) failed: No data available
device-mapper: reload ioctl on (253:5) failed: No data available
device-mapper: reload ioctl on (253:5) failed: No data available
device-mapper: reload ioctl on (253:5) failed: No data available
My ProxMox VMs was not starting afer a power outage. So I ran:
# lvconvert --repair pve/data
The command finished well, without errors, but a lot more faster as usual. Then I deleted the "data_meta0" generated by the previous command. I think this action made the problem bigger. I didn't know what i was really doing, just following a guide.
# lvremove /dev/pve/data_meta0
I executed the reapair again, to get a data_meta0 again:
# lvconvert --repair pve/data
Tried to enable one vm disk, but all 4 vm disks was failing with the same error:
# lvchange -ay pve/vm-100-disk-1
device-mapper: reload ioctl on (253:7) failed: No data available
Tried creating another volume to repair meta into that vol and swapped it to pool meta:
lvcreate -an -Zn -L4G --name repaired_01 pve
./thin_repair -i /dev/mapper/pve-data-meta0 -o /dev/mapper/pve-repaired_01
lvchange -an pve/data
lvconvert --thinpool pve/data --poolmetadata /dev/mapper/pve-repaired_01
Then restarted the computer and noticed that pve/data was inactive too now. So I tested again, getting with a new error (all 4 vm disks and pve/data volumes was failing with the same different error):
# lvchange -ay pve/data
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
# lvchange -ay pve
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Thin pool pve-data-tpool (253:6) transaction_id is 0, while expected 7.
Googled about that and find a possible solution (backing up and restoring vg):
# vgcfgbackup pve -f /root/pvebak
Volume group "pve" successfully backed up.
# nano /root/pvebak
(modified transaction_id, changin 7 for 0 on pve/data)
# vgcfgrestore pve --force -f /root/pvebak
Restored volume group pve.
Then tried to activate volumes again, but getting same results as the beggining, now pve/data gets activated, but 4 vms not and showing the first error message again
# lvchange -ay pve
device-mapper: reload ioctl on (253:5) failed: No data available
device-mapper: reload ioctl on (253:5) failed: No data available
device-mapper: reload ioctl on (253:5) failed: No data available
device-mapper: reload ioctl on (253:5) failed: No data available