Fix ZFS mount. rebooted the server and now cannot mount "zfs mount" (can be in readonly state)
Budget: €30 – €250 EUR
The essence is that one of my disks is in a "FAULTED" state, consequently, the ZFS pool is in a "SUSPENDED" state.
The data center diagnosed that the disk is in a bad condition (there are 104 disks in the pool).
Since I don't have RAID, and there is enough free space, I ran "zpool remove" on the disk named "mnt."
After a while, I stupidly rebooted the machine.
The removal was done to transfer data from the faulty disk to another one so that it could be safely replaced without data loss.
Now, when I run "zfs status," it shows:
remove: Evacuation of 15532536426325627714 in progress since Sat Jul 29 08:01:04 2023
1.77G copied out of 1.93T at 96.5K/s, 0.09% done, (copy is slow, no estimated time)
So, I have no idea how long this will take.
I can only import (i.e., mount) the pool if I give it the "readonly" flag:
zpool import -f -FXn mnt
I also tried running a command above that should restore it, but apparently, it will take too much time given the current number of disks. (run, no output for at least 4 hours)
Oh, and all this works only if I manually set the "zfs_max_missing_tvds" parameter to 1.
So, yes, I made a mistake by rebooting the server while the disk removal process was running.
The task now is to somehow revert everything, i.e., mount the pool back without the "readonly" flag and let it continue working with the disk.
I tried canceling the current "remove" task, but since the pool can only be mounted in "readonly" mode properly, it starts complaining that it's not accessible.
If i do not change zfs_max_missing_tvds value i'm getting:
zfs cannot import 'mnt': one or more devices is currently unavailable
Within the zpool import command i can see message like:
The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag.
Hopefully somebody can help me here, hot to run back mount smoothly if only i have not rebooted the machine everything would be alight.
pool: mnt
state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
scan: scrub repaired 0B in 08:53:52 with 0 errors on Sun Jul 9 09:17:53 2023
remove: Evacuation of 15532536426325627714 in progress since Sat Jul 29 08:01:04 2023
1.77G copied out of 1.93T at 84.7K/s, 0.09% done, (copy is slow, no estimated time)
config:
NAME STATE READ WRITE CKSUM
mnt DEGRADED 0 0 0
scsi-35000cca29bd7139c ONLINE 0 0 0
scsi-35000cca29bcd83cc ONLINE 0 0 0
scsi-35000cca29bb7c9e0 ONLINE 0 0 0
scsi-35000cca29bcdcc88 ONLINE 0 0 0
scsi-35000cca29bce7bd0 ONLINE 0 0 0
scsi-35000cca259201748 ONLINE 0 0 0
scsi-35000cca29bcda450 ONLINE 0 0 0
scsi-35000cca29bce0508 ONLINE 0 0 0
scsi-35000cca25920028c ONLINE 0 0 0
scsi-35000cca29bcd9dc8 ONLINE 0 0 0
scsi-35000cca259201514 ONLINE 0 0 0
scsi-35000cca29bcda90c ONLINE 0 0 0
scsi-35000cca29bcbd67c ONLINE 0 0 0
scsi-35000cca29bcc9028 ONLINE 0 0 0
scsi-35000cca259207580 ONLINE 0 0 0
scsi-35000cca29bce02d4 ONLINE 0 0 0
scsi-35000cca29bdbfa3c ONLINE 0 0 0
scsi-35000cca29bcbe09c ONLINE 0 0 0
scsi-35000cca29bce8528 ONLINE 0 0 0
scsi-35000cca29bcc9590 ONLINE 0 0 0
scsi-35000cca29bcdce20 ONLINE 0 0 0
scsi-35000cca29bcca690 ONLINE 0 0 0
scsi-35000cca29bb67c0c ONLINE 0 0 0
sdz ONLINE 0 0 0
scsi-35000cca259200c10 ONLINE 0 0 0
scsi-35000cca29bcc8f48 ONLINE 0 0 0
scsi-35000cca29bdb987c ONLINE 0 0 0
scsi-35000cca29bce0448 ONLINE 0 0 0
scsi-35000cca29bcda598 ONLINE 0 0 0
scsi-35000cca2592039e0 ONLINE 0 0 0
scsi-35000cca29bccfe00 ONLINE 0 0 0
scsi-35000cca29bc92d40 ONLINE 0 0 0
scsi-35000cca297221944 ONLINE 0 0 0
scsi-35000cca297221ba8 ONLINE 0 0 0
scsi-35000cca29721fe08 ONLINE 0 0 0
scsi-35000cca29bdb5cb8 ONLINE 0 0 0
sdam ONLINE 0 0 0
sdan ONLINE 0 0 0
sdao ONLINE 0 0 0
sdap ONLINE 0 0 0
scsi-35000cca29721cd78 ONLINE 0 0 0
15532536426325627714 FAULTED 0 0 0 was /dev/sdar1
scsi-35000cca29bdb82d0 ONLINE 0 0 0
errors: List of errors unavailable: pool I/O is currently suspended
The data center diagnosed that the disk is in a bad condition (there are 104 disks in the pool).
Since I don't have RAID, and there is enough free space, I ran "zpool remove" on the disk named "mnt."
After a while, I stupidly rebooted the machine.
The removal was done to transfer data from the faulty disk to another one so that it could be safely replaced without data loss.
Now, when I run "zfs status," it shows:
remove: Evacuation of 15532536426325627714 in progress since Sat Jul 29 08:01:04 2023
1.77G copied out of 1.93T at 96.5K/s, 0.09% done, (copy is slow, no estimated time)
So, I have no idea how long this will take.
I can only import (i.e., mount) the pool if I give it the "readonly" flag:
zpool import -f -FXn mnt
I also tried running a command above that should restore it, but apparently, it will take too much time given the current number of disks. (run, no output for at least 4 hours)
Oh, and all this works only if I manually set the "zfs_max_missing_tvds" parameter to 1.
So, yes, I made a mistake by rebooting the server while the disk removal process was running.
The task now is to somehow revert everything, i.e., mount the pool back without the "readonly" flag and let it continue working with the disk.
I tried canceling the current "remove" task, but since the pool can only be mounted in "readonly" mode properly, it starts complaining that it's not accessible.
If i do not change zfs_max_missing_tvds value i'm getting:
zfs cannot import 'mnt': one or more devices is currently unavailable
Within the zpool import command i can see message like:
The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag.
Hopefully somebody can help me here, hot to run back mount smoothly if only i have not rebooted the machine everything would be alight.
pool: mnt
state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
scan: scrub repaired 0B in 08:53:52 with 0 errors on Sun Jul 9 09:17:53 2023
remove: Evacuation of 15532536426325627714 in progress since Sat Jul 29 08:01:04 2023
1.77G copied out of 1.93T at 84.7K/s, 0.09% done, (copy is slow, no estimated time)
config:
NAME STATE READ WRITE CKSUM
mnt DEGRADED 0 0 0
scsi-35000cca29bd7139c ONLINE 0 0 0
scsi-35000cca29bcd83cc ONLINE 0 0 0
scsi-35000cca29bb7c9e0 ONLINE 0 0 0
scsi-35000cca29bcdcc88 ONLINE 0 0 0
scsi-35000cca29bce7bd0 ONLINE 0 0 0
scsi-35000cca259201748 ONLINE 0 0 0
scsi-35000cca29bcda450 ONLINE 0 0 0
scsi-35000cca29bce0508 ONLINE 0 0 0
scsi-35000cca25920028c ONLINE 0 0 0
scsi-35000cca29bcd9dc8 ONLINE 0 0 0
scsi-35000cca259201514 ONLINE 0 0 0
scsi-35000cca29bcda90c ONLINE 0 0 0
scsi-35000cca29bcbd67c ONLINE 0 0 0
scsi-35000cca29bcc9028 ONLINE 0 0 0
scsi-35000cca259207580 ONLINE 0 0 0
scsi-35000cca29bce02d4 ONLINE 0 0 0
scsi-35000cca29bdbfa3c ONLINE 0 0 0
scsi-35000cca29bcbe09c ONLINE 0 0 0
scsi-35000cca29bce8528 ONLINE 0 0 0
scsi-35000cca29bcc9590 ONLINE 0 0 0
scsi-35000cca29bcdce20 ONLINE 0 0 0
scsi-35000cca29bcca690 ONLINE 0 0 0
scsi-35000cca29bb67c0c ONLINE 0 0 0
sdz ONLINE 0 0 0
scsi-35000cca259200c10 ONLINE 0 0 0
scsi-35000cca29bcc8f48 ONLINE 0 0 0
scsi-35000cca29bdb987c ONLINE 0 0 0
scsi-35000cca29bce0448 ONLINE 0 0 0
scsi-35000cca29bcda598 ONLINE 0 0 0
scsi-35000cca2592039e0 ONLINE 0 0 0
scsi-35000cca29bccfe00 ONLINE 0 0 0
scsi-35000cca29bc92d40 ONLINE 0 0 0
scsi-35000cca297221944 ONLINE 0 0 0
scsi-35000cca297221ba8 ONLINE 0 0 0
scsi-35000cca29721fe08 ONLINE 0 0 0
scsi-35000cca29bdb5cb8 ONLINE 0 0 0
sdam ONLINE 0 0 0
sdan ONLINE 0 0 0
sdao ONLINE 0 0 0
sdap ONLINE 0 0 0
scsi-35000cca29721cd78 ONLINE 0 0 0
15532536426325627714 FAULTED 0 0 0 was /dev/sdar1
scsi-35000cca29bdb82d0 ONLINE 0 0 0
errors: List of errors unavailable: pool I/O is currently suspended