LVM datarescue
For future reference: how I fixed a stuck but partially complete pvmove.
I recently noticed some read errors off of one of my "storage" harddisks. Layout was something like 1*2tb + 1*1,5tb in a linear mapping, 1 VG (storage_1), 2 LVs (shares, homevideo). The 1,5gig harddisk failed. Contents: mostly backup-copies of DV tapes, FLACs from audio-CDs, and backups of digital/digitized photos.
So I ordered some disks, built a raid5 out of 4 2tb disks, and added it to storage_1. I pvmoved the 2tb disk to the new raid. Then I tried to move the failed 1,5tb disk to the new raid. This process stopped when pvmove was at about 50%, when the drive went completely dead. It would spin up, but not show up anywhere, tried with different computers and 2 usb-sata bridges.
It seems pvmove was able to move eveything of LV shares to the raid, the filesystem was still accessible and didn't show any sign of corruption.
Situation after reboot:
- LVs fail to activate (needed lvchange -a y --partial)
- after activation, homevideo is completely dead, shares seems to work without any problems (though it generates lots of device-mapper: raid1: Unable to read primary mirror during recovery messages)
- percentage-counter counting, unsure if actual disk activity happens
- pvmove complains:
# pvmove /dev/dm-4: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler /dev/dm-4: read failed after 0 of 4096 at 971224580096: Eingabe-/Ausgabefehler /dev/dm-4: read failed after 0 of 4096 at 971224637440: Eingabe-/Ausgabefehler /dev/dm-4: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler /dev/dm-5: read failed after 0 of 4096 at 1500298280960: Eingabe-/Ausgabefehler /dev/dm-5: read failed after 0 of 4096 at 1500298338304: Eingabe-/Ausgabefehler Couldn't find device with uuid u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5. Cannot change VG storage_1 while PVs are missing. Consider vgreduce --removemissing.
# lvs -a -o +devices storage_1 /dev/dm-4: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler /dev/dm-4: read failed after 0 of 4096 at 971224580096: Eingabe-/Ausgabefehler /dev/dm-4: read failed after 0 of 4096 at 971224637440: Eingabe-/Ausgabefehler /dev/dm-4: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler /dev/dm-5: read failed after 0 of 4096 at 1500298280960: Eingabe-/Ausgabefehler /dev/dm-5: read failed after 0 of 4096 at 1500298338304: Eingabe-/Ausgabefehler Couldn't find device with uuid u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5. LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices homevideo storage_1 -wI-a- 1,08t pvmove0(126141) homevideo storage_1 -wI-a- 1,08t /dev/md127(553732) [pvmove0] storage_1 p-C-ao 1,36t unknown device 38,07 unknown device(0),/dev/md127(604932) [pvmove0] storage_1 p-C-ao 1,36t unknown device 38,07 unknown device(126141),/dev/md127(731073) shares storage_1 -wI-a- 2,59t /dev/md127(128000) shares storage_1 -wI-a- 2,59t pvmove0(0) shares storage_1 -wI-a- 2,59t /dev/md127(485699) shares storage_1 -wI-a- 2,59t /dev/md127(0)
# dmsetup table storage_1-pvmove0-missing_1_0: 0 1896923136 error storage_1-shares: 0 2930270208 linear 9:127 1048579072 storage_1-shares: 2930270208 1033347072 linear 252:5 0 storage_1-shares: 3963617280 557326336 linear 9:127 3978849280 storage_1-shares: 4520943616 1048576000 linear 9:127 3072 storage_1-homevideo: 0 1896923136 linear 252:5 1033347072 storage_1-homevideo: 1896923136 419430400 linear 9:127 4536175616
# dmsetup info -c storage_1-pvmove0 252 5 L--w 2 2 0 LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFyOZtSj8iJexF2g6PMNRff3rYytIZ5RYJ storage_1-pvmove0-missing_1_0 252 4 L--w 1 1 0 LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFyOZtSj8iJexF2g6PMNRff3rYytIZ5RYJ-missing_1_0 storage_1-shares 252 6 L--w 0 4 0 LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFvbEPv7JJKoctUb2kNtGqjFrfsrHiS0v3 storage_1-homevideo 252 7 L--w 0 2 0 LVM-ONKgAr1yc19fIeDyMClPFjDv92vxDEAFwFm52QuLABa7OKlHdYp3yEe8wxp6K07I
# dmsetup status storage_1-pvmove0: 0 1033347072 linear storage_1-pvmove0: 1033347072 1896923136 mirror 2 252:4 9:127 89145/1852464 1 SA 1 core storage_1-pvmove0-missing_1_0: 0 1896923136 error storage_1-shares: 0 2930270208 linear storage_1-shares: 2930270208 1033347072 linear storage_1-shares: 3963617280 557326336 linear storage_1-shares: 4520943616 1048576000 linear storage_1-homevideo: 0 1896923136 linear storage_1-homevideo: 1896923136 419430400 linear
Then, I examined the latest backup of metadata of PV storage_1 in /etc/lvm/archive:
storage_1 {
id = "ONKgAr-1yc1-9fIe-DyMC-lPFj-Dv92-vxDEAF"
seqno = 51
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
physical_volumes {
pv0 {
id = "u1F6AW-pvCR-gM1c-O1c5-oWa0-s1Nd-oKTkf5"
device = "unknown device" # Hint only
status = ["ALLOCATABLE"]
flags = ["MISSING"]
dev_size = 2930277168 # 1,36452 Terabytes
pe_start = 384
pe_count = 357699 # 1,36451 Terabytes
}
pv1 {
id = "YuX6jm-DfAG-ZE8Y-z0Mn-2bnC-aoTC-azc3zC"
device = "/dev/md127" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 11721071616 # 5,45805 Terabytes
pe_start = 3072
pe_count = 1430794 # 5,45805 Terabytes
}
}
logical_volumes {
shares {
id = "vbEPv7-JJKo-ctUb-2kNt-GqjF-rfsr-HiS0v3"
status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
flags = []
segment_count = 4
segment1 {
start_extent = 0
extent_count = 357699 # 1,36451 Terabytes
type = "striped"
stripe_count = 1 # linearstripes = [
"pv1", 128000
]
}
segment2 {
start_extent = 357699
extent_count = 126141 # 492,738 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pvmove0", 0
]
}
segment3 {
start_extent = 483840
extent_count = 68033 # 265,754 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 485699
]
}
segment4 {
start_extent = 551873
extent_count = 128000 # 500 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 0
]
}
}
homevideo {
id = "wFm52Q-uLAB-a7OK-lHdY-p3yE-e8wx-p6K07I"
status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
flags = []
segment_count = 2
segment1 {
start_extent = 0
extent_count = 231558 # 904,523 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pvmove0", 126141
]
}
segment2 {start_extent = 231558
extent_count = 51200 # 200 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 553732
]
}
}
pvmove0 {
id = "yOZtSj-8iJe-xF2g-6PMN-Rff3-rYyt-IZ5RYJ"
status = ["READ", "WRITE", "PVMOVE", "LOCKED"]
flags = []
allocation_policy = "contiguous"
segment_count = 2
segment1 {
start_extent = 0
extent_count = 126141 # 492,738 Gigabytes
type = "mirror"
mirror_count = 2
extents_moved = 126141 # 492,738 Gigabytes
mirrors = [
"pv0", 0,
"pv1", 604932
]
}
segment2 {
start_extent = 126141
extent_count = 231558 # 904,523 Gigabytes
type = "mirror"
mirror_count = 2
extents_moved = 0 # 0 Kilobytes
mirrors = [
"pv0", 126141,
"pv1", 731073
]
}
}
}
}
Notice segment2 of shares and segment1 of pvmove0: extents_moved = 126141, suggesting that everything belonging to LV "shares" was indeed successfully mirrored from the failed drive. So i edited the metadata to look like
storage_1 {
id = "ONKgAr-1yc1-9fIe-DyMC-lPFj-Dv92-vxDEAF"
seqno = 51
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
physical_volumes {
pv1 {
id = "YuX6jm-DfAG-ZE8Y-z0Mn-2bnC-aoTC-azc3zC"
device = "/dev/md127" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 11721071616 # 5,45805 Terabytes
pe_start = 3072
pe_count = 1430794 # 5,45805 Terabytes
}
}
logical_volumes {
shares {
id = "vbEPv7-JJKo-ctUb-2kNt-GqjF-rfsr-HiS0v3"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
segment_count = 4
segment1 {
start_extent = 0
extent_count = 357699 # 1,36451 Terabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 128000
]
}
segment2 {
start_extent = 357699
extent_count = 126141 # 492,738 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 604932
]
}
segment3 {
start_extent = 483840
extent_count = 68033 # 265,754 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 485699
]
}
segment4 {
start_extent = 551873
extent_count = 128000 # 500 Gigabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 0
]
}
}
}
}
and vgcfgrestore(8)'ed it. Server needed a further reboot to get rid of some stuck pvmove kernel thread that was trying to pick up where the former pvmove had to stop when the drive failed. So far, everything seems to be in order, xfs_repair didn't complain about anything, and find did put out a credible list of files.
Now i "just" have to copy those 120 DV tapes to the computer again... Lesson: there is no data worth keeping on a harddisk that isn't important enough to keep on a raid.