Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
mdadm - stuck reshape operation
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Cephas
n00b
n00b


Joined: 25 Mar 2011
Posts: 10

PostPosted: Wed Apr 27, 2016 10:42 am    Post subject: mdadm - stuck reshape operation Reply with quote

Hi,

I have a 3 disk RAID 5 array that I tried to add a 4th disk to.

Code:
> mdadm --add /dev/md6 /dev/sdb1
> mdadm --grow --raid-devices=4 /dev/md6


This operation started successfully and proceeded until it hit 51.1%

Code:
> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md6 : active raid5 sda1[0] sdb1[5] sdf1[3] sde1[4]
      3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      [==========>..........]  reshape = 51.1% (998533632/1953382400) finish=9046506.1min speed=1K/sec
      bitmap: 0/15 pages [0KB], 65536KB chunk


It has been sitting on the same 998533632 position for days. I've tried a few reboots, but it never progresses.
Stopping the array, or trying to start the logical volume in it hangs.
Altering the min / max speed parameters has no effect.
When I reboot and resemble the array the speed indicated steadily drops to almost 0.

Code:
> mdadm --assemble /dev/md6 --verbose --uuid 90c2b5c3:3bbfa0d7:a5efaeed:726c43e2


I haven't tried anything more drastic than a reboot yet,
Below is as much information as I can think to provide at this stage. Please let me know what else I can do.
I'm happy to change kernels, kernel config or anything else require to get better info.

Kernel: 4.4.3
mdadm 3.4

Code:
> ps aux | grep md6
root      5041 99.9  0.0      0     0 ?        R    07:10 761:58 [md6_raid5]
root      5042  0.0  0.0      0     0 ?        D    07:10   0:00 [md6_reshape]


This is consistent. 100% cpu on the raid component, but not the reshape
My guess is that the sync and reshape operation are entangled somehow.

Code:
> mdadm --detail --verbose /dev/md6
/dev/md6:
        Version : 1.2
  Creation Time : Fri Aug 29 21:13:52 2014
     Raid Level : raid5
     Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
  Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Apr 27 07:10:07 2016
          State : clean, reshaping
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

 Reshape Status : 51% complete
  Delta Devices : 1, (3->4)

           Name : Alpheus:6  (local to host Alpheus)
           UUID : 90c2b5c3:3bbfa0d7:a5efaeed:726c43e2
         Events : 47975

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       4       8       65        1      active sync   /dev/sde1
       3       8       81        2      active sync   /dev/sdf1
       5       8       17        3      active sync   /dev/sdb1


Code:
> iostat
Linux 4.4.3-gentoo (Alpheus)    04/27/2016      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.84    0.00   24.50    0.09    0.00   73.57

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.02         2.72         1.69     128570      79957
sdb               0.01         0.03         1.69       1447      79889
sdd               3.85         2.27        56.08     106928    2646042
sde               0.02         2.73         1.69     128610      79961
sdf               0.02         2.72         1.69     128128      79961
sdc               4.08         5.44        56.08     256899    2646042
md0               2.91         7.62        55.08     359714    2598725
dm-0              0.00         0.03         0.00       1212          0
dm-1              0.00         0.05         0.00       2151          9
dm-2              2.65         6.52         3.42     307646     161296
dm-3              0.19         1.03        51.66      48377    2437420
md6               0.00         0.02         0.00       1036          0


Code:
> dmesg
[ 1199.426995] md: bind<sde1>
[ 1199.427779] md: bind<sdf1>
[ 1199.428379] md: bind<sdb1>
[ 1199.428592] md: bind<sda1>
[ 1199.429260] md/raid:md6: reshape will continue
[ 1199.429274] md/raid:md6: device sda1 operational as raid disk 0
[ 1199.429275] md/raid:md6: device sdb1 operational as raid disk 3
[ 1199.429276] md/raid:md6: device sdf1 operational as raid disk 2
[ 1199.429277] md/raid:md6: device sde1 operational as raid disk 1
[ 1199.429498] md/raid:md6: allocated 4338kB
[ 1199.429807] md/raid:md6: raid level 5 active with 4 out of 4 devices, algorithm 2
[ 1199.429810] RAID conf printout:
[ 1199.429811]  --- level:5 rd:4 wd:4
[ 1199.429812]  disk 0, o:1, dev:sda1
[ 1199.429814]  disk 1, o:1, dev:sde1
[ 1199.429816]  disk 2, o:1, dev:sdf1
[ 1199.429817]  disk 3, o:1, dev:sdb1
[ 1199.429993] created bitmap (15 pages) for device md6
[ 1199.430297] md6: bitmap initialized from disk: read 1 pages, set 0 of 29807 bits
[ 1199.474604] md6: detected capacity change from 0 to 4000527155200
[ 1199.474611] md: reshape of RAID array md6
[ 1199.474613] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1199.474614] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 1199.474617] md: using 128k window, over a total of 1953382400k.


Code:
> lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                             8:0    0  1.8T  0 disk
└─sda1                          8:1    0  1.8T  0 part
  └─md6                         9:6    0  3.7T  0 raid5
sdb                             8:16   0  1.8T  0 disk
└─sdb1                          8:17   0  1.8T  0 part
  └─md6                         9:6    0  3.7T  0 raid5
sdc                             8:32   0  2.7T  0 disk
├─sdc1                          8:33   0   16M  0 part
└─sdc2                          8:34   0  2.7T  0 part
  └─md0                         9:0    0  2.7T  0 raid1
    ├─vg--mirror-swap         253:0    0    4G  0 lvm   [SWAP]
    ├─vg--mirror-boot         253:1    0  256M  0 lvm   /boot
    ├─vg--mirror-root         253:2    0  256G  0 lvm   /
    └─vg--mirror-data--mirror 253:3    0  2.5T  0 lvm   /data/mirror
sdd                             8:48   0  2.7T  0 disk
├─sdd1                          8:49   0   16M  0 part
└─sdd2                          8:50   0  2.7T  0 part
  └─md0                         9:0    0  2.7T  0 raid1
    ├─vg--mirror-swap         253:0    0    4G  0 lvm   [SWAP]
    ├─vg--mirror-boot         253:1    0  256M  0 lvm   /boot
    ├─vg--mirror-root         253:2    0  256G  0 lvm   /
    └─vg--mirror-data--mirror 253:3    0  2.5T  0 lvm   /data/mirror
sde                             8:64   0  1.8T  0 disk
└─sde1                          8:65   0  1.8T  0 part
  └─md6                         9:6    0  3.7T  0 raid5
sdf                             8:80   0  1.8T  0 disk
└─sdf1                          8:81   0  1.8T  0 part
  └─md6                         9:6    0  3.7T  0 raid5


Thanks for any pointers
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2970
Location: Germany

PostPosted: Wed Apr 27, 2016 12:55 pm    Post subject: Reply with quote

There are no I/O errors in dmesg? What kind of disks are you using, and is it SATA/USB/...? Show smartctl -a and run a -t long selftest for all disks?

Maybe one disk has problems without reporting errors properly, and instead spending lots of time in internal error correction... that is one case where software raid fails because it relies on disks reporting errors properly.

I'm running RAID-5 in 4.5.x kernel without problems so maybe that's also worth a try, just in case there was a kernel bug at some point, I don't know.

What's your stripe cache set to? http://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html
Back to top
View user's profile Send private message
deck2
n00b
n00b


Joined: 06 Jul 2016
Posts: 3

PostPosted: Wed Jul 06, 2016 3:51 am    Post subject: Reply with quote

Hello,

Did you happen to find a solution for this? I observe the exact same issue.

I had a drive legitimately fail during reshape which caused mdadm to lock up with full cpu usage. I subsequently stopped the array, and rebooted. Upon trying to start the array, the md127_raid5 process immediately spikes to 100% usage, and the array becomes completely unresponsive. Other processes involving the array show blocking on IO.

I've tried pretty much everything, including failing out the faulty drive. When ever I try to start the array, the process locks up.

Code:

mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 --verbose --backup-file=/tmp/grow_md127.bak
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb1 is identified as a member of /dev/md127, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 4.
mdadm: /dev/md127 has an active reshape - checking if critical section needs to be restored
mdadm: No backup metadata on /tmp/grow_md127.bak
mdadm: accepting backup with timestamp 1467397557 for array with timestamp 1467774201
mdadm: backup-metadata found on device-4 but is not needed
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sdc1 to /dev/md127 as 2
mdadm: added /dev/sdb1 to /dev/md127 as 3
mdadm: added /dev/sde1 to /dev/md127 as 4
mdadm: added /dev/sdd1 to /dev/md127 as 1
mdadm: /dev/md127 has been started with 4 drives (out of 5).


Code:

[ 2662.922539] md: bind<sdc1>
[ 2662.926020] md: bind<sdb1>
[ 2662.926333] md: bind<sde1>
[ 2662.926564] md: bind<sdd1>
[ 2662.960617] raid6: sse2x1   gen()  3484 MB/s
[ 2662.977603] raid6: sse2x1   xor()  3673 MB/s
[ 2662.994612] raid6: sse2x2   gen()  5652 MB/s
[ 2663.011601] raid6: sse2x2   xor()  5978 MB/s
[ 2663.028613] raid6: sse2x4   gen()  7156 MB/s
[ 2663.045601] raid6: sse2x4   xor()  3837 MB/s
[ 2663.045603] raid6: using algorithm sse2x4 gen() 7156 MB/s
[ 2663.045604] raid6: .... xor() 3837 MB/s, rmw enabled
[ 2663.045606] raid6: using intx1 recovery algorithm
[ 2663.051916] async_tx: api initialized (async)
[ 2663.058743] xor: measuring software checksum speed
[ 2663.068598]    prefetch64-sse: 13312.000 MB/sec
[ 2663.078597]    generic_sse: 12720.000 MB/sec
[ 2663.078599] xor: using function: prefetch64-sse (13312.000 MB/sec)
[ 2663.139818] md: raid6 personality registered for level 6
[ 2663.139822] md: raid5 personality registered for level 5
[ 2663.139823] md: raid4 personality registered for level 4
[ 2663.140432] md/raid:md127: reshape will continue
[ 2663.140454] md/raid:md127: device sdd1 operational as raid disk 1
[ 2663.140455] md/raid:md127: device sde1 operational as raid disk 4
[ 2663.140456] md/raid:md127: device sdb1 operational as raid disk 3
[ 2663.140457] md/raid:md127: device sdc1 operational as raid disk 2
[ 2663.140983] md/raid:md127: allocated 5432kB
[ 2663.142246] md/raid:md127: raid level 5 active with 4 out of 5 devices, algorithm 2
[ 2663.142251] RAID conf printout:
[ 2663.142253]  --- level:5 rd:5 wd:4
[ 2663.142254]  disk 1, o:1, dev:sdd1
[ 2663.142256]  disk 2, o:1, dev:sdc1
[ 2663.142257]  disk 3, o:1, dev:sdb1
[ 2663.142258]  disk 4, o:1, dev:sde1
[ 2663.142436] created bitmap (15 pages) for device md127
[ 2663.142682] md127: bitmap initialized from disk: read 1 pages, set 2 of 29807 bits
[ 2663.735434] md127: detected capacity change from 0 to 6000790732800
[ 2663.735465] md: reshape of RAID array md127
[ 2663.735477] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 2663.735483] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 2663.735498] md: using 128k window, over a total of 1953382400k.
[ 2663.989420]  md127: p1



Code:
ps aux | grep md127
root      4300 99.7  0.0      0     0 ?        R    23:03  15:05 [md127_raid5]
root      4301  0.0  0.0      0     0 ?        D    23:03   0:00 [md127_reshape]
root      4334  0.0  0.0   4900   712 pts/0    D+   23:03   0:00 mdadm -S /dev/md127


Code:
mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Sun May 18 16:54:52 2014
     Raid Level : raid5
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
   Raid Devices : 5
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Jul  5 23:03:21 2016
          State : clean, degraded, reshaping
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

 Reshape Status : 94% complete
  Delta Devices : 1, (4->5)

           Name : rza.eth0.net:0  (local to host rza.eth0.net)
           UUID : 9d5d1606:414b51f8:b5173999:7239c63f
         Events : 345118

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1
       4       8       17        3      active sync   /dev/sdb1
       5       8       65        4      active sync   /dev/sde1




Code:

uname -R
4.5.7-200.fc23.x86_64


Code:

mdadm -V
mdadm - v3.3.4 - 3rd August 2015



Sorry for posting cross distro.
Back to top
View user's profile Send private message
Cephas
n00b
n00b


Joined: 25 Mar 2011
Posts: 10

PostPosted: Wed Jul 06, 2016 4:04 am    Post subject: Reply with quote

I ended up starting from scratch unfortunately.
Array just had tv shows and movies on it, so everything was recoverable over time.
Wiped the superblocks and re-created.

Posted in the mdadm kernel group a bit. nothing worked.
Tried latest mdadm from source, plus just about every funky rebuild command I could find.

Sorry.
Back to top
View user's profile Send private message
deck2
n00b
n00b


Joined: 06 Jul 2016
Posts: 3

PostPosted: Wed Jul 06, 2016 4:08 am    Post subject: Reply with quote

Thanks for the reply. I'll post back for future reference if I figure something out.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2970
Location: Germany

PostPosted: Wed Jul 06, 2016 11:31 am    Post subject: Reply with quote

Try a SystemRescueCD ... your mdadm is old, for one, and not sure what to make of fc kernel.

If the kernel task keeps hanging in 100% CPU even with the current stable kernel, it's probably something for the linux-raid mailing list.
Back to top
View user's profile Send private message
deck2
n00b
n00b


Joined: 06 Jul 2016
Posts: 3

PostPosted: Thu Jul 07, 2016 2:32 am    Post subject: Reply with quote

Thanks frostschutz, I've seen your name come up a bunch in the various forums, your posts on mdadm are very helpful.

I've tried a couple different kernels / versions of mdadm. I don't think the version I am using to too old, its 3.3.4 which is only superseded by 3.4, which Neil claims could still be 'buggy'.

I tried going back to a couple older kernels, and the behaviour is the same.

The only additional piece of info is that I have observed in some cases, md127_reshape does run very briefly, and the count of reshaped blocks is incrementing slightly right when the array first assembles. I think this supports Cephas' suspicion that md127_raid5 and md127_reshape are getting locked into some manner of IO contention. In theory I could restart my PC 5000 times, and the reshape might complete.

My next step is to try to figure out how to either manually start the reshape without raid5 running, or find a way to block raid5 from running when the array is assembled.

I am also going to try building 3.4 from source and see what happens.

I'll try posting on the mailing list also and see if there are any other suggestions there.

thanks
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum