Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Need help on RAID5->LVM->ext3 throughput
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
RayOfLight
n00b
n00b


Joined: 27 Aug 2013
Posts: 67
Location: Catalunya

PostPosted: Sun Jan 21, 2018 12:54 am    Post subject: Need help on RAID5->LVM->ext3 throughput Reply with quote

Hello,

I've been running a 6x2TB RAID5 setup for about many years, and all this time I have felt the speed of this setup is awfully slow, so I should be ashamed of having 6 HDDs working slower than one by itself :(

when I run hdparm I get:

Timing cache reads at about 11GB/s, and
Timing buffered disk reads, about 150MB/s for raw /dev/sd* drives

when I go up to /dev/md, the speed goes up to 390 MB/s

if I go to the lvm drive, /dev/dm-*, the speed stays at md speed: 390 MB/s or so.

so far so good!

now, dd timing, if I run dd if=(sd*,md*,dm-*), I get about 135MB/s, 450MB/s, and 450MB/s respectively, that's awesome!

But now the sad truth:

Code:

/usr/bin/time sudo fsck.ext3 -vttFfDpC0 /dev/mapper/vg0-linux-isos
Pas 1: Memòria utilitzada: 7064k/2348k (6921k/144k), temps: 1133.61/18.77/ 3.89
Pas 1: I/O read: 1182MB, write: 0MB, rate: 1.04MB/s
Pas 2: Memòria utilitzada: 7592k/1488k (7043k/550k), temps: 22.90/ 0.43/ 0.18
Pas 2: I/O read: 153MB, write: 0MB, rate: 6.68MB/s
Pic de memòria: Memòria utilitzada: 7908k/1488k (7231k/678k), temps: 1156.89/19.22/ 4.07
Pass 3A: Memòria utilitzada: 13776k/1488k (11788k/1989k), temps: 89.48/ 0.61/ 0.74
Pass 3A: I/O read: 224MB, write: 129MB, rate: 3.94MB/s
Pas 3: Memòria utilitzada: 13776k/1488k (11271k/2506k), temps: 89.48/ 0.62/ 0.74
Pas 3: I/O read: 224MB, write: 129MB, rate: 3.94MB/s
Pas 4: Memòria utilitzada: 13776k/568k (10445k/3332k), temps:  0.03/ 0.03/ 0.00
Pas 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pas 5: Memòria utilitzada: 13776k/380k (4766k/9011k), temps:  0.05/ 0.05/ 0.00
Pas 5: I/O read: 0MB, write: 0MB, rate: 0.00MB/s

      345506 nodes-i emprats (22.49% de 1536000)
       32738 fitxers no contigus (9.5%)
         245 directoris no contigus (0.1%)
     nombre de nodes-i amb blocs ind/dind/tind: 27425/2175/38
   156534008 blocs emprats (39.81% de 393216000)
           0 blocs dolents
          44 fitxers grans

      298993 fitxers normals
       28074 directoris
           0 fitxers de dispositiu orientat a caràcters
           0 fitxers de dispositiu orientat a blocs
           1 fifo
           0 enllaços
       18402 enllaços simbòlics (17891 enllaços simbòlics ràpids)
          27 sòcols
------------
      345497 fitxers
Memòria utilitzada: 13776k/380k (4769k/9008k), temps: 1414.84/19.99/ 5.00
I/O read: 1560MB, write: 177MB, rate: 1.23MB/s
19.99user 5.23system [b]23:35.10elapsed 1%CPU[/b] (0avgtext+0avgdata 17168maxresident)k
3611288inputs+360376outputs (0major+4663minor)pagefaults 0swaps


As you can see, the fsck of a 1.5TB LVM LV takes about 24 minutes on a 4Ghz 12 core machine, and with iostat (150 tps and 1-2MB/s for md and 20tps and 0.6MB/s per sd* disk) and atop (20% raw sd* usage, 99% lvm usage, 0% raid usage), I'm unable to see where the bottleneck is (fsck only uses one core), also, the speeds are shown as 1.04 to 6.68MB/s, which I'm really ashamed of, I need some help!

What is your fsck thoughput?? Is it really so slow? What can I tune?

PS: AFAICR, these drives where the first commercial generation sporting 4KB sector size :) Also, to me, slowest step was #3
Back to top
View user's profile Send private message
blopsalot
Apprentice
Apprentice


Joined: 28 Jan 2017
Posts: 231

PostPosted: Sun Jan 21, 2018 1:26 am    Post subject: Reply with quote

one of the main advantages of ext4 over ext3 is significantly faster fsck. or consider xfs.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2970
Location: Germany

PostPosted: Sun Jan 21, 2018 1:31 am    Post subject: Reply with quote

RAID might improve linear speeds but seeks remain as slow as always. That is also why RAID0 to make system faster is mostly a hoax. That's also why SSD is so insanely fast... no seek times. That matters a lot more than linear speeds (which is the one thing HDDs don't completely stink at).

Your fsck is a task that relies heavily on seeking all over the place. If read-only, it should perform about the same as a single drive would. If there are writes, mdadm has to do additional reads to recalculate parity, additional writes to update parity, as well as mdadm metadata (update times, event counts, ...). Many small writes are kind of worst case for RAID as that's where the RAID overhead kicks in.
Back to top
View user's profile Send private message
RayOfLight
n00b
n00b


Joined: 27 Aug 2013
Posts: 67
Location: Catalunya

PostPosted: Sun Jan 21, 2018 1:39 am    Post subject: Reply with quote

Thanks for the replies, although my ext3 FSs (I built the array 8 years ago) are tuned with low inode count and a big bytes-per-inode amount, I'll check on changing my FS, I've just read that when XFS fails, it loses data and ext4 is better, I'll look if I go full-BTRFS... .

On the other comment, fsck is almost full read-only, I'm not complaining right now on write speeds, and about 600-1400 kB/s is... crazy, are you sure that's normal?

I have to be glad that LV is 1.5TB and not 10TB, I'll try moving its contents to another LV and creating a new LV with a different FS.
Back to top
View user's profile Send private message
blopsalot
Apprentice
Apprentice


Joined: 28 Jan 2017
Posts: 231

PostPosted: Sun Jan 21, 2018 2:05 am    Post subject: Reply with quote

RayOfLight wrote:
Thanks for the replies, although my ext3 FSs (I built the array 8 years ago) are tuned with low inode count and a big bytes-per-inode amount, I'll check on changing my FS, I've just read that when XFS fails, it loses data and ext4 is better, I'll look if I go full-BTRFS... .

On the other comment, fsck is almost full read-only, I'm not complaining right now on write speeds, and about 600-1400 kB/s is... crazy, are you sure that's normal?

I have to be glad that LV is 1.5TB and not 10TB, I'll try moving its contents to another LV and creating a new LV with a different FS.


sys-block/blktrace will give you a better idea why it's slow, but yes unfortunately, it may be "normal" considering ext3, constant seeking, and RAID overhead.
Back to top
View user's profile Send private message
RayOfLight
n00b
n00b


Joined: 27 Aug 2013
Posts: 67
Location: Catalunya

PostPosted: Sun Jan 21, 2018 2:27 am    Post subject: Reply with quote

I'm working on it, I have just discovered that my kernel has no ext3 support, only ext4.
fstab is set to ext3 and it has never complained very loudy:

Code:
[   11.815037] EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
[   11.945205] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)


after unmounting ext3, updating fstab to ext4 and rebooting:

Code:
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)


but all of this is pointless, so if I read correctly, when dumpe2fs returns:
Code:
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file


it means that the FS is really ext3 (8 years after, I don't remember if I set ext3 or ext4...), as I require "uninit_bg" to be able to improve fsck times.

I will follow debian's guide to convert ext3 to ext4 and reevaluate, thank you :)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13855

PostPosted: Sun Jan 21, 2018 5:24 am    Post subject: Reply with quote

If this is a first generation 4K drive, it is possible that you misaligned the partition. Some of the first generation drives did not adequately advertise their alignment requirements, so tools tended to align partitions for DOS compatibility. That can hurt throughput by causing unnecessary I/O, especially when writes are made (though you did say this is a read-mostly workload).

The kernel ext4 driver can be used for ext3 filesystems, and this is a very common approach. It will not enable any of the ext4-specific features that would prevent the filesystem later being interpreted as ext3. For that, you need to explicitly migrate the drive to ext4.
Back to top
View user's profile Send private message
RayOfLight
n00b
n00b


Joined: 27 Aug 2013
Posts: 67
Location: Catalunya

PostPosted: Sun Jan 21, 2018 6:04 am    Post subject: Reply with quote

ST2000DL003-9VT166

I remember being extra concious when building the array back then, but I still don't know how to check if it is aligned or not as smartmontools report:

Code:
Sector Size:      512 bytes logical/physical


and if I remember well, that is just misleading (the drive hides its 4K and reports 512 to the OS)
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1301
Location: Frankfurt, Germany

PostPosted: Sun Jan 21, 2018 11:04 am    Post subject: Reply with quote

Quote:
I will follow debian's guide to convert ext3 to ext4 and reevaluate, thank you

I wouldn't do that. I would backup my files to another disk/partition using tar, reformat the original partition to ext4 and restore the files.

Quote:
The recommended method for upgrading an ext3 filesystem to ext4 is to back up the entire volume, reformat the storage device with ext4, and restore the entire volume onto the fresh filesystem. This has the effect of defragmenting all files, adding directory indexes, and speeding up both e2fsck and metadata access by clustering metadata together. Furthermore, all files will have higher size limits and better capabilities than before.

See: https://ext4.wiki.kernel.org/index.php/UpgradeToExt4


Last edited by mike155 on Sun Jan 21, 2018 11:09 am; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43208
Location: 56N 3W

PostPosted: Sun Jan 21, 2018 11:05 am    Post subject: Reply with quote

RayOfLight,

Look at your partition tables in sectors. That's always 512B sectors.
Divide the start sector by 8. If you get remainder zero, you are correctly aligned.

Don't pay too much attention to what smartctl -i says. Many early 4k sector drives lied to the outside world about the physical sector size.

This drive is clearly not 4k aligned.
Code:
Device     Boot    Start        End    Sectors   Size Id Type
/dev/sda1  *          63      80324      80262  39.2M fd Linux raid autodetect
/dev/sda2          80325    1124549    1044225 509.9M 82 Linux swap / Solaris
/dev/sda4        1124550 1953520064 1952395515   931G  5 Extended
/dev/sda5        1124613   11631059   10506447     5G fd Linux raid autodetect
/dev/sda6       11631123 1953520064 1941888942   926G fd Linux raid autodetect
The start sectors are mostly odd numbers.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum