Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Hardware RAID howto available? LSI MegaRAID on Gentoo?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
seifn06
Tux's lil' helper
Tux's lil' helper


Joined: 19 Sep 2004
Posts: 85
Location: Lowell, Michigan

PostPosted: Wed Apr 08, 2020 2:48 pm    Post subject: Hardware RAID howto available? LSI MegaRAID on Gentoo? Reply with quote

Hello! I'm looking for info or a howto on working with hardware RAID on Gentoo. My question right now is: how do I access a RAID1 array I have set up in hardware on my Gentoo Linux computer? Should I expect to see a device reference in my /dev directory corresponding to my RAID array? What, if any, kernel options and/or Portage packages do I need? Also, would it be advisable for to match my RAID array's strip or stripe sizes to parameters in the ext4 file system I hope to put on it? If so, which parameters? I'm asking so I can optimize my RAID array performance with Gentoo. Any info or links to manuals, guides, how-to's are most appreciated.

I have a working Gentoo Sources kernel 5.4.28 installed and running with root drive on a single SSD, partitioned with parted as per instructions in the handbook.
https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks

I then installed an LSI MegaRAID SAS 9260-4i controller in a PCI express slot on my computer and then connected 2 x 2TB SSD's to that controller card. I was able to boot into the LSI RAID controller's BIOS and configure a virtual drive (RAID1) and fully initialize this array. The RAID controller's BIOS shows it recognizes both my physical drives and the logical/virtual drive I created in my RAID1 array at boot. I booted back into Gentoo and am unsure how to access this virtual drive. I see a /dev/md0 device now on my computer and hoped that might be a reference to my new RAID array so I tried running parted on this device to partition it so I could then set up a file system.

Code:
# parted -a optimal /dev/md0
Warning: Error fsyncing/closing /dev/md0: Input/output error
Retry/Ignore? Ignore


and

Code:
# parted /dev/md0
Warning: Error fsyncing/closing /dev/md0: Input/output error
Retry/Ignore? Ignore


I've seen the Gentoo Software RAID - Gentoo Wiki handbook, but don't believe I need this because I've got a RAID controller in hardware.

In my manually-configured kernel, I enabled the following options:
Code:
Device Drivers > Multiple devices driver support (RAID and LVM) >
<*> RAID Support
[*] Autodetect RAID arrays during kernel boot
<*> Device mapper support
<*> Mirror target
<*> Zero target


I do NOT see any MegaRAID options in my kernel config -- either searching through my .config file or through the menus in make menuconfig. Though I note that dalu apparently had such options in their kernel back in late 2017:
https://forums.gentoo.org/viewtopic-t-1074638-highlight-lsi+megaraid.html

I'm unsure what, if any, other kernel options I need to enable so my kernel can use my RAID1 array. I searched through my kernel config file for "LSI" and "MegaRAID" and found nothing seemingly applicable.

Broadcom bought out LSI apparently and I see on Broadcom's website that they have many Linux drivers for my particular controller:
https://www.broadcom.com/support/download-search?pg=Storage+Adapters,+Controllers,+and+ICs&pf=RAID+Controller+Cards&pn=&pa=&po=&dk=9260&pl=

A quick emerge --search for raid, lsi, broadcom shows nothing promising to me. Am I going to need to roll my own driver here?

For reference:
Code:
# lspci -k
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
        Subsystem: ASUSTeK Computer Inc. PRIME H310M-D
        Kernel driver in use: skl_uncore
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Desktop)
        Subsystem: ASUSTeK Computer Inc. UHD Graphics 630 (Desktop)
        Kernel driver in use: i915
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH USB 3.1 xHCI Host Controller
        Kernel driver in use: xhci_hcd
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH Shared SRAM
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH HECI Controller
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH SATA AHCI Controller
        Kernel driver in use: ahci
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #3 (rev f0)
        Kernel driver in use: pcieport
00:1c.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #5 (rev f0)
        Kernel driver in use: pcieport
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
        Kernel driver in use: pcieport
00:1d.3 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #12 (rev f0)
        Kernel driver in use: pcieport
00:1f.0 ISA bridge: Intel Corporation H370 Chipset LPC/eSPI Controller (rev 10)
        Subsystem: ASUSTeK Computer Inc. H370 Chipset LPC/eSPI Controller
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH cAVS
        Kernel driver in use: snd_hda_intel
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH SMBus Controller
        Kernel driver in use: i801_smbus
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
        Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH SPI Controller
02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2108 [Liberator] (rev 05)
        Subsystem: Broadcom / LSI MegaRAID SAS 9260-4i
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
        Subsystem: ASUSTeK Computer Inc. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
        Kernel driver in use: r8169


Any/all help is most appreciated.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7677
Location: Saint Amant, Acadiana

PostPosted: Wed Apr 08, 2020 3:14 pm    Post subject: Reply with quote

To my knowledge LSI MegaRAID is fakeraid.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
seifn06
Tux's lil' helper
Tux's lil' helper


Joined: 19 Sep 2004
Posts: 85
Location: Lowell, Michigan

PostPosted: Wed Apr 08, 2020 3:44 pm    Post subject: Reply with quote

Jaglover, in general, how does one know whether a RAID controller is fakeraid?
I've read https://skrypuch.com/raid/
My LSI controller allows for RAID 0, 1, 5, 6, 00, 10, 50 and 60. And it supports hot swapping. What language might I expect to see in my LSI manual that might tell me this is fakeraid?
Back to top
View user's profile Send private message
joanandk
Tux's lil' helper
Tux's lil' helper


Joined: 12 Feb 2017
Posts: 105

PostPosted: Wed Apr 08, 2020 3:56 pm    Post subject: Re: Hardware RAID howto available? LSI MegaRAID on Gentoo? Reply with quote

seifn06 wrote:
I was able to boot into the LSI RAID controller's BIOS and configure a virtual drive (RAID1) and fully initialize this array. The RAID controller's BIOS shows it recognizes both my physical drives and the logical/virtual drive I created in my RAID1 array at boot. I booted back into Gentoo and am unsure how to access this virtual drive. I see a /dev/md0 device now on my computer and hoped that might be a reference to my new RAID array so I tried running parted on this device to partition it so I could then set up a file system.

Code:
# parted -a optimal /dev/md0
Warning: Error fsyncing/closing /dev/md0: Input/output error
Retry/Ignore? Ignore


and

Code:
# parted /dev/md0
Warning: Error fsyncing/closing /dev/md0: Input/output error
Retry/Ignore? Ignore


I've seen the Gentoo Software RAID - Gentoo Wiki handbook, but don't believe I need this because I've got a RAID controller in hardware.


Hi,
I have used MegaRaid on a Dell PowerEdge 2950 (4 years ago). There is no need to have "RAID Support" enabled in the Kernel, as hardware RAID is offsetting all work to the hardware. The RAID array is hidden for the operating system. Logical drives are shown as block devices /dev/sdxN.

To configure the MegaRAID, you need to use sys-block/megactl (opensource, there are also proprietary tools to configure the RAID) to configure/modify your array. Please be careful modifying logical drive, where your Gentoo installation lies!

Hope this shines some light on the topic. How to use megactl, please search for megacli howto with your favorite search engine.

BR
Back to top
View user's profile Send private message
AJM
Tux's lil' helper
Tux's lil' helper


Joined: 25 Sep 2002
Posts: 132
Location: Aberdeen, Scotland

PostPosted: Wed Apr 08, 2020 3:57 pm    Post subject: Reply with quote

I'm 99% sure that lots of MegaRAID controllers are real RAID - from memory the Dell PERC controllers are rebadged LSI MegaRAID.

Have you tried the kernel driver at Device Drivers / SCSI Device Support / SCSI low-level drivers / MEGARAID_SAS ?

From past experience, the virtual drive configured via the BIOS should show up just as an ordinary SATA drive, not md.
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2126
Location: Frankfurt, Germany

PostPosted: Wed Apr 08, 2020 4:00 pm    Post subject: Reply with quote

AJM wrote:
I'm 99% sure that lots of MegaRAID controllers are real RAID - from memory the Dell PERC controllers are rebadged LSI MegaRAID.

Have you tried the kernel driver at Device Drivers / SCSI Device Support / SCSI low-level drivers / MEGARAID_SAS ?

From past experience, the virtual drive configured via the BIOS should show up just as an ordinary SATA drive, not md.

That's exactly what I just started to write. But since you already posted it, I can only say that I fully agree.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Wed Apr 08, 2020 5:47 pm    Post subject: Reply with quote

seifn06,

Its in the kernel somewhere. There are several choices in the SCSI and Fusion menu.

Code:
02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2108 [Liberator] (rev 05)
        Subsystem: Broadcom / LSI MegaRAID SAS 9260-4i
is not quite enough information to pinpoint it.

Code:
lspci -nnk
would be good. I don't expect to see a driver but the PCI Vendor and Device IDs will allow the kernel sources to be grepped for your card.

If you set it up in JBOD mode, you can use mdadm and software raid.
Once you have the kernel driver installed, it should show up as a single /dev/sdX device.

Hardware raid comes with a health warning. The on disk data layout is card dependent, so if your raid card dies, you often need an identical card to get your data back.
Then, RAID is not a backup, that's why you have backups.

With kernel raid, you plug the drives into another system and it just works. RAID1 is not very CPU intensive either, so you won't notice.
If you hardware RAID card has battery backup, it beats software raid in a power cut.

--edit --

Wild educated guess is that you need megaraid_sas
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
seifn06
Tux's lil' helper
Tux's lil' helper


Joined: 19 Sep 2004
Posts: 85
Location: Lowell, Michigan

PostPosted: Wed Apr 08, 2020 9:08 pm    Post subject: Reply with quote

Solved! In a nut shell:

(1) In my kernel config, I had to enable the
Code:
Device Drivers > SCSI Device Support >
[*] SCSI low-level drivers


(2) And then once the SCSI low_level drivers was enabled, I had to drill down (again in kernel config) into SCSI low-level drivers and enable
Code:
<*> LSI Logic Legacy MegaRAID driver
<*> LSI Logic MegaRAID SAS RAID module


(3) I compiled the new kernel, installed it and rebooted to a kernel panic which complained about root= not being correctly set.

(4) I booted with my Gentoo install CD and noted that the install CD correctly detected the RAID array at /dev/sdb and my primary OS/root file system on /dev/sda4. I thought maybe I could fix the kernel panic by modifying /etc/fstab (WRONG). I was using UUIDs in my fstab already. (Do UUID's change based on which drive Grub uses as the root device?) While booted with the install CD, I used parted to put a gpt label on the RAID array at /dev/sdb, then make a single partition out of the drive at /dev/sdb1. I then installed an ext4 file system on /dev/sdb1 and rebooted.

(5) The new kernel panicked once it finally recognized my RAID because Grub failed to correctly auto detect root=, apparently. I'm using Grub with UEFI.
So I booted with the Gentoo install CD again and in /etc/default/grub I had to add the following line:
Code:
GRUB_DEVICE=/dev/sdb4


(6) I had to re-install / regenerate my Grub configuration file and reboot:
Code:
# grub-mkconfig -o /boot/grub/grub.cfg


Now my I can boot my kernel without it panicking and I can use the RAID array at /dev/sda1 while OS and primary rootfs are on dev/sdb4. For whatever reason my Gentoo install CD detected my primary/root file system at /dev/sda4 and the RAID array at /dev/sdb1 while the new kernel had reversed the 'a' and 'b'. Can I force Gentoo to recognize the RAID device as /dev/sdb instead of /dev/sda?

Before I posted initially, I thought I had searched through my kernel's .config for "MEGA" prior to enabling these settings and I don't remember seeing the
CONFIG_MEGARAID_LEGACY=y
CONFIG_MEGARAID_SAS=y
that are there now. Do these settings get omitted from the .config file if the SCSI low-level drivers option is not enabled? I would have thought they would just be set to 'n' instead of 'y'.

joanandk, I saw those programs in Portage, but did not need those to get this working and TBH I haven't looked into them yet. Would those programs allow me to monitor my RAID array (i.e. the health)?

AJM and mike155: you were right. Thank you for pointing me in the right direction.

NeddySeagoon, I've seen some of your other posts on the topic of RAID and even before your most recent post here I was prepared to junk the RAID card and go the software RAID route. And I may still do that because I'm concerned about data loss if I lose the RAID controller card. But I wanted to see if I could get this working. I'm using the RAID array for a small office file server and still intend to backup the data onto at least a secondary/external media. My LSI RAID card is a pain in the a** to get into the BIOS to configure -- only possible if I tell my Asus motherboard that the RAID controller is the first boot device and then I have to catch it at boot, press CTRL-H to get into that controller and make RAID configuration changes. I also had to find a PS/2 keyboard (because my LSI card is older and only recognizes PS/2 keyboard input) and though I cannot find it now I know I changed a setting in my ASUS motherboard BIOS for "legacy bios" so it would show the RAID controller's BIOS during the boot sequence just so I could get into the card to configure it.

Also, running the lspci -nnk on my machine confirmed your wild educated guess about the kernel module:
Code:
# lspci -nnk
02:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 05)
        Subsystem: Broadcom / LSI MegaRAID SAS 9260-4i [1000:9260]
        Kernel driver in use: megaraid_sas


Thank you again, everyone, for your speedy input. It is most appreciated!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Wed Apr 08, 2020 9:45 pm    Post subject: Reply with quote

seifn06,

You have gained a lot of experience in the course of that post.
Experience is what you get just after you needed it. :)

When a kernel menu item is off, all the item in the menu are hidden and take their default values.
That's usually off but there are exceptions.
As the options are hidden, they don't appear it the search results either, so you can't find a symbol even if you know its name.
However, there in an undocumented feature in make menuconfig. Press the 'z' key to toggle the display of hidden symbols.
Now they appear in the search results and you can read the help to see why they are not selectable.

/dev/sda, /dev/sdb and so on are allocated as devices are detected. As you discovered, its not always deterministic.
Trying to force the use of fixed kernel allocated names will drive you up the wall. That's not what you do.

fstab is not a problem, you can use UUID, PARTUUID, LABEL or whatever there. Not the /dev/disk-by-* symbolic links as they may be missing when fstab is read.

On the kernel command line you can use UUID, PARTUUID or LABEL, but UUID and LABEL both need the userspace mount command, which compels you to use an initrd.
The kernel can use PARTUUID without any help.
Its root=PARTUUID= in grub.cfg.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7416

PostPosted: Thu Apr 09, 2020 9:55 am    Post subject: Reply with quote

NeddySeagoon wrote:
/dev/sda, /dev/sdb and so on are allocated as devices are detected. As you discovered, its not always deterministic.
Trying to force the use of fixed kernel allocated names will drive you up the wall. That's not what you do.

But it's also not random :)
The livecd put his raid as sdb because the card is after its intel controller in pci list ; so having them both set as module, intel controller will always be loaded first, and takes the sda

And the solution is quiet simple: making lsi drivers Y in kernel, and making intel controller as M ; this way, kernel have no choice than seeing the lsi first (intel will only be seen when udev will trigger it)
And you then don't need any label/UUID trick and/or initrd

selfn06, you might wish to a look at my thread : https://forums.gentoo.org/viewtopic-t-1007788-highlight-.html
Back to top
View user's profile Send private message
seifn06
Tux's lil' helper
Tux's lil' helper


Joined: 19 Sep 2004
Posts: 85
Location: Lowell, Michigan

PostPosted: Thu Apr 09, 2020 2:24 pm    Post subject: Reply with quote

Thank you for the post and link, krinn. The "Common problem: booting from a livecd" in your other post summarized the problem I encountered when my kernel finally recognized my RAID array. I may try re-compiling my kernel with the Intel controller support as module and the LSI MegaRAID as 'y'/built-in and see if that allows my kernel to detect the RAID array as /dev/sdb1.
Many thanks again for everyone's input -- I very much appreciate it!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45435
Location: 56N 3W

PostPosted: Thu Apr 09, 2020 6:36 pm    Post subject: Reply with quote

seifn06,

That's the wrong way round.

What krinn says is correct for a small number of drives but it doesn't scale.
It may well not work if you leave a USB storage device connected.

The PARTUUID or UUID solutions are robust and scale.
LABEL is up to you. The wheel comes off if your labels are not unique.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
joanandk
Tux's lil' helper
Tux's lil' helper


Joined: 12 Feb 2017
Posts: 105

PostPosted: Mon Apr 13, 2020 7:18 pm    Post subject: Reply with quote

seifn06 wrote:
joanandk, I saw those programs in Portage, but did not need those to get this working and TBH I haven't looked into them yet. Would those programs allow me to monitor my RAID array (i.e. the health)?


If a degradation of your array occurs, this will be shown in the dmesg / syslog events by the megaraid_sas module.

seifn06 wrote:
While booted with the install CD, I used parted to put a gpt label on the RAID array at /dev/sdb, then make a single partition out of the drive at /dev/sdb1. I then installed an ext4 file system on /dev/sdb1 and rebooted.


If you want to use EFI booting, you should create at least two partions: EFI system (around 200-250MB) and Linux partition (ext4 in your case). You must declare the EFI system partition as such. If you do not want to go through the hassle of using efibootmgr to set up the boot entry, you could make use of the default fallback of EFI: /efi/boot/bootx64.efi. To make this clear: You must enable EFI stub in your kernel AND define the "built-in kernel command". By defining the kernel command string, you are hard encoding the Gentoo system partition. The compiled kernel then goes into the EFI-system-Partition/efi/boot/bootx64.efi, the rest (kernel modules, Gentoo installation) goes into your ext4 partition. This would free you from using any boot loader (Grub) AND efibootmgr.

I have been using this method some years. It works on all my setups. This method has a draw back: If you new kernel is unable to boot, you will need a live CD to fix your broken system. With a boot loader you have the possibility to define a fallback kernel (which is the previous kernel) and boot into it if the new kernel fails.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum