Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
filesystem planning: zfs vs ext4/mdadm/lvm and ssd vs hdd
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
mek42
n00b
n00b


Joined: 18 Feb 2015
Posts: 12

PostPosted: Tue Feb 24, 2015 12:52 pm    Post subject: filesystem planning: zfs vs ext4/mdadm/lvm and ssd vs hdd Reply with quote

Planning a first time Gentoo install this weekend on freshly built box for use as a family workstation. I'm going to have two each 128 Gb SATA SSD and 450 Gb SAS near-line 15krpm drives. I want to run each pair as RAID1. I'm thinking about lvm for partition resizing. The system will have 64 Gb RAM.

I have never used Gentoo, I have Arch on my laptop without bumblebee working and prior to that was a long time OpenSUSE user, including administrating a personal 3-node dual-cpu single-core Opteron Beowulf cluster back in grad school. I'm not afraid to do some reading and learn some things, but I am not a strong admin generalist. I have also never used RAID at all, much less mdadm, lvm or zfs (other than possibly incidental to playing with a no cost Solaris install).

First, am I reading correctly that if I use zfs I can have easily resizeable partitions and RAID1 without needing lvm and mdadm? If so, what are the pros and cons of zfs vs ext4 with lvm and mdadm? Should I even bother with lvm?

How do I decide between initramfs and dracut?

For partitioning I am thinking of:

ssd: /boot / /usr /opt - making sure to allocate only 3/4 the space of the drive
hdd: /home /var /etc /tmp

Am I neglecting anything important in this plan?

Thanks!
Back to top
View user's profile Send private message
DawgG
l33t
l33t


Joined: 17 Sep 2003
Posts: 824

PostPosted: Tue Feb 24, 2015 2:29 pm    Post subject: Reply with quote

if you spend so much on hardware why don't you get a real (=hardware) raid-controller?
other than that, i would personally use btrfs for everything except /boot (i still use grub-leg on ext2).
everything will be very flexible with volumes etc. (you don't need partitions anymore) and you'll also get nice stuff like snapshots etc.
what i am still not quite happy with is that you sometimes have to run btrfs balance.
_________________
DUMM KLICKT GUT.
Back to top
View user's profile Send private message
BlueFusion
Guru
Guru


Joined: 08 Mar 2006
Posts: 371

PostPosted: Tue Feb 24, 2015 3:37 pm    Post subject: Reply with quote

Consider Btrfs. ZFS is a great filesystem but I find Btrfs stabilizing quite nicely. I've been running it for years with no issues and I find its performance and features on par with ZFS.

If using grub2, you don't need any partitions on any drives used. If you use legacy grub, you will need to have a separate /boot partition.
Back to top
View user's profile Send private message
mek42
n00b
n00b


Joined: 18 Feb 2015
Posts: 12

PostPosted: Tue Feb 24, 2015 3:52 pm    Post subject: Reply with quote

For either btrfs or zfs (considering both, still leaning toward zfs) how do I tell them to keep high write directories on the hdd and off the ssd?
Back to top
View user's profile Send private message
BlueFusion
Guru
Guru


Joined: 08 Mar 2006
Posts: 371

PostPosted: Tue Feb 24, 2015 4:09 pm    Post subject: Reply with quote

Either filesystem you use, just use multiple separate filesystems. Make the SSDs a separate RAID1 from the HDDs.
Back to top
View user's profile Send private message
mek42
n00b
n00b


Joined: 18 Feb 2015
Posts: 12

PostPosted: Tue Feb 24, 2015 4:38 pm    Post subject: Reply with quote

If I use btrfs or zfs as two filesystems, will I still need to use some sort of initial ramdisk?
Back to top
View user's profile Send private message
BlueFusion
Guru
Guru


Joined: 08 Mar 2006
Posts: 371

PostPosted: Wed Feb 25, 2015 5:58 am    Post subject: Reply with quote

Yes you will. I just compile the kernel and initamfs with genkernel.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Wed Feb 25, 2015 2:09 pm    Post subject: Reply with quote

DawgG wrote:
if you spend so much on hardware why don't you get a real (=hardware) raid-controller?
other than that, i would personally use btrfs for everything except /boot (i still use grub-leg on ext2).
everything will be very flexible with volumes etc. (you don't need partitions anymore) and you'll also get nice stuff like snapshots etc.
what i am still not quite happy with is that you sometimes have to run btrfs balance.


raid setups and controller DO NOT safe your data against single/multi-bit corruption

if I read correctly

so in the long run - there's no alternative other than filesystems which offer checksumming features on metadata and data
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
mek42
n00b
n00b


Joined: 18 Feb 2015
Posts: 12

PostPosted: Wed Feb 25, 2015 2:56 pm    Post subject: Reply with quote

DawgG wrote:
if you spend so much on hardware why don't you get a real (=hardware) raid-controller?
other than that, i would personally use btrfs for everything except /boot (i still use grub-leg on ext2).
everything will be very flexible with volumes etc. (you don't need partitions anymore) and you'll also get nice stuff like snapshots etc.
what i am still not quite happy with is that you sometimes have to run btrfs balance.


I haven't spent so much. Everything other than the ssd drives are from the used market. With hardware RAID, if the RAID fails I need to get identical controller card. Also, I'd rather spend time researching 3 or 4 software choices than researching capabilities and expected longevities of however many used RAID cards are in my budget. Further, RAID doesn't take much cpu time, I'll have 16 cores and 64 Gb EAM - the software RAID may be faster than an older card.
Back to top
View user's profile Send private message
DawgG
l33t
l33t


Joined: 17 Sep 2003
Posts: 824

PostPosted: Wed Feb 25, 2015 3:44 pm    Post subject: Reply with quote

Quote:
raid setups and controller DO NOT safe your data against single/multi-bit corruption

how true. of course the smart thing is using btrfs on a hw-raid and doing regular backups/snapshots.

Quote:
I haven't spent so much.

it just read quite expensive.
i've just had too many consumer-grade hdds fail whereas since SCSI was replaced by SAS i cannot remember one single really dangerous problem in that area.

Quote:
just compile the kernel and initamfs with genkernel.

you don't need an initramfs. if you compile everything "into" the kernel (eg btrfs and raid-controller-support) an initramfs is only needed for boot-time bling-bling before the local filesystems are mounted.

GOOD LUCK!
_________________
DUMM KLICKT GUT.
Back to top
View user's profile Send private message
mek42
n00b
n00b


Joined: 18 Feb 2015
Posts: 12

PostPosted: Wed Feb 25, 2015 6:32 pm    Post subject: Reply with quote

Thank you for the well wishes. My cost was about 1100 USD shipped, that does not include tower case or fans, which are being reused. Which reminds me I need to quickly figure out how to log cpu temperatures, as I have read it is ok to use active socket 940 heatsinks on socket F, but would like to see some data. The costs were about 200 each for motherboard, RAM, and video card (a single nVidea 760, mobo is SLI so maybe second later), the cpus (Opteron 8389) came to 100 total, the total of all four drives was 250 and I also needed to get a power supply, SAS cables and cpu paste. I also bought a SATA optical drive, which is almost optional in today's world of downloading.
Back to top
View user's profile Send private message
Kethreveris
n00b
n00b


Joined: 16 Feb 2009
Posts: 16
Location: Rethwallen

PostPosted: Fri Mar 20, 2015 3:35 am    Post subject: BTRFS verses ZFS Reply with quote

Linux BTRFS is a great file system. I use it on my workstation, old netbook and new netbook, (all
with Gentoo).

BTRFS can be compiled into the kernel, so you DON'T need an initramfs. You can even specify
certain BTRFS things on the kernel line in grub, (even with Grub legacy 0.9xx).

For example, I like and have alternate boot environments for ROOT-FS. So for Gentoo updates,
I create a snapshot of "/", then make it an ABE, and boot off of it. Thus, if my Gentoo updates
go south, (rare, but it happens), I have a reboot as an easy fix. (Though I DO have backups.)
But, if you go this route, make separate BTRFS sub-volumes for /usr/portage/distfiles, /home,
and /usr/src. Plus other shared trees on the OS mirror pair. And have "/boot" an EXT4 file
system, (for TRIM / DISCARD support). That way they can be shared between ABEs.


All that said, ZFS is a cleaner implementation, (at least for Solaris & FreeBSD/NAS). A few things
I REALLY like with ZFS, that BTRFS does not have, or as nicely;
  • Reservations
  • Quotas
  • File copy count on single disk pools, (a type of data mirroring, not disk mirroring)
  • df output give correct size

Wish BTRFS Quotas showed up under "df" command. (Unless I am doing BTRFS quotas wrong...)

My company uses Solaris 10 & 11, some with ZFS, (OS or App, or both). So I have lots of experiance
with ZFS. All the way back to the first official Solaris 10 release with ZFS, (update 2, June 2006).

Even saw a double disk failure on a root pool, (OS pair of disks). ZFS calmly told me about it, let
me know what file was lost, and kept on running. Turns out a worthless support file, nothing I would
miss. After the disks were replaced, (one at time), I simply removed the damaged file and restored
it. No down time.
_________________
Kethreveris, (aka Kethrery)


Last edited by Kethreveris on Sat Apr 11, 2015 10:45 pm; edited 2 times in total
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7195

PostPosted: Fri Mar 20, 2015 4:26 am    Post subject: Reply with quote

mek42 wrote:
Further, RAID doesn't take much cpu time

It may not takes "that much" cpu time, but it will take it when you need it the most.

It's as simple as that: how much time it take your brain to repeat "hello". Few no? (well i hope)
Now tell me how much time it will take to repeat "hello" while you are reading a book?
With hardware raid: you have a person saying "hello" while you read the book. And yes, it makes a lot more difference than the misconception of "little time" taken.

You don't busy drives with a stupid loop, you busy drives as the result of your cpu doing heavy work: so the "little cpu time" is taken from your busy cpu.
Back to top
View user's profile Send private message
Kethreveris
n00b
n00b


Joined: 16 Feb 2009
Posts: 16
Location: Rethwallen

PostPosted: Sun Mar 22, 2015 4:41 pm    Post subject: Reply with quote

Yes, hardware RAID does reduce the overhead. Software RAID-1 takes more CPU & memory than
a hardware RAID-1.

Their are plenty of times that hardware RAID makes sense. Especially in servers. Desktops not so
much in my opinion.

However, it depends on the use. For example;
  • Simple things, browsers, OpenOffice and such, software RAID does not impact performance much.
  • Video editting, number crunching, (aka scientific modeling with large data sets on disk), or alot of compiling code, then hardware RAID may make sense.
  • Gaming really likes hardware RAID, (assuming you can use your hard or flash drives for the game and it's storage).

If going hardware RAID route, make sure you get a decent controller. Especially not one of the "fakeraid"
cards out their.


Even with hardware RAID, their are uses for both ZFS and BTRFS. Snapshots, writable clones, data
integrity and sub-volume management being high on my list.

Using BTRFS and writable clones, (aka writable snapshots), makes Gentoo updates MUCH nicer. Before
BTRFS I had 2 root partitions, with Grub pointing to both, but with my current as Grub's default. If I needed
to backout an update, a simple reboot to the other root partition. BTRFS allows me to have as many ABEs,
(Alternate Boot Environments), as I want. I usually keep the last 3 or 4, just in case.

But, ZFS and BTRFS add overhead not found with EXT2-4 or XFS, (or many other file systems). So if speed
is your goal, hardware RAID with caching controller and a lighter weight file system is best.

I personally am willing to sacrifice a bit of speed to get some of the advanced features of ZFS or BTRFS.
_________________
Kethreveris, (aka Kethrery)
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7316
Location: Saint Amant, Acadiana

PostPosted: Sun Mar 22, 2015 4:52 pm    Post subject: Reply with quote

ZFS on hardware RAID makes not much sense, ZFS has no clue what is going on in that case, why use it at all.
Does software RAID slow you down? http://skrypuch.com/raid/
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
dab_s_bad
n00b
n00b


Joined: 08 May 2008
Posts: 54
Location: Toledo City, Cebu, Philippines

PostPosted: Sun Mar 22, 2015 9:47 pm    Post subject: Reply with quote

greetings to all and am sorry for butting-in,

Anyways, I'm still trying to learn my way to either LVM or Btrfs to gain the benefit of having a "resize-able" filesystem/partition option for my system.
Seeing those logical volumes and subvolumes do wonders on shrink/grow made me interested but I don't find a good example for a single disk on either laptop or desktop [all info on google seems to example multiple disks].


for Kethreveris,

since you have mentioned that you use Btrfs on netbook(s) [old/new ones], can you share relevant information on fstab and other things so that I can try to examine and tryout for myself?

or to anyone that have a good example (lvm/btrfs) for a single disk laptop/desktop setup, I would appreciate it, thanks.


best regards,
dab_s_bad
Back to top
View user's profile Send private message
Kethreveris
n00b
n00b


Joined: 16 Feb 2009
Posts: 16
Location: Rethwallen

PostPosted: Sat Apr 04, 2015 8:07 pm    Post subject: Reply with quote

dab_s_bad wrote:
...
for Kethreveris,

since you have mentioned that you use Btrfs on netbook(s) [old/new ones], can you share relevant information on fstab and other things so that I can try to examine and tryout for myself?

or to anyone that have a good example (lvm/btrfs) for a single disk laptop/desktop setup, I would appreciate it, thanks.


best regards,
dab_s_bad

Hi,
Here is my setup, simplified a bit. (I tend to keep old kernels, old Grub entries and 4 or 5 alternate boot environments.)

BTRFS is not necessarily good for re-sizing. Quota groups are a bit clumsy, but do work. Anyway, I use BTRFS mostly
for ABEs, (Alternate Boot Environments), and management of my various file systems.

If you build all BTRFS required features into the kernel, (not as modules), you can avoid initram disks.

One feature that is not clear, is that the newer ABE, (subvolid=1227), is a writable snap shot of the prior ABE, (subvolid=1224).
Thus, the amount of space used is less than 2 plain ABEs. In fact, space used for 1227 is trivial until after I have updated the
newer ABE.

Last, if you use BTRFS for ABEs, make sure you have certain things separate. I have shown "/home", and the Portage
Distributiion files as they can be shared between ABEs. Same with "/usr/src", (which is not shown below).

I have a plain text document to make Linux ABEs using BTRFS. Any interest for me to post it here?


Code:
# fdisk -l
...
Device     Boot   Start        End    Sectors   Size Id Type
/dev/sda1          2048     104447     102400    50M 83 Linux
/dev/sda2        104448    6453247    6348800     3G 82 Linux swap / Solaris
/dev/sda3       6453248 1953523711 1947070464 928.4G 83 Linux


sys-boot/grub-0.97-r14
/boot/grub/grub.conf
title X200CA linux-3.16.5-gentoo.5 SVID=1227 root=sda3
          root (hd0,0)
          kernel /linux-3.16.5-gentoo.5 rootflags=subvolid=1227 root=/dev/sda3

title X200CA linux-3.16.5-gentoo.5 SVID=1224 root=sda3
          root (hd0,0)
          kernel /linux-3.16.5-gentoo.5 rootflags=subvolid=1224 root=/dev/sda3
...


/etc/fstab:
/dev/sda1   /boot                      ext4     journal_checksum,async,relatime,nodiratime,discard 1 1
/dev/sda2   none                       swap     sw,pri=1                                           0 0
/dev/sda3   /                          btrfs    async,relatime,nodiratime,discard,subvolid=1227    1 0
/dev/sda3   /home                      btrfs    async,relatime,nodiratime,discard,subvolid=262     1 0
/dev/sda3   /usr/portage/distfiles     btrfs    async,relatime,nodiratime,discard,subvolid=266     1 0

/dev/sda3   /abe/subvolid_0            btrfs    noauto,async,relatime,nodiratime,discard,subvolid=0    1 0
/dev/sda3   /abe/root_20150207         btrfs    noauto,async,relatime,nodiratime,discard,subvolid=1224 1 0
/dev/sda3   /abe/root_20150314         btrfs    noauto,async,relatime,nodiratime,discard,subvolid=1227 1 0


# btrfs sub list /
ID 262 gen 119066 top level 5 path home
ID 266 gen 106539 top level 5 path usr_portage_distfiles
ID 1224 gen 103709 top level 5 path root_20150207
ID 1227 gen 119061 top level 5 path root_20150314

_________________
Kethreveris, (aka Kethrery)


Last edited by Kethreveris on Sat Apr 04, 2015 8:20 pm; edited 2 times in total
Back to top
View user's profile Send private message
Kethreveris
n00b
n00b


Joined: 16 Feb 2009
Posts: 16
Location: Rethwallen

PostPosted: Sat Apr 04, 2015 8:18 pm    Post subject: Reply with quote

Jaglover wrote:
ZFS on hardware RAID makes not much sense, ZFS has no clue what is going on in that case, why use it at all.
Does software RAID slow you down? http://skrypuch.com/raid/

Using ZFS on top of hardware RAID does make some sense for File System management.
Think of it as replacing LVM, (Logical Volume Manager), or Symantec / Veritas Volume Manager.

We can create as many file systems as needed, at any mount point. And use Reservations,
(reserved disk space), or Quotas, (FS growth limits), on each file system. This means things
like limiting "/home" or even "/home/user_space_hog" is not only possible, but easy.

Last, using ZFS on top of hardware RAID is useful for ABEs, (Alternate Boot Environments).
Basically you make a new ABE, patch it, then reboot to activate it. And if not good, reboot
back to the prior ABE.
_________________
Kethreveris, (aka Kethrery)
Back to top
View user's profile Send private message
dab_s_bad
n00b
n00b


Joined: 08 May 2008
Posts: 54
Location: Toledo City, Cebu, Philippines

PostPosted: Sun Apr 05, 2015 6:55 am    Post subject: Reply with quote

thanks for the example
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Sun Apr 05, 2015 3:56 pm    Post subject: Reply with quote

dab_s_bad wrote:
since you have mentioned that you use Btrfs on netbook(s) [old/new ones], can you share relevant information on fstab and other things so that I can try to examine and tryout for myself?

or to anyone that have a good example (lvm/btrfs) for a single disk laptop/desktop setup, I would appreciate it, thanks.

Well it's not single-disk (I'd moved installation to a new machine, keeping the old drive), but this should show you LVM.

Note that I'm using /dev/mapper/vg-lv as opposed to /dev/vg/lv links which aren't available in early startup; most people use the latter, but the linked setup is for udev without initramfs (which requires use of the /dev/mapper names.)

I've run out of space on /usr before, so I like the fallback option of resizing logical volumes (there's spare in each of the volume groups mentioned) and being able to separate partitions (and thus drives if need be) into different physical volumes which are in separate vgs.
In my case the var-foo ones are separated, as are gent2 vs debian (experimental group for w/e OS I want to play with.)

I especially like that lvm (or device-mapper more generally) is below the FS, not part of it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum