Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rootfs over nfs4 saga
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
redwood
Apprentice
Apprentice


Joined: 27 Jan 2006
Posts: 286

PostPosted: Tue May 03, 2016 2:09 am    Post subject: rootfs over nfs4 saga Reply with quote

Hi,

I have a NUC box which I would like to be able to boot diskless. In order to test the hardware out, I installed a spare 16G M.2 NGF SSD and installed sabayon kde-plasma which just barely fit. Everything works fine. I decided to then mount /home/{users} over nfsv3 and then discovered that plasma doesn't work correctly without filelocking which requires nfsv4. So I switched my nfs server over to nfsv4 (but still kept nfsv3 for some legacy boxes which I can't upgrade without purchasing new license keys).

Anyhow, the nuc is working fine booting rootfs from local SSD and then mounting /home/{users} over nfsv4. But on a recent system upgrade, I ran out of disk space. So I could buy a bigger SSD or go with my original plan which was to have diskless NUC boxes. I already had a tftp/pxe server, so I thought I could unpack a tarball of my NUC box to a folder and export it over nfsv4 to my NUC box. Thought it would take about an hour :LOL.

There is an old Gentoo wiki about setting up diskless nodes using nfsv3, syslinux (pxe) and/or etherboot:
https://wiki.gentoo.org/wiki/Diskless_nodes

Arch has a newer guide using nfsv3 + grub2
https://wiki.archlinux.org/index.php/Diskless_system

I followed the Gentoo guide but switched to grub2 (core.0) instead of pxelinux.0
I still need tftp + pxe for installing firmware upgrades to various appliances, phones, etc
so I thought I could combine the boot images into a directory /tftpboot/pxe/{pxelinux,boot}
then discovered that after running
Code:
grub2-mknetdir --net-directory=./pxe
that
grub really couldn't find my kernels unless I made a symlink /tftpboot/boot -> ./pxe/boot

While I was trying to sort out why I was staring at a grub prompt, I ran across a good article on grub
http://cvpcs.org/projects/linux/pxegen
and decided to generate my own grub.pxe instead of the core.0 (made by grub2-mknetdir)
but my custom grub.pxe image fared worse than the core.0


The Sabayon kernels lacked ROOT_NFS support and also used modules for the nics, so I emerged
gentoo-sources-4.1.15-r1 and compiled in ROOT_NFS and my nic driver.
Since Sabayon uses systemd, I need to use an initramfs which I created with genkernel.

The tricky part of my grub configuration was figuring out what grub calls my root (it's not /dev/nfs)

Code:

menuentry 'CustomKernel'{
load_video
set gfxpayload=keep
set root=(pxe)
linux /bzImage ip=dhcp root=/dev/nfs nfsroot=master:/nuc,nfsvers=4 ro splash verbose
initrd initramfs
}



Now my NUC client pxe boots with grub2 and loads linux, then gets an initramfs
and all goes fine EXCEPT my rootfs is left mounted ro (and a lot of systemd startup services
need to be able to write to rootfs). I can of course remount

Code:
mount.nfs4 -t nfs -o remount,rw master:/nuc /


but systemd services are borked.

I guess there are several solutions:
1) Give up on diskless nodes and buy a decent sized SSD
2) switch back to nfsv3 for rootfs and mount /home using nfsv4 (to keep plasma happy)
3) fix the initramfs so that it works properly with nfsv4.

I unpacked the initramfs and see that /bin/mount is just a symlink to busybox (presumably using nfsv3)
During bootup I see message about problem with "NEWROOT" (but the screen goes by too fast)

Grep'ing though the unpacked initramfs for NFS, I find:
Code:

./etc/initrd.d/00-nfs.sh:    NFSOPTIONS=${NFSROOT#*,}
./etc/initrd.d/00-nfs.sh:    if [ "${NFSOPTIONS}" = "${NFSROOT}" ]; then
./etc/initrd.d/00-nfs.sh:        NFSOPTIONS="${DEFAULT_NFSOPTIONS}"
./etc/initrd.d/00-nfs.sh:        NFSOPTIONS="${DEFAULT_NFSOPTIONS},${NFSOPTIONS}"
./etc/initrd.d/00-nfs.sh:    good_msg "NFS options: ${NFSOPTIONS}"
./etc/initrd.d/00-nfs.sh:    mount -t nfs -o ${NFSOPTIONS} "${NFSROOT}" "${path}"
./etc/initrd.defaults:DEFAULT_NFSOPTIONS="ro,nolock,rsize=1024,wsize=1024"


Has anybody patched genkernel or made a custom initramfs adding nfsv4 support?
MIght it be as simple as adding mount.nfs4 to the initramfs /bin and changing the
"mount -t nfs -o ${NFSOPTIONS} ..." line in 00-nfs.sh above?

Thanks for any advice, thoughts.


Last edited by redwood on Tue May 03, 2016 10:09 pm; edited 1 time in total
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7059

PostPosted: Tue May 03, 2016 9:23 am    Post subject: Reply with quote

Quote:
Under Linux 2.6.32 and later kernel versions, mount.nfs can mount all NFS file system versions. Under earlier Linux kernel
versions, mount.nfs4 must be used for mounting NFSv4 file systems while mount.nfs must be used for NFSv3 and v2.


you can pass nfsvers= and vers= as nfs options if you need to specify the version.
Back to top
View user's profile Send private message
redwood
Apprentice
Apprentice


Joined: 27 Jan 2006
Posts: 286

PostPosted: Tue May 03, 2016 1:01 pm    Post subject: Reply with quote

Quote:
you can pass nfsvers= and vers= as nfs options if you need to specify the version.


Yes, that's what I've done in my grub.cfg and fstab, but I think the problem is the initrc runscripts which apparently use nfsv3 by default when mounting & pivoting the rootfs. I think mount is provided by busybox in my initramfs, but once my nfs rootfs is mounted, the real non-busybox mount.nfs4 is available in /bin, so maybe I don't need to add mount.nfs4 to my initramfs, but rather just modify the bootup initscript to mount,pivot my nfs root using mount.nfs4 rather than the default nfs3.

The kernel is mounted ro at boot then at some point (the pivot?) is mounted rw, which fails in my case. Maybe has nothing to do with initramfs, right? since once my kernel is mounted my real /bin is available for openrc or systemd bootup.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2571

PostPosted: Tue May 03, 2016 8:22 pm    Post subject: Reply with quote

Moved from Portage & Programming to Unsupported Software as Sabayon is not Gentoo.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7059

PostPosted: Tue May 03, 2016 10:46 pm    Post subject: Reply with quote

I know nothing on sabayon, systemd, and initram.
But this might help you https://forums.gentoo.org/viewtopic-p-7592928.html#7592928
Back to top
View user's profile Send private message
redwood
Apprentice
Apprentice


Joined: 27 Jan 2006
Posts: 286

PostPosted: Wed May 04, 2016 4:20 am    Post subject: Reply with quote

Thanks for the link.


I changed my grub, fstab and exports to use nfs3 instead of nfs4 thinking that would fix the problem, but it didn't.
So I think I'll just change everything back to pure nfs4.


In the meantime I've just manually remounted rw
Code:
mount -t nfs -o remount,rw master:/tftpboot/sabayon /


Then I looked through systemd's journal and kept seeing errors like
Code:

Error getting authority
Error initializing authority
Error calling StartServiceByName

which seems to indicate a problem with polkit. So I reinstalled the polkit package.
And after
Code:
systemctl restart polkit; systemctl status polkit
the policykit seems to be running again.
And I was then able to restart some failed services (but not all since some can only be called as a dependency)

I saw some segfaults when trying to start sddm, so I removed that package and replaced with lightdm, and after starting
that service, was able to login with fluxbox (yeah!)

Looking through the journal, the big error was the one by systemd-remount-fs.service
which the log said there was an error remounting my ro root as rw because of an incorrect mount option.
(Even though I can manually remount my rootfs as rw using those same mount options)

So for good measure I decided to reinstall systemd.

The installation of systemd complained about a few kernel configuration options which weren't set properly to work with systemd.
All the incorrect options were basically due to the very first option in menuconfig -- the one specific for gentoo tweaks for setting up systemd and/or openrc.

I started this project a month ago or so but got interrupted by tax season. So I'm not sure now how I configured my kernel.
I had planned on taking a working sabayon config file, running
Code:
make menuconfig
and compiling in my NUC's nic and NFS_ROOT support. Normally when I compile a kernel I compile in just what I need and don't bother compiling unnecessary drivers, but sabayon pretty much has compiled modules for every driver, and since their kernel worked, I didn't go back through the menuconfig line by line. At one time sabayon included the option of systemd or openrc but then in recent releases has removed support for openrc. In any event, in my kernel's config, under Gentoo options, I only had openrc checked and not systemd. So I've now checked systemd (and unchecked openrc) and am recompiling. Then I'll reinstall systemd and see if it complains about kernel options again.

Fingers crossed, maybe setting the kernel option ->Gentoo->systemd will fix the nfs remount issue.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum