Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Linux 5.'ish - nfs with proto=rdma fails but proto=tcp works
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1775
Location: KUUSANKOSKI, Finland

PostPosted: Thu Apr 09, 2020 8:43 pm    Post subject: Linux 5.'ish - nfs with proto=rdma fails but proto=tcp works Reply with quote

As the title says. I can't get rdma working anymore with my nfs shares.
It worked with 4.19. I guess I'm missing something because this system is amd64 while all 5.6 gentoo-sources are ~amd64. I, however, tried with later nfs-utils, but the error is still the same.

Before downgrading to latest amd64 gentoo-sources, I thought to ask if anyone else here has the same problem (I doubt. Not many use rdma... :|)

proto=rdma:
# mount --verbose -t nfs -s -o ro,soft,proto=rdma,port=20049 10.0.11.1:/pub /srv/pub
mount.nfs: timeout set for Thu Apr  9 23:41:30 2020
mount.nfs: trying text-based options 'soft,proto=rdma,port=20049,sloppy,vers=4.2,addr=10.0.11.1,clientaddr=10.0.11.2'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'soft,proto=rdma,port=20049,sloppy,vers=4,minorversion=1,addr=10.0.11.1,clientaddr=10.0.11.2'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'soft,proto=rdma,port=20049,sloppy,vers=4,addr=10.0.11.1,clientaddr=10.0.11.2'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'soft,proto=rdma,port=20049,sloppy,addr=10.0.11.1'
mount.nfs: mount(2): Permission denied
mount.nfs: an incorrect mount option was specified
proto=tcp:
# mount --verbose -t nfs -s -o ro,soft,proto=tcp,port=2049 10.0.11.1:/pub /srv/pub
mount.nfs: timeout set for Thu Apr  9 23:40:30 2020
mount.nfs: trying text-based options 'soft,proto=tcp,port=2049,sloppy,vers=4.2,addr=10.0.11.1,clientaddr=10.0.11.2'
The thing is... mount utility (or mount.nfs) doesn't even try to mount the filesytem with proto=rdma.

I bet this is kernel related, but... I haven't gotten far. I've triple checked my kernel config. Feel free to ask if some CONFIG_* is set or not. I'll grep it for you.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1775
Location: KUUSANKOSKI, Finland

PostPosted: Sun Apr 12, 2020 6:13 pm    Post subject: Reply with quote

I've now settled to latest stable kernel, 5.4.28.

I've changed nfs and infiniband configs to modular and to built-in: no change.

If I add rdma or proto=rdma to the mount options the mounting fails, with the same error as always.

While all this happening, my server has been untouched in terms of updating or changing any config. I've tried to restart nfs server there in hopes that it would solve the problem, but no dice.

I know rdma is working because I can run rdma benchmarks via qperf between my desktop and server.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Zucca
Veteran
Veteran


Joined: 14 Jun 2007
Posts: 1775
Location: KUUSANKOSKI, Finland

PostPosted: Sun Apr 12, 2020 6:41 pm    Post subject: Reply with quote

Looks like some syscall fails...
Code:
# LIBMOUNT_DEBUG="all" LIBBLKID_DEBUG=all mount.nfs 10.0.11.1:/pub /srv/pub -v -s -o ro,proto=rdma
19810: libmount:     INIT: library debug mask: 0xffff
19810: libmount:     INIT: library version: 2.33.2
19810: libmount:     INIT:     feature: btrfs
19810: libmount:     INIT:     feature: namespaces
19810: libmount:     INIT:     feature: assert
19810: libmount:     INIT:     feature: debug
Available "LIBMOUNT_DEBUG=<name>[,...]|<mask>" debug masks:
   all      [0xffff] : info about all subsystems
   cache    [0x0004] : paths and tags cache
   cxt      [0x0200] : library context (handler)
   diff     [0x0400] : mountinfo changes tracking
   fs       [0x0040] : FS abstraction
   help     [0x0001] : this help
   locks    [0x0010] : mtab and utab locking
   loop     [0x2000] : loop devices routines
   options  [0x0008] : mount options parsing
   tab      [0x0020] : fstab, mtab, mountinfo routines
   update   [0x0080] : mtab, utab updates
   utils    [0x0100] : misc library utils
   monitor  [0x0800] : mount tables monitor
   btrfs    [0x1000] : btrfs specific routines
19810: libmount:      CXT: [0x55d28e3d52c0]: ----> allocate
19810: libmount:      CXT: [0x55d28e3d52c0]: enabling flag 0020
19810: libmount:      CXT: [0x55d28e3d52c0]: enabling flag 2000000
19810: libmount:      CXT: [0x55d28e3d52c0]: initialized for [u]mount.<type> helper [rc=0]
19810: libmount:      CXT: [0x55d28e3d52c0]: enabling flag 0010
19810: libmount:      CXT: [0x55d28e3d52c0]: enabling flag 0008
19810: libmount:    CACHE: [0x55d28e3d5570]: alloc
19810: libmount:    CACHE: [0x55d28e3d5570]: canonicalize path /srv/pub
19810: libmount:    CACHE: [0x55d28e3d5570]: add entry [ 1] (path): /srv/pub: /srv/pub
19810: libmount:      CXT: [0x55d28e3d52c0]: use default optsmode
19810: libmount:      CXT: [0x55d28e3d52c0]: OPTSMODE: ignore=0, append=0, prepend=1, replace=0, force=0, fstab=1, mtab=1
19810: libmount:      CXT: [0x55d28e3d52c0]: fstab not required -- skip
19810: libmount:      CXT: [0x55d28e3d52c0]: mount: preparing
19810: libmount:      CXT: [0x55d28e3d52c0]: OPTSMODE: ignore=0, append=0, prepend=1, replace=0, force=0, fstab=1, mtab=1
19810: libmount:      CXT: [0x55d28e3d52c0]: fstab not required -- skip
19810: libmount:      CXT: [0x55d28e3d52c0]: merging mount flags
19810: libmount:      CXT: [0x55d28e3d52c0]: final flags: VFS=00000001 user=00000000
19810: libmount:      CXT: [0x55d28e3d52c0]: mount: evaluating permissions
19810: libmount:      CXT: [0x55d28e3d52c0]: mount: fixing optstr
19810: libmount:      CXT: [0x55d28e3d52c0]: mount: fixing vfs optstr
19810: libmount:      CXT: applying 0x00000001 flags to 'ro'
19810: libmount:      CXT: new optstr 'ro'
19810: libmount:      CXT: [0x55d28e3d52c0]: mount: fixing user optstr
19810: libmount:      CXT: applying 0x00000000 flags to '(null)'
19810: libmount:      CXT: new optstr '(null)'
19810: libmount:      CXT: [0x55d28e3d52c0]: fixed options [rc=0]: vfs: 'ro' fs: 'proto=rdma' user: '(null)', optstr: 'ro,proto=rdma'
19810: libmount:      CXT: [0x55d28e3d52c0]: preparing source path
19810: libmount:      CXT: [0x55d28e3d52c0]: preparing target path
19810: libmount:      CXT: [0x55d28e3d52c0]: final target '/srv/pub'
19810: libmount:      CXT: [0x55d28e3d52c0]: preparing fstype
19810: libmount:      CXT: [0x55d28e3d52c0]: FS type: nfs [rc=0]
mount.nfs: timeout set for Sun Apr 12 21:43:33 2020
mount.nfs: trying text-based options 'proto=rdma,sloppy,vers=4.2,addr=10.0.11.1,clientaddr=10.0.11.2'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'proto=rdma,sloppy,vers=4,minorversion=1,addr=10.0.11.1,clientaddr=10.0.11.2'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'proto=rdma,sloppy,vers=4,addr=10.0.11.1,clientaddr=10.0.11.2'
mount.nfs: mount(2): Invalid argument
mount.nfs: trying text-based options 'proto=rdma,sloppy,addr=10.0.11.1'
mount.nfs: mount(2): Permission denied
mount.nfs: an incorrect mount option was specified
19810: libmount:      CXT: [0x55d28e3d52c0]: syscall status set to: -1
19810: libmount:      CXT: [0x55d28e3d52c0]: prepare update
19810: libmount:      CXT: [0x55d28e3d52c0]: utab path initialized to: /run/mount/utab
19810: libmount:      CXT: [0x55d28e3d52c0]: checking for writable tab files
19810: libmount:    UTILS: utab: /run/mount/utab
19810: libmount:    UTILS: try write /run/mount/utab dir: /run/mount
19810: libmount:    UTILS:  access OK [/run/mount]
19810: libmount:      CXT: [0x55d28e3d52c0]: skip update: syscall failed [status=-1]
19810: libmount:      CXT: [0x55d28e3d52c0]: don't update: no update prepared
19810: libmount:      CXT: [0x55d28e3d52c0]: <---- reset [status=0] ---->
19810: libmount:       FS: [0x55d28e3d53f0]: free [refcount=0]
19810: libmount:    CACHE: [0x55d28e3d5570]: free [refcount=0]
19810: libmount:      CXT: [0x55d28e3d52c0]: Setting (null) as target namespace
19810: libmount:      CXT: [0x55d28e3d52c0]: <---- free


The "Permission denied" comes when mount.nfs tries to mount with nfs version that required absolute path on the server side. It does not appear if I pass vers=4.2 (for example) to the mount options.
_________________
..: Zucca :..

Code:
ERROR: '--failure' is not an option. Aborting...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum