Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Problems with kvm, bridged interface, and nfs server on host
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
rich0
Developer
Developer


Joined: 15 Sep 2002
Posts: 160

PostPosted: Fri Dec 20, 2013 7:54 pm    Post subject: Problems with kvm, bridged interface, and nfs server on host Reply with quote

I figured I'd try posting this here and see if anyone has any suggestions.

I was happily running an NFS server (NFS v3), supporting various clients including some that use an NFS root. I also used this box as a host for virtualbox. The box was running gentoo-sources-3.10.17.

Then I decided to migrate from virtualbox to KVM. In order to run KVM with bridged networking a bridged interface must be set up. I set this up using openrc to manage it:
config_eth0="null"
bridge_brkvm="eth0"
config_brkvm="192.168.0.5/24"
brctl_brkvm="setfd 0
sethello 30
stp off"

(I got that recipe off of some kvm site, perhaps Gentoo-related, though it is all a blur now.)

That worked fine for getting VMs to work, but I noticed that my NFS clients were unable to connect to the server. I'd get a ton of messages on the server:
Dec 20 09:47:36 rich rpc.mountd[4323]: authenticated mount request from 192.168.0.10:1015 for /data/diskless/mythliv2 (/data/di
skless/mythliv2)
(the port number varies, but I'll get one of these every 20 seconds or so as the box tries to mount its root)

I can't connect to NFS from VMs either, but rpcinfo shows all the services running as far as I can tell from either the host or a client (obviously I couldn't test from the boxes that use an NFS root)). After a lot of hair-ripping I got it to work by enabling NFS v4 on the clients and server - I didn't change anything in my configs otherwise.

I was content to leave it at that, but I just tested out gentoo-sources and vanilla upstream 3.12.5 and on both of those NFS breaks again, with the same error. I suspect that the issue is related. I haven't gotten much further than that on debugging 3.12.5, but I figured I'd toss this out there and see what others think. There were obviously NFS changes in 3.12.5 as there are a few new config options. One involves supporting security labels for NFS, and the other involves NFS v4.2 support. I get the same problem whether I set those both to N or Y.

I didn't try disabling the bridge interface on 3.12.5 yet. That would be easy enough to do but it obviously won't work for KVM that way. Since NFS is served by the kernel itself I suspect that somewhere along the way a layer is getting skipped so that the kernel isn't actually listening on the right interface. Other daemons on both the host and VMs seem unaffected by the interface change.
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1746

PostPosted: Sat Dec 21, 2013 2:09 pm    Post subject: Reply with quote

Check out my way to make quemu happy

/etc/conf.d/net snippet
Code:

# Configure the bridge - "man brctl" for more details
config_vn0="10.0.1.1/24"
brctl_vn0="setfd 0
sethello 10
stp off"
rc_net_vn0_provide="lan !net"


qemu starting script ( forwards command line params to qemu, feel free to run it as `./qemu_launcher -hda <some disk image>` )
Code:
#! /bin/bash
### let's define some variables to make everything neat and nice
BRIDGE=vn0
USER=$(whoami)
IFNAME=qtap$RANDOM
MAC=$( printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256)) )

echo "Trying to run qemu with networking up"
sudo /usr/bin/tunctl -u $USER -t $IFNAME && echo "Interface $IFNAME created"
sudo /sbin/brctl addif $BRIDGE $IFNAME && echo "Interface $IFNAME attached to $BRIDGE"
sudo /bin/ifconfig $IFNAME up promisc  && echo "Interface $IFNAME up"

qemu-kvm -net nic,macaddr="$MAC" -net tap,ifname="$IFNAME",script="no",downscript="no" -m 1024 -smp 3 $@

echo "Cleaning up the mess:"
sudo /bin/ifconfig $IFNAME down && echo "Interface $IFNAME down"
sudo /sbin/brctl delif $BRIDGE $IFNAME && echo "Interface $IFNAME removed from $BRIDGE"
sudo /usr/bin/tunctl -d $IFNAME && echo "Interface $IFNAME destroyed"


/etc/sudoers snippet
Code:
### allow qemu setting vitual interfaces
Cmnd_Alias QEMU = /usr/bin/tunctl, /sbin/brctl, /bin/ifconfig
<my username> ALL=(ALL) NOPASSWD: QEMU


obviously you need module 'tun' to be loaded. 3.10.7 used to load it automagically, with 3.10.17 I had to add boot script for this.
Official qemu way involves running VM as root and then dropping permissions, but I don't like this approach so I've written script above. I'm sure you can tweak it to your needs.

Also in my setup vn0 doesn't enslave any real NICs, I'm running all IP forwarding with iptables rules (so I have auto-configured wan device, and 2 nics + virtual as separate subnets routing to each other and br0 without IP assigned working as internal bridge for qemu network when I'm plaing games with it :) Dhcpd is set on this host to be authoritative and bound to 2 lan devices and vn0. Wan device is autoconfigured with ISP's dhcp and br0 is not configured at all as I only want it to mimic a switch the host is not connected to). However, this shouldn't really matter
Back to top
View user's profile Send private message
rich0
Developer
Developer


Joined: 15 Sep 2002
Posts: 160

PostPosted: Sat Dec 21, 2013 4:26 pm    Post subject: Reply with quote

szatox wrote:
Check out my way to make quemu happy


Have you tested serving NFS from the host with that config?
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1746

PostPosted: Sat Dec 21, 2013 4:36 pm    Post subject: Reply with quote

yes, I mounted nfs shares in all directions between host and several VM's. Avahi also works, including mdns. The only thing that didn't work for me was ganglia, but I'm new to it so I wouldn't blame it on virtual network.
Back to top
View user's profile Send private message
rich0
Developer
Developer


Joined: 15 Sep 2002
Posts: 160

PostPosted: Sat Dec 21, 2013 4:59 pm    Post subject: Reply with quote

szatox wrote:
yes, I mounted nfs shares in all directions between host and several VM's. Avahi also works, including mdns. The only thing that didn't work for me was ganglia, but I'm new to it so I wouldn't blame it on virtual network.


What kernel are you running, and can you grep NFS its config file?

I don't have any issues at all with networking between the guests and host. The issue is with NFS on the host, and it impacts NFS to non-VMs on the network as well. I don't see any differences in how the bridge interface is set up.

I'm starting qemu in a different way, but that shouldn't have any impact as the issue exists whether I'm running qemu or not (most of the time the host won't be running qemu anyway, so NFS has to work when qemu isn't running).
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1746

PostPosted: Sat Dec 21, 2013 5:42 pm    Post subject: Reply with quote

I've run probably all stable gentoo-sources since 3.7 (right now it's 3.10.17), and a few others
The only thing is NFS config that might couse troubles is that /etc/exports is whitespace sensitive, so all entries must follow this pattern including spaces and lack of those.
Code:
/usr/portage 10.0.0.0/16(async,rw,no_root_squash)

Code:
CONFIG_NFS_FS=m
CONFIG_NFS_V2=m
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_SWAP is not set
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
# CONFIG_NFSD_V4 is not set
CONFIG_NFS_COMMON=y


But here, if you have problem with making NFS work at all, do you have NFS support in your kernel (particularly server part, as those and client don't depend on each other)? And ofc net-fs/nfs-utils installed?
Another fail point could be your iptables rules (-P INPUT DROP?). And this one doesn't produce any specific error messages.
Finaly, NFS4 has changed naming conventions, and configs written for NFS2 and 3 do not work with NFS4 - just in case... Those 2 are in fact most likely
Back to top
View user's profile Send private message
rich0
Developer
Developer


Joined: 15 Sep 2002
Posts: 160

PostPosted: Sat Dec 21, 2013 7:12 pm    Post subject: Reply with quote

szatox wrote:
But here, if you have problem with making NFS work at all, do you have NFS support in your kernel (particularly server part, as those and client don't depend on each other)? And ofc net-fs/nfs-utils installed?


Yup - I doubt it would have worked before I changed /etc/conf.d/net if I didn't. If I disable the bridge and switch back to eth0 it starts working and I don't even need to reboot. If I switch to NFS v4 it also works on 3.10.17, but not on 3.12.5.

Quote:
Another fail point could be your iptables rules (-P INPUT DROP?).


I'm not using iptables. I checked and iptables -L just returns ACCEPT as the default on everything and no rules.

Quote:
Finaly, NFS4 has changed naming conventions, and configs written for NFS2 and 3 do not work with NFS4 - just in case... Those 2 are in fact most likely


Interesting. I didn't change my config at all to get it to work when I enabled NFSv4. I didn't have NFSv4 enabled in my kernel previously.

Various shares are configured differently, and none of them work when things aren't working. That's why I suspect this is some kind of networking/routing issue and not an NFS issue.
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1746

PostPosted: Sat Dec 21, 2013 7:48 pm    Post subject: Reply with quote

Perhaps you have both, NFS 3 and 4 active (and using 3, or in your particular case it didn't matter) as multiple versions can be supported at the same time. Well, I wasn't interested in v4 enough to dig in more.

Anyway, recently someone reported problems with dhcp on similar setup, where the onlly difference against my setup seemed to be phisical NIC bound/unbound to a bridge. I know it sounds stupid.
So, from the working configuration leave eth0 as it is, create bridge manually with brctl and only bind tap to it and see what happens. (or even assign IP to tap device without bridging it) and see if it works.

Also, have you tried setting sniffer on bridge or tap? Maybe that would hint what's wrong.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum