Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
PCIe NIC Late Init on Boot
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
ScootNBagNz
n00b
n00b


Joined: 13 Oct 2019
Posts: 15
Location: Iowa

PostPosted: Wed May 06, 2020 3:50 am    Post subject: PCIe NIC Late Init on Boot Reply with quote

Hello,

The NIC is initialized late in the boot process, how can I shuffle some items around?

grep -i 'br0\|mlx4\|enp2s0\|enp0s31f6' /var/log/syslog

Code:
00:38 42 kernel: [    2.969523] mlx4_core: Mellanox ConnectX core driver v4.0-0
00:38 42 kernel: [    2.969532] mlx4_core: Initializing 0000:02:00.0
00:38 42 kernel: [    3.256204] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
00:38 42 /etc/init.d/net.br0[2834]: Cannot add non-existent interface enp2s0 to br0
00:38 42 /etc/init.d/net.br0[2705]: ERROR: net.br0 failed to start
00:38 42 /etc/init.d/netmount[2922]: ERROR: cannot start netmount as net.br0 would not start
00:42 42 kernel: [    9.954895] mlx4_core 0000:02:00.0: DMFS high rate steer mode is: disabled performance optimized steering
00:42 42 kernel: [    9.955270] mlx4_core 0000:02:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
00:42 42 kernel: [    9.986130] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
00:42 42 kernel: [    9.986222] mlx4_en 0000:02:00.0: Activating port:1
00:42 42 kernel: [    9.988411] mlx4_en: 0000:02:00.0: Port 1: Using 4 TX rings
00:42 42 kernel: [    9.988411] mlx4_en: 0000:02:00.0: Port 1: Using 4 RX rings
00:42 42 kernel: [    9.988642] mlx4_en: 0000:02:00.0: Port 1: Initializing port
00:42 42 kernel: [    9.989041] mlx4_en 0000:02:00.0: registered PHC clock
00:42 42 kernel: [    9.989145] mlx4_en 0000:02:00.0: Activating port:2
00:42 42 kernel: [    9.989227] mlx4_en: 0000:02:00.0: Port 2: Using 4 TX rings
00:42 42 kernel: [    9.989227] mlx4_en: 0000:02:00.0: Port 2: Using 4 RX rings
00:42 42 kernel: [    9.989399] mlx4_en: 0000:02:00.0: Port 2: Initializing port
00:42 42 kernel: [    9.990046] mlx4_core 0000:02:00.0 enp2s0: renamed from eth0
00:42 42 kernel: [   10.189500] mlx4_core 0000:02:00.0 enp2s0d1: renamed from eth1
00:42 42 kernel: [   10.273231] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
00:42 42 kernel: [   10.273653] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
00:42 42 kernel: [   10.273654] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
00:46 42 kernel: [   13.689745] mlx4_en: enp2s0: Link Up



Once I am in an instance of bash, openrc brings up my networking stack no problem. I tried to use cronie to @reboot openrc, but crons are executed before the "Link Up"

How can I bring up my networking stack on boot?

rc-update show -v | grep net.br0
Code:
net.br0 |      default     


vim /etc/conf.d/net
Code:
bridge_br0="enp2s0 enp0s31f6"
# Bridge static config
config_br0="**IP* netmask 255.255.255.0"
routes_br0="default via **Gateway*"
dns_servers_br0="**DNS*"
bridge_forward_delay_br0=0
bridge_hello_time_br0=1000


ls -al /etc/init.d/net.br0
Code:
lrwxrwxrwx 1 root root 18 May  4 18:06 /etc/init.d/net.br0 -> /etc/init.d/net.lo


Thanks for the help! I love reading these forums daily. Gentoo and portage are top tier!
Back to top
View user's profile Send private message
molletts
n00b
n00b


Joined: 16 Feb 2013
Posts: 61

PostPosted: Wed May 06, 2020 10:01 am    Post subject: Reply with quote

I don't know whether it'll work in your case but, based on my router setup where I need an interface to be up (but unconfigured) prior to PPPoE init, you could try adding:
Code:
config_enp2s0="null"
rc_net_br0_need="net.enp2s0"

to your /etc/conf.d/net and creating a net.enp2s0 symlink in /etc/init.d. (For completeness, if it works, you could add the other interface to it too.)
Back to top
View user's profile Send private message
ScootNBagNz
n00b
n00b


Joined: 13 Oct 2019
Posts: 15
Location: Iowa

PostPosted: Wed May 13, 2020 9:26 pm    Post subject: Reply with quote

Unfortunately that doesn't work, net.enp2s0 bombs out as it can't find the interface when br0 tries to come up. This is also occurring on my second server without a bridge, but the same 10gbps nic.


I think my issue is that the module for the driver is loaded after openrc loads the default services, and I want to have the driver loaded directly in the kernel, but I don't really know what I am doing.

IIRC, my kernel is currently:
Code:
Device Drivers  --->
   <M> InfiniBand support  --->
      <M> InfiniBand userspace MAD support
      <M> InfiniBand userspace access (verbs and CM)
      <M> Mellanox HCA support
      <M> QLogic PCIe HCA support
      <M> Ammasso 1100 HCA support
      <M> Mellanox ConnectX HCA support
      <M> NetEffect RNIC Driver
      <M> Emulex One Connect HCA support
      <*> IP-over-InfiniBand
         [*] IP-over-InfiniBand Connected Mode support
      <M>   InfiniBand SCSI RDMA Protocol

The above references instead of my actual kernel: https://wiki.gentoo.org/wiki/InfiniBand

I am using genkernel all --menuconfig , but I cannot <*> Inifiband support as well as the child items, for menuconfig alerts:
Quote:
This feature depends on another which as been configured as a module. As a result, this feature will be built as a module.

InfiniBand support does not appear to be a child item inside device drivers, so I am lost as to which other feature is conflicting with this one.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15324

PostPosted: Thu May 14, 2020 12:57 am    Post subject: Reply with quote

Generally, I suggest that if you expect a driver will be loaded for the entire time the system is running, then the driver should be built-in, not a module. Making it a module brings you no advantages if you load it on boot and never unload it.

Ask menuconfig for help finding the dependency by highlighting the item you want to make built-in and pressing h.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6491

PostPosted: Thu May 14, 2020 12:58 am    Post subject: Reply with quote

The Infiniband option is a few rows below the graphics and sound ones. It needs IPv6 support:
Code:
Depends on: HAS_IOMEM [=y] && HAS_DMA [=y] && NET [=y] && INET [=y] && (m && MODULES [=y] || IPV6 [=y]!=m) && !ALPHA
Back to top
View user's profile Send private message
molletts
n00b
n00b


Joined: 16 Feb 2013
Posts: 61

PostPosted: Thu May 14, 2020 9:05 am    Post subject: Reply with quote

ScootNBagNz wrote:
Unfortunately that doesn't work, net.enp2s0 bombs out as it can't find the interface when br0 tries to come up. This is also occurring on my second server without a bridge, but the same 10gbps nic.

I think my issue is that the module for the driver is loaded after openrc loads the default services, and I want to have the driver loaded directly in the kernel, but I don't really know what I am doing.

Modules should be loaded by (e)udev as part of the "boot" runlevel, long before any network initialisation happens. You could try adding a dependency on udev-settle for net.enp2s0, to ensure that udev has finished loading modules (I had to use this on my home server to stop mdraid randomly failing to start because the disk controller hadn't finished scanning its ports by the time the system wanted to assemble the array - maybe your InfiniBand NIC driver also takes a while to initialise).

You would have to add:
Code:
rc_net_enp2s0_need="udev-settle"

to /etc/conf.d/net for this.

Also, have you got the /etc/init.d/net.enp2s0 symlink set up pointing to net.lo, just like the net.br0 one?
Back to top
View user's profile Send private message
ScootNBagNz
n00b
n00b


Joined: 13 Oct 2019
Posts: 15
Location: Iowa

PostPosted: Fri May 15, 2020 4:13 am    Post subject: Reply with quote

Thanks for the help squad!

I attempted to build in infiniband support, but it didn't work for me. Though i am 100% sure that version of my kernel was still broken as I was implementing efi boot for the first time (my fstab had a typo and I was using uuid <_< ) as well around with all this, so perhaps with that I wouldn't need udev-settle in my /etc/conf.d/net. It seems that the NIC does take time to initialize, though :?: :?: :?: Just using the genkernel all defaults was enough to get networking happy at boot with molletts' suggestion and help from the wikis' suggestion of config="null" to avoid the interfaces from reaching for dhcp:

cat /etc/conf.d/net
Code:
bridge_br0="enp2s0 enp0s31f6"

rc_net_enp2s0_need="udev-settle"
config_enp2s0="null"
config_enp0s31f6="null"

# Bridge static config
config_br0="ip netmask"
routes_br0="default via gateway"
dns_servers_br0="dns"

bridge_forward_delay_br0=0
bridge_hello_time_br0=1000
rc_net_br0_need="net.enp2s0 net.enp0s31f6"


ls -al /etc/init.d/net*
Code:

lrwxrwxrwx 1 root root    18 May  4 18:06 /etc/init.d/net.br0 -> /etc/init.d/net.lo
lrwxrwxrwx 1 root root     6 May 14 21:42 /etc/init.d/net.enp0s31f6 -> net.lo
lrwxrwxrwx 1 root root     6 May 10 13:00 /etc/init.d/net.enp2s0 -> net.lo



syslog:
Code:

[   10.389388] br0: port 1(enp2s0) entered blocking state
[   10.389390] br0: port 1(enp2s0) entered disabled state
[   10.389441] device enp2s0 entered promiscuous mode
[   10.400605] br0: port 2(enp0s31f6) entered blocking state
[   10.400606] br0: port 2(enp0s31f6) entered disabled state
[   10.400666] device enp0s31f6 entered promiscuous mode
[   10.406191] device br0 entered promiscuous mode
[   11.934524] mlx4_en: enp2s0: Link Up
[   11.934599] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[   11.935254] br0: port 1(enp2s0) entered blocking state
[   11.935258] br0: port 1(enp2s0) entered forwarding state
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum