Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Bond two NICs together and connect to a Cisco switch
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Fri Jul 16, 2004 12:06 pm    Post subject: HOWTO: Bond two NICs together and connect to a Cisco switch Reply with quote

I use an Intel Dual Server NIC, 2* 100MBit and a Cisco 2950 48 port Switch and of course gentoo Linux with kernel 2.6

This kind of bonding provides load balancing and fault tolerance.

1. Compile the bonding driver as module. The advatage is, you can give some paramters to the driver.
Device Driver -> Networking Support -> <M> Bonding driver support
Compile your kernel
Code:
cd /usr/src/linux
make && make modules_install


2. To autoload the driver put the line
Code:
bonding miimon=100 mode=0

into /etc/modules.autoload.d/kernel-2.6

3. Edit /etc/conf.d/net . For a fixed IP, mine looks like the following. If you look at the
Code:
iface_bond0="192.168.14.5 netmask 255.255.255.0 broadcast 192.168.14.255 up"
gateway="bond0/192.168.14.210"

There is no entry for eth0 or eth1!

4. To install ifenslave.c, do
Code:

cd /usr/src/linux/Documentation/networking
gcc -Wall -Wstrict-prototypes -O -I/usr/src/linux/include ifenslave.c -o ifenslave
cp ifenslave /sbin/ifenslave


5. Edit the file /etc/conf.d/local.start to enslave the two NICs
Code:
ifenslave bond0 eth0 eth1


6. make a symlink for bond0
Code:
ln -sf /etc/init.d/net.eth0 /etc/init.d/net.bond0


7. Make sure you don't start eth0 or eth1, but start bond0
Code:
rc-update del eth0
rc-update del eth1
rc-update add bond0 default


8.Reboot.

9. After reboot ifconfig should look similar to this. If you look at the RX and TX values of eth0 and eth1, you can see a nice load balancing.
Code:
bond0     Link encap:Ethernet  HWaddr 00:11:22:33:44:55 
          inet addr:192.168.14.5  Bcast:192.168.14.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:833 errors:0 dropped:0 overruns:0 frame:0
          TX packets:226 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:360503 (352.0 Kb)  TX bytes:16845 (16.4 Kb)

eth0      Link encap:Ethernet  HWaddr 00:11:22:33:44:55 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:181991 (177.7 Kb)  TX bytes:8241 (8.0 Kb)

eth1      Link encap:Ethernet  HWaddr 00:11:22:33:44:55 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:413 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:178512 (174.3 Kb)  TX bytes:8604 (8.4 Kb)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)


10. To configure EtherChannel use telnet to the Cisco. I have bonded port 23 and 24 together:
Code:
Password:
cata2950>enable
Password:
cata2950#


cata2950#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
cata2950(config)#interface range FastEthernet0/23 -24
cata2950(config-if-range)#channel-group 1 mode auto
Creating a port-channel interface Port-channel1
cata2950(config-if-range)#end

cata2950#show running-config
[...]
interface FastEthernet0/23
 no ip address
 channel-group 1 mode auto
!
interface FastEthernet0/24
 no ip address
 channel-group 1 mode auto
!
[...]

cata2950#copy running-config startup-config
Destination filename [startup-config]?
Building configuration...
[OK]
cata2950#exit


For more info about the EtherChannel configuration go to the Cisco website

For more info about the bonding please read /usr/src/linux/Documentation/networking/bonding.txt

Fell free to add stuff or correct me.


Last edited by tam on Mon Aug 30, 2004 7:31 am; edited 2 times in total
Back to top
View user's profile Send private message
mkoroschetz
n00b
n00b


Joined: 20 Aug 2004
Posts: 3
Location: Miami / USA

PostPosted: Fri Aug 20, 2004 3:05 pm    Post subject: Master 'bond0', Slave 'eth0': Error: Enslave failed Reply with quote

The step by step instructions are great and easy to understand and follow.

Unfortunately in my setup I am getting the error when executing: ifenslave bond0 eth0

Master 'bond0', Slave 'eth0': Error: Enslave failed

and the only thing that happens is that "bond0" takes the HW address of the slave "(eth0) but there is no entry for eth0 in ifconfig and obviously it does not work.

Code:
bond0     Link encap:Ethernet  HWaddr [b]08:00:20:c5:7a:15[/b]
          inet addr:192.168.1.250  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)


Environment:

HW platform: SPARC
Kernel: Gentoo - Linux version 2.4.26-sparc-r2
Bonding: Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
--> Compiled as module.
--> I am autloading de module with /etc/modules.autoload.d/kernel-2.4 (same as in step 2. above)
NIC: eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:c5:7a:15
--> Compiled into the Kernel
Tested ifenslave.c which installed with the Kernel, as well as the latest avialable: ifenslave-1.1.0-r1

Any suggestions will be greatly appreciated. Thanks in advance.
Back to top
View user's profile Send private message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Fri Aug 20, 2004 4:23 pm    Post subject: Re: Master 'bond0', Slave 'eth0': Error: Enslave failed Reply with quote

mkoroschetz wrote:
Master 'bond0', Slave 'eth0': Error: Enslave failed


If you only have one NIC, you don't need the bonding driver.
Back to top
View user's profile Send private message
mkoroschetz
n00b
n00b


Joined: 20 Aug 2004
Posts: 3
Location: Miami / USA

PostPosted: Fri Aug 20, 2004 11:36 pm    Post subject: Re: Master 'bond0', Slave 'eth0': Error: Enslave failed Reply with quote

Sorry for the misleading information . I am aware that it makes not much sense to do bonding with only one interface. The command that I need to work is really:
Code:
ifenslave bond0 eth0 eth1

but I get the same error message:
Code:
Master 'bond0', Slave 'eth0': Error: Enslave failed
Master 'bond0', Slave 'eth1': Error: Enslave failed

I don't see any other error message reported in the log files.

On the other hand a bond has to work with just one interface, because in case of a failure, that is exactly what could happen.

Thanks
Back to top
View user's profile Send private message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Sat Aug 21, 2004 10:05 am    Post subject: Reply with quote

Di you bring up eth0 or eth1 before bond0? If so don't do this. I will add this to the guide.

See new 7. in the guide.
Back to top
View user's profile Send private message
mkoroschetz
n00b
n00b


Joined: 20 Aug 2004
Posts: 3
Location: Miami / USA

PostPosted: Thu Aug 26, 2004 4:51 am    Post subject: Re: Master 'bond0', Slave 'eth0': Error: Enslave failed Reply with quote

No, I did not bring up either eth0 or eth1 before bond0. There are no entries for eth0 and eth1 in /etc/init.d/net.

On this line of thinking the only thing to mention would be, that I have the driver for eth0 and eth1 compilled into the kernel. This means that the drivers get loaded before the bonding module.

By the way I am doing this on a sparc system, and eth0 and eth1 are SUNHME cards.
Back to top
View user's profile Send private message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Thu Aug 26, 2004 12:08 pm    Post subject: Re: Master 'bond0', Slave 'eth0': Error: Enslave failed Reply with quote

mkoroschetz wrote:
On this line of thinking the only thing to mention would be, that I have the driver for eth0 and eth1 compilled into the kernel. This means that the drivers get loaded before the bonding module.

I have also compiled the driver for the NIC(s) into the kernel.

I don't know anbything about Sparc systems.
Back to top
View user's profile Send private message
Zigen
n00b
n00b


Joined: 16 May 2003
Posts: 16
Location: Santa Barbara, CA

PostPosted: Mon Aug 30, 2004 6:46 am    Post subject: Reply with quote

Would it be possible to bond 4 NICS together?

1x Integrated NIC
1x Dual NIC Card
1x Single NIC card
Back to top
View user's profile Send private message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Mon Aug 30, 2004 6:56 am    Post subject: Reply with quote

Zigen wrote:
Would it be possible to bond 4 NICS together?

Sure. The only thing you have to change is
Code:
ifenslave bond0 eth0 eth1 eth2 eth3
Back to top
View user's profile Send private message
Zigen
n00b
n00b


Joined: 16 May 2003
Posts: 16
Location: Santa Barbara, CA

PostPosted: Mon Aug 30, 2004 7:15 am    Post subject: Reply with quote

Will this work if I have 3x 100 mbit and 1 x gigabit NIC ports on my server?
Back to top
View user's profile Send private message
tam
Guru
Guru


Joined: 04 Mar 2003
Posts: 569
Location: freiburg.de

PostPosted: Mon Aug 30, 2004 7:38 am    Post subject: Reply with quote

Zigen wrote:
Will this work if I have 3x 100 mbit and 1 x gigabit NIC ports on my server?

I did not try this, but there nothing in the docs that this won't work, so I think it should work. Pls try it and report here.
Back to top
View user's profile Send private message
Otm
n00b
n00b


Joined: 27 Apr 2003
Posts: 21

PostPosted: Fri Sep 03, 2004 2:48 am    Post subject: Reply with quote

Will this work if I have no Cisco but i have 3 ISP connection (ADSL) ?
Back to top
View user's profile Send private message
jdgill0
Veteran
Veteran


Joined: 25 Mar 2003
Posts: 1366
Location: Lexington, Ky -- USA

PostPosted: Thu Sep 16, 2004 1:08 pm    Post subject: Reply with quote

Otm wrote:
Will this work if I have no Cisco but i have 3 ISP connection (ADSL) ?


The help within menuconfig for bonding says the following
Quote:
If you have two Ethernet connections to some other computer, you can make them behave like one double speed connection using this driver. Naturally, this has to be supported at the other end as well, either with a similar Bonding Linux driver, a Cisco 5500 switch or a SunTrunking SunSoft driver.


So it seems bonding is dependent upon the router having this capability or the computer at the other end having the bonding support in its kernel. Obviously, with your question you don't have control over the computer at the other end.

If this will not work for you, I am also in the process of trying something else that is similar. I read a website that talked about load balancing with multiple nics using a "virtual ip address" for all the nics. It did not explain in detail unfortunately. What I have done is assign the same IP address to ALL the nics in our master node (we are attempting to use a cluster and bandwidth is killing us). I don't know if what I have done will even work, as I made the changes remotely from home last night. Hopefully I have a chance to go into the lab today and try this out and do some more research about virtual ip addresses for multiple nics.

I also plan to try the bonding technique. Our cluster is using the school's switches, so I am unsure whether or not the client nodes will need bonding support in their kernel.
Back to top
View user's profile Send private message
ichoes
n00b
n00b


Joined: 18 Oct 2004
Posts: 5

PostPosted: Mon Oct 18, 2004 4:45 pm    Post subject: Re: Master 'bond0', Slave 'eth0': Error: Enslave failed Reply with quote

mkoroschetz wrote:
The step by step instructions are great and easy to understand and follow.

Unfortunately in my setup I am getting the error when executing: ifenslave bond0 eth0

Master 'bond0', Slave 'eth0': Error: Enslave failed

and the only thing that happens is that "bond0" takes the HW address of the slave "(eth0) but there is no entry for eth0 in ifconfig and obviously it does not work.

Code:
bond0     Link encap:Ethernet  HWaddr [b]08:00:20:c5:7a:15[/b]
          inet addr:192.168.1.250  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)


Environment:

HW platform: SPARC
Kernel: Gentoo - Linux version 2.4.26-sparc-r2
Bonding: Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
--> Compiled as module.
--> I am autloading de module with /etc/modules.autoload.d/kernel-2.4 (same as in step 2. above)
NIC: eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:c5:7a:15
--> Compiled into the Kernel
Tested ifenslave.c which installed with the Kernel, as well as the latest avialable: ifenslave-1.1.0-r1

Any suggestions will be greatly appreciated. Thanks in advance.


I dont know if you already solved your problem but the thing is with sun sparc is that they use the same MAC Address for all cards. You have to change the setting in the eprom to use their unique MAC address. I had to do it because in the university were i was working one of the routers was complaining that i was using the same ip addres for to NIC. After i changed the settings of the NIC everything went back to normal.
Back to top
View user's profile Send private message
xcorpse
n00b
n00b


Joined: 23 Feb 2005
Posts: 2
Location: london

PostPosted: Wed Feb 23, 2005 2:47 pm    Post subject: sunqe and bonding Reply with quote

i have tried to use the sun qfe's with the older bonding driver (circa 2.4.x), however at the time the driver (sunqe) did not use the mii interface (which the bonding driver relied on for link status information). this requirement has changed in the kernel 2.6.x bonding driver, it now uses netif_carrier_ok by default so cards that are not mii aware can be used.
_________________
no name ... no slogan
Back to top
View user's profile Send private message
mattsk
n00b
n00b


Joined: 11 Apr 2003
Posts: 46
Location: Newcastle, Australia

PostPosted: Wed Mar 02, 2005 7:52 am    Post subject: Reply with quote

I've got a similar setup happening. However, in my case it was important that the bonding module be brought up before any of the ethernet modules. I also had the problem that the eth1394 module would sometimes load itself before the module for my ethernet ports, meaning that sometimes I had to execute
Code:
/sbin/ifenslave bond0 eth0 eth1

and sometimes
Code:
/sbin/ifenslave bond0 eth1 eth2


I solved this with an entry in /etc/modules.d/ called bond (which was put there by the ifenslave ebuild, I think ... it seemed easier to emerge ifenslave than to do the above).

Code:

# read /usr/src/linux/Documentation/networking/bonding.txt for help!

alias bond0 bonding
options bond0 mode=802.3ad miimon=100

alias eth0 tg3
alias eth1 tg3

# Make sure bond0 is initialised before the tg3 module.
below tg3 bond0

# Make sure our network interfaces are brought up before the firewire ethernet
pre-install eth1394 /sbin/modprobe eth0

#alias bond1 bonding
#options bond1 -o bonding1 arp_interval=200 arp_ip_target=10.0.0.1

# Parameters:
# max_bonds int, description "Max number of bonded devices"
# miimon int, description "Link check interval in milliseconds"
# use_carrier int, description "Use netif_carrier_ok (vs MII ioctls) in miimon; 0 for off, 1 for on (default)"
# mode string, description "Mode of operation : 0 for round robin, 1 for active-backup, 2 for xor"
# arp_interval int, description "arp interval in milliseconds"
# arp_ip_target string array (min = 1, max = 16), description "arp targets in n.n.n.n form"
# updelay int, description "Delay before considering link up, in milliseconds"
# downdelay int, description "Delay before considering link down, in milliseconds"
# primary string, description "Primary network device to use"
# multicast string, description "Mode for multicast support : 0 for none, 1 for active slave, 2 for all slaves (default)"
# lacp_rate string, description "LACPDU tx rate to request from 802.3ad partner (slow/fast)"


I deliberately used pre-install (instead of below) for eth1394 since the use of the below directive will cause the bdlow modules also be removed after (as well as installed before) the module in question. In this case, if I remove tg3 with modprobe -r it will also remove bonding, just as if I installed tg3 with modprobe, it would first install bond0. I could probably have gotten away with pre-install for this just as well.

Any comments as to the correctness of this module setup would be welcomed. I'd *REALLY* like some mechanism in the /etc/conf.d/net file to name eth0 and eth1 as the slaves of bond0, rather than having to frag around with the local.start file.
_________________
-- Matt Sk (etc)
Back to top
View user's profile Send private message
briansrapier
n00b
n00b


Joined: 05 May 2005
Posts: 22
Location: Inwood, WV, USA

PostPosted: Wed May 11, 2005 4:22 pm    Post subject: Reply with quote

I realize that this is an older post, but more relavent than most others that I have come across. I am running Gentoo 2005.0, 2.6.11-gentoo-r6 kernel built from vanilla-sources with bonding compiled as a module. My issue is not with bonding starting up; it does that just fine:

bond0 Link encap:Ethernet HWaddr 00:11:09:B7:93:6A
inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:354065 errors:0 dropped:0 overruns:0 frame:0
TX packets:962 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:41367676 (39.4 Mb) TX bytes:289828 (283.0 Kb)

eth0 Link encap:Ethernet HWaddr 00:11:09:B7:93:6A
inet6 addr: fe80::211:9ff:feb7:936a/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:354065 errors:0 dropped:0 overruns:0 frame:0
TX packets:963 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41367676 (39.4 Mb) TX bytes:289986 (283.1 Kb)
Base address:0x9000 Memory:fb100000-fb120000

eth1 Link encap:Ethernet HWaddr 00:11:09:B7:93:6A
inet6 addr: fe80::211:9ff:feb7:936a/64 Scope:Link
UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Base address:0xa000 Memory:fb000000-fb020000

I have the following options set in modules.d/bond:

alias bond0 bonding
options bond0 mode=1 miimon=100 updelay=100 downdelay=100 arp_interval=200 arp_target=10.0.0.1 primary=eth0

The issue that I have is with failover from primary to secondary. When I 'hard fail' eth0 by pulling the connection or shutting down the interface, bonding fails to roll over to the second interface. However, if I plug the connection back in within a minute (+/-), the connection on eth0 remains active, which wasn't the case before I implemented bonding.

If I manually failover to the second NIC using ifenslave, I get:

root@host>ifenslave -vc bond0 eth1
ifenslave.c:v1.1.0 (December 1, 2003)
o Donald Becker (becker@cesdis.gsfc.nasa.gov).
o Detach support added on 2000/10/02 by Willy Tarreau (willy at meta-x.org).
o 2.4 kernel support added on 2001/02/16 by Chad N. Tindel
(ctindel at ieee dot org).
ABI ver is 2
Slave eth1: flags 1803.
Master 'bond0': Error: SIOCBONDCHANGEACTIVE failed: Invalid argument
Master 'bond0', Slave 'eth1': Error: Change active failed

Currently, the server in question is connected, both eth0 and eth1, to a D-Link managed switch. Once put into production, the server will be directly connected to an ImageStream Linux router. Could either of these devices be causing my problems?
Back to top
View user's profile Send private message
Ausdonky
n00b
n00b


Joined: 12 May 2004
Posts: 15
Location: Brisbane, Oztralia :)

PostPosted: Tue Jun 14, 2005 1:08 pm    Post subject: Reply with quote

Hi brian..

looks like your switch might not be setup correctly.. can you supply a model number for the one you are using?? Does it have 802.3ad trunking support?? (Not all D-link managed switches have 802.3ad support!)

wrt the ImageStream server.. if the problem is that your switch does not have trunking support the linux server should be ok as long as you setup bonding on that machine also..

HTH

Andrew
Back to top
View user's profile Send private message
smile032
n00b
n00b


Joined: 01 Jul 2005
Posts: 6
Location: Jakarta

PostPosted: Mon Jul 18, 2005 1:38 pm    Post subject: net.bond1 ?? Reply with quote

bond0 has already been configured for eth3 and eth4, and all is running well. Then I try to bond eth1 and eth2. I edit /etc/conf.d/net and comment eth1 and eth2 configuration, and add a new line iface_bond1="bla bla bla" to set the bond1 IP. I do cp /etc/init.d/net.bond0 /etc/init.d/net.bond1, and then try to run the net.bond1 interface. What I found is "no device found". I think that bond1 device is not found by the kernel. How can I add the device bond1? I believe that I should modprobe a module to solve this problem.. Any idea ??

Thanks in advance
Back to top
View user's profile Send private message
gnork
n00b
n00b


Joined: 08 Sep 2004
Posts: 23
Location: Germany

PostPosted: Mon Aug 08, 2005 11:37 am    Post subject: Reply with quote

Add the following to /etc/modules/d/bond (created by "emerge ifenslave")

Code:

alias bond1 bonding


For further configuration have a look at /usr/src/linux/Documentation/networking/bonding.txt
There are some issues with the loading sequence of the modules, so I really suggest you read that first.

HTH
gnork
_________________
--
Life is like an analogy
-
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum