Teaming with Broadcom NICs

25 04 2013

I’m in the process of installing new Cisco 3570 switches to replace some ageing Dell models. The setup for the servers will be a minimum of two network cards, each uplinked to a different node of the two node 3750 stack.

When everything is working this will mean 2Gbps aggregate bandwidth between the server and the switch, and any other servers or switches connected with at least a 2-port etherchannel, but if an individual switch in the stack or individual NIC fails, the network connectivity will drop to 1Gbps (kicking off lots of network monitoring alerts) but still function. Therefore there is no *single* point of failure.

My “practise” server had two Broadcom NICS, so I installed the BACS4 software available on the Broadcom site. I have BACS3 on another server already so was relatively familiar with the user interface.

However, try as I might, I could not get a context menu to appear when right-clicking on the “Teams” menu item to create a new team.

Short answer – this is caused by User Account Control (UAC), and can be remedied by launching the Broadcom Control Suite 4 from the Control Panel ‘As Administrator’.

While I’m writing about this, I may as well detail the settings.

I chose the “802.3ad Link Aggregation using Link Aggregation Control Protocol (LACP)” option in the BACS software. This is the industry-standard link aggregation protocol (as opposed to PAgP, for instance, the Cisco-proprietary link aggregation protocol).

On the Cisco switch end, I used the following config to create the Port-channel:

interface po X

switchport mode access

switchport access vlan XX

no shut

I then configured the individual switch ports as follows

int gi 1/0/XXX

switchport mode access

switchport access vlan XX

channel-group X active

no shut

int gi 2/0/XXX

switchport mode access

switchport access vlan XX

channel-group X active

The X in the “channel-group” command must match the number defined in the port-channel config in order to add the port to the port-channel.

A number of commands can be used to troubleshoot – firstly I’d use show int po X to get the standard interface statistics. This will give an indication of whether the port-channel has been configured properly, as it will show the full bandwidth in the first couple lines of output, for instance:

MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

Here, there are two * 1Gbps interfaces in the port-channel, so it shows 2000000 Kbps as the “bandwidth” (BW).

If you have more than one port in the port-channel, the bandwidth should reflect the combined bandwidth of all of the ports combined.

Other things to note of course, are pretty standard – all ports should be the same speed and type (access or trunk); LACP doesn’t support half-duplex, no individual transfer will ever be faster than the speed of one of the interfaces, in this case, 1Gbps.etc etc.

Now on to the “production” servers!



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: