Configuring Link Aggregation on FreeNAS 8 Using LACP and Cisco EtherChannel

In this post I’m going to demonstrate how to setup link aggregation (LAGG) on FreeNAS 8 using Cisco switches.  Aggregation is a method of bonding multiple ethernet links together to provide additional bandwidth and redundancy.

FreeNAS supports several different protocols for LAGG but LACP is the most robust option.  LACP is a negotiated protocol that works by exchanging frames with the peer to automatically bundle multiple links together.

If you do not have a Cisco switch you can still use LACP as long as your switch supports IEEE 802.3ad.  If it does not support the protocol then you will need to use one of the other protocols such as load balance, or round robin.

How many links can be bonded?

Cisco supports the use of up to 8 physical links in a single active bundle.  You can bond either Fast, Gigabit, or 10-Gigabit connections together but all links in the bundle must be operating at the same speed.  To achieve an even distribution of traffic across the links you should use either 2, 4 or 8 connections.

How is traffic distributed? 

Packets are distributed accross the links in the channel using a hashing algorithm.  Traffic is hashed to one link in the channel based on source and destination mac address, IP address, and port number.  A single session cannot span multiple links in the bundle which means that aggregation only increases throughput when multiple different hosts will be connecting to the server.

Network Cards

For my LAGG configuration I’m using 4 PCIe Intel gigabit ethernet cards but you can use whatever you have lying around as long as the link speeds are all the same.  You can even mix and match onboard network cards with PCIe cards.

If you have limed expansion slots in your system you might want to install a dual , or quad port NIC instead.

Switch Hardware

My FreeNAS box is connected to a Cisco 3750G 24 port gigabit switch.  Ebay usually has pretty good prices for used Cisco equipment but with gigabit switch prices dropping you can buy a new switch with support for 802.3ad without spending a ton of cash.  The Cisco SG200-26, a sub $300 switch has 24 ports, plus 2 SFP ports.  Another good option is the Netgear GS748TNA 48 port switch.

FreeNAS Versions

Currently I recommend using the latest stable release of FreeNAS , version 8.0.2 x64 seems to handle LAGG without any issues.  When I attempted to setup aggregation on 8.2 beta (and alpha) FreeNAS kept crashing as soon as the LAGG interface was created.  I’m sure this will be resolved as the developers continue to debug 8.2 but at this time I would avoid those releases for anything besides testing.

Configuring LAGG From the GUI

If you have an extra interface in your NAS for management access you can configure a LAGG interface from the GUI.  You can’t add an active interface to a LAGG so if you only have 4 network cards and you want to aggregate all of them then you will need to perform the interface configuration from the console using the steps in the next section.

The link aggregation settings in the web interface can be found under Network \ Network Settings \ Link Aggregation.

Click on ‘Add LAGG Interface’ to start the configuration process.

Select LACP as the protocol then hold down CTRL while you select all of the interfaces that will be bonded together.  Press OK to finish adding the interface.

After adding the interface you should see lagg0 listed as a new virtual interface on the link aggregation tab.  In parenthesis next to the name of the interface is the protocol being used and the names of the member interfaces.

If you make a mistake you can delete the virtual interface and start over.  You can also use the edit interface feature to add or remove an interface from the LAGG.

Click on ‘Edit Interface’ to assign an IP address to the LAGG group.  Enter the IP address and mask , or enable DHCP instead.   If you want to add multiple IP addresses to the interface you can use the alias feature to do so.

After the settings are applied you can click on the interfaces tab to confirm that the IP address was successfully applied to the interface.

At this point you are ready to configure the switch to support the new bundled interfaces.

LAGG Configuration Using the Console

If your FreeNAS system doesn’t have a dedicated management NIC then the only way to configure a LAGG group is by using the console.  The only downside to using the console for this task is that it requires two reboots to apply all of the configuration changes.  When using the web interface these additional reboots are not necessary.

To begin setting up link aggregation from the console select option #2.

Then select option 3 to configure LACP as the LAGG protocol to use.

Select the first interface to join to the nas, hit enter, and repeat until you have finished adding interfaces to the LAGG group.  Hit ‘q’, to exit this menu when you’re done.

At this point FreeNAS must be rebooted in order for the new console settings to be applied.  From the main console menu select option 10 to reboot the system.

After the box has finished rebooting you can configure the new lagg0 interface by selecting option #1 ‘Configure Network Interfaces’.  Select the option that corresponds to your new bonded interface and hit enter.  The interface will not be visible here until after rebooting.

In the interface settings menu you can enable DHCP or manually configure an IP address for the interface.   After you’ve finished configuring the interface FreeNAS must be rebooted again before the settings will take affect.

When the box comes back from the reboot you should be able to ping the new IP address and connect to the web interface on the LAGG address.

Cisco Switch Configuration

In this section I’ll explain how to configure IOS on a Cisco switch to support the LACP connection from FreeNAS.

First make sure that all of the interfaced connected to the NAS are enabled and that they are all linked at the same speed.  The ports will not bundle together unless they are all configured exactly the same.  I’m going to be bonding 4 1Gb connections together to form a 4Gbps port channel.

Below are the commands I used to configure one of my switchports.  These commands join the port to channel group or port channel 1 and instructs the switch to use LACP as the protocol.  Configure each of the ports in your link group in the same way.

#conf t
(config)#int gi2/0/13
(config-if)#des FreeNAS LAGG - 1
(config-if)#channel-group 1 mode active
(config-if)#channel-protocol lacp

Next I configured the port channel interface by setting a description and enabling the port.

(config)#int port-channel 1
(config-if)#des FreeNAS LAGG Group
(config-if)#description FreeNAS LACP Group
(config-if)#no shut

Once the configuration is in place the port channel interface Po1 should show a status of connected.  If it doesn’t double check the configuration of each member port to ensure they are all the same.  If it’s still not active them take a look at the FreeNAS configuration and make sure all of the configuration changes have applied, it’s not a bad idea to try rebooting.

To verify the status of each of the individual ports in the bundle run the command ‘show etherchannel port-channel’, each of the ports should have an active status.

Redundancy and Failover Testing

Besides increased bandwidth one of the benefits of using LAGG is it ensures redundnacy in the event of a loss of link on a member.  To test the failover capabilities I started a continuous ping to the FreeNAS IP address then pulled out one of the cables.  The port channel only dropped one packet before re-converging, not bad!

If your building a high availability NAS then you can see the importance of implementing link aggregation to maintain up-time.

Performance Testing

The best way to benchmark pure network throughput is with a program such as iPerf which is now included in FreeNAS 8 or later.  I was able to achieve 2.5Gbps of throughput pretty easily using 3 iPerf sessions sending TCP traffic spread out across spare machines on my network.  At some point I plan to do some additional performance testing but first I need to add some additional network cards to my Linux box in order to make traffic generation easier.

CPU utilization increased on the FreeNAS server increased significantly but memory usage remained low while I was running the iPerf tests.

General Thoughts on Link Aggregation in FreeNAS

Link aggregation is relatively simple to setup using FreeNAS and there are some very nice benefits to utilizing it.  For most home networks a single gigabit connection will probably provide enough bandwidth in most cases.  Typically hard drive performance is the bottleneck in most smaller storage systems.

For enterprise or high performance applications link aggregation is essential.  As the number of spindles in a storage system increases the network connection will eventually become a bottleneck.  Aggregation is especially useful for iSCSI since it can see multiple paths to a storage system and utilize separate links.

In some cases falt tolerance is more important than increased throughput.  The failover link aggregation mode in FreeNAS might be a better choice than LACP since it would allow you to connect multiple links to different switches.  LACP connections must all be connected to the same switch which leaves the NAS vulnerable to a switch failure.

If you’ve had any success or failure with link aggregation I’d be interested to hear from you, feel free to leave a comment below.