# Link Aggregation (aka Trunking) The RTL827x allows to combine multiple ports to a single logical link (Link Aggregation / Trunking) according to IEEE 802.3ad. LAGs allow to combine the individual physical links into a single link with the combined throughput and automatic redundancy when one of the link fails. Up to 4 Link Aggregation Groups (LAGs) can be defined on the switch devices. ## LAG control Four registers `RTL837X_TRK_MBR_CTRL_BASE(lag) (0x4f38-0x4f44)` define the LAG membership via a port mask of the logical port numbers. A hash algorithm applied to L2, L3 and L4 properties of a packet are used to decide which of the links (ports) is being used to transfer the packet. The possible properties used in the hash are: ``` #define LAG_HASH_SOURCE_PORT_NUMBER 0x01 #define LAG_HASH_L2_SMAC 0x02 #define LAG_HASH_L2_DMAC 0x04 #define LAG_HASH_L3_SIP 0x08 #define LAG_HASH_L3_DIP 0x10 #define LAG_HASH_L4_SPORT 0x20 #define LAG_HASH_L4_DPORT 0x40 #define LAG_HASH_DEFAULT (LAG_HASH_L2_SMAC | LAG_HASH_L2_DMAC | LAG_HASH_L3_SIP | LAG_HASH_L3_DIP | LAG_HASH_L4_SPORT | LAG_HASH_L4_DPORT) ``` The hash algorithm used to select links (exit ports) is defined for each LAG individually in `RTL837X_TRK_HASH_CTRL_BASE (0x4f48-0x4f54)`. ## Trunking API The code currently provides the following functions: ``` /* * Configure LAGs * Sets the members via port bitmask of a given Link Aggregation Group * The groups have numbers 0-3 * The bitmask represents up to 10 ports * If currently no LAG has algorithm used, a default is applied */ void port_lag_members_set(__xdata uint8_t lag, __xdata uint16_t members) __banked; /* * Configures the hash algorithm used for a LAG * lag is the Group to configure and hash is a bitmask */ void port_lag_hash_set(__xdata uint8_t lag, __xdata uint8_t hash_bits) __banked; ``` ## LAG configuration on the Serial Console For testing the following commands are provided on the serial console: ``` > lag [p1] [p2]... Create or set a LAG. Trunk-ID is 1 or 2. Ports are physical ports If only the LAG-ID is given but no members, the LAG is deleted > laghash 0 [hash1] [hash2]... Uses the given packet properties when hashing the packet to select the link Names for the hashes are spa, smac, dmac, sip, dip, sport, dport ``` When a lag is creates, by default the hash is based on smac, dmac, sip, dip, sport, dport. When you use your own hash settings, make sure that the hash always uses both the source and destination property of the packet, as otherwise pakets will not be routed symmetrically. ## LAG configuration via the Web Interface In the web-interface select Link Aggregation in the left navigation panel. The page will look like this: ![Alt text](images/LAG_config.png?raw=true "Link Aggregation Web-Page") Each of th 4 LAGs is configured separately. After the web-page has loaded, the current configuration can be edited by clicking on the port-images to include that port or exclude it from a LAG. When pressing on the Create/Update button, the LAG will be automatically created if not yet done, or updated. If a lage is updated to not having any members, then it is effectively deleted. All LAGs are created with the default hash-function (see above). This currently cannot be changed from the Web. ## A Test using a single Linux Desktop The following is a simple test using 2 RTL 2.5 GBit switches with at least 1 SFP+-port each. You will also need 4 10GBit SFP+ modules (DAC or Fiber) and 2 SFP+ ports on your desktop. The following shows the network configuration ``` ----------------- ----------------- Linux Comuter | | 2.5 GBit | | same Linux Computer ---------- 10G | P1 |------------| P1 | 10G ---------- 192.168.9.1 | SFP+ |==========| Switch 1 | 2.5 GBit | Switch 2 |==========| SFP+ | 192.168.9.2 enp1s0f0 ---------- | P2 |------------| P2 | ---------- enp1s0f1 | | | | ------------------ ----------------- ``` On _both_ switches create a LAG with ports 1 and 2 inside and the default hash algorithm which takes source and destination ports into account, e.g. just use the default: ``` > lag 0 1 2 ``` The following shows the configuration on the desktop using a dual 10GBit card with 2 SFP+ modules: ``` [234690.755634] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver [234690.755637] ixgbe: Copyright (c) 1999-2016 Intel Corporation. [234690.921614] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0 [234690.921914] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link) [234690.921999] ixgbe 0000:01:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: FFFFFF-0FF [234690.922002] ixgbe 0000:01:00.0: 28:41:c6:xx:xx:aa [234690.924946] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection [234690.990024] ixgbe 0000:01:00.0 enp1s0f0: renamed from eth0 [234691.056447] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0 [234691.089417] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0 [234691.089706] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link) [234691.089788] ixgbe 0000:01:00.1: MAC: 2, PHY: 19, SFP+: 18, PBA No: FFFFFF-0FF [234691.089790] ixgbe 0000:01:00.1: 28:41:c6:xx:xx:ab [234691.160997] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection [234691.166102] ixgbe 0000:01:00.1 enp1s0f1: renamed from eth0 [234691.231579] ixgbe 0000:01:00.1: registered PHC device on enp1s0f1 [234691.236965] ixgbe 0000:01:00.0 enp1s0f0: detected SFP+: 5 [234691.485031] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX [234691.557003] ixgbe 0000:01:00.1 enp1s0f1: detected SFP+: 18 [234691.753061] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX ``` Now set up 2 network namespaces and put each interface inside one: ``` sudo ip netns add netns_eth0 sudo ip netns add netns_eth1 sudo ip link set enp1s0f0 netns netns_eth0 sudo ip link set enp1s0f1 netns netns_eth1 ``` Configure network interface addresses 192.168.9.2 and 192.168.9.1 in each namespace: ``` sudo ip netns exec netns_eth0 ifconfig enp1s0f0 192.168.9.1 netmask 255.255.255.0 sudo ip netns exec netns_eth0 ip a 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 24: enp1s0f0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 28:41:c6:xx:xx:aa brd ff:ff:ff:ff:ff:ff altname enx2841c6xxxxaa inet 192.168.9.1/24 scope global enp1s0f0 valid_lft forever preferred_lft forever inet6 fe80::2a41:c6ff:fexx:xxaa/64 scope link proto kernel_ll valid_lft forever preferred_lft forever sudo ip netns exec netns_eth1 ifconfig enp1s0f1 192.168.9.2 netmask 255.255.255.0 sudo ip netns exec netns_eth1 ip a 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 25: enp1s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 28:41:c6:xx:xx:ab brd ff:ff:ff:ff:ff:ff altname enx2841c6xxxxab inet 192.168.9.2/24 scope global enp1s0f1 valid_lft forever preferred_lft forever inet6 fe80::2a41:c6ff:fexx:xxab/64 scope link proto kernel_ll valid_lft forever preferred_lft forever ``` Test is using ping. On both switches one of the 2.5Gbit links and all 10GBit links should show activity: ``` $ sudo ip netns exec netns_eth1 ping 192.168.9.1 PING 192.168.9.1 (192.168.9.1) 56(84) bytes of data. 64 bytes from 192.168.9.1: icmp_seq=1 ttl=64 time=0.082 ms 64 bytes from 192.168.9.1: icmp_seq=2 ttl=64 time=0.130 ms ^C --- 192.168.9.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1030ms rtt min/avg/max/mdev = 0.082/0.106/0.130/0.024 ms ``` You can also verify that the redundancy works by unplugging the active link, the ping should continue undisturbed with the other link now tranporting the pakets. In 2 shells, start 2 instances of iperf, listening on 2 different ports. You will need to make sure that the hash algorithm assigns different switch ports for the different port numbers. You can check this by running the iperf3 client against each server instance and verify that different links show activity: ``` sudo ip netns exec netns_eth0 iperf3 -s sudo ip netns exec netns_eth0 iperf3 -s -p 5333 ``` Now you can run the clients in parallel: ``` $ sudo ip netns exec netns_eth1 iperf3 -c 192.168.9.1 & sudo ip netns exec netns_eth1 iperf3 -p 5333 -c 192.168.9.1 [1] 295484 Connecting to host 192.168.9.1, port 5201 [ 5] local 192.168.9.2 port 60996 connected to 192.168.9.1 port 5201 Connecting to host 192.168.9.1, port 5333 [ 5] local 192.168.9.2 port 39660 connected to 192.168.9.1 port 5333 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 283 MBytes 2.37 Gbits/sec 485 272 KBytes [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 283 MBytes 2.37 Gbits/sec 479 379 KBytes [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec 444 260 KBytes [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec 578 267 KBytes [ 5] 2.00-3.00 sec 281 MBytes 2.36 Gbits/sec 385 263 KBytes [ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 373 375 KBytes [ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 430 385 KBytes [ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 452 273 KBytes [ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 319 256 KBytes [ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 425 269 KBytes [ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 364 264 KBytes [ 5] 5.00-6.00 sec 281 MBytes 2.36 Gbits/sec 561 264 KBytes [ 5] 6.00-7.00 sec 281 MBytes 2.35 Gbits/sec 446 255 KBytes [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 582 263 KBytes [ 5] 7.00-8.00 sec 281 MBytes 2.35 Gbits/sec 494 263 KBytes [ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 539 181 KBytes [ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 617 389 KBytes [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 490 232 KBytes [ 5] 9.00-10.00 sec 281 MBytes 2.35 Gbits/sec 363 215 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec 4347 sender [ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver iperf Done. [ 5] 9.00-10.00 sec 282 MBytes 2.36 Gbits/sec 536 380 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec 5015 sender [ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver iperf Done. [1]+ Done sudo ip netns exec netns_eth1 iperf3 -c 192.168.9.1 ``` As you can see, the total throughput was 4.71 GBit/sec which is close to the maximum possible with a single 5GBit link.