diff --git a/doc/lacp.md b/doc/lacp.md index ab09db39..b116ef9a 100644 --- a/doc/lacp.md +++ b/doc/lacp.md @@ -1,79 +1,23 @@ # LACP (Link Aggregation Control Protocol, IEEE 802.3ad) -A static link aggregation group works only if both ends agree about it in -advance. Nothing checks that they do: cable a member to the wrong neighbour and -the group keeps hashing frames onto a link that goes somewhere else. LACP is the -protocol that makes the two ends agree, and keeps agreeing — it discovers which -links reach the same partner, aggregates only those, and drops a link out of the -group when the partner stops answering. +A static link aggregation group works only if both ends are configured for the +same links, and nothing checks that they are. LACP lets the two ends agree on +which links reach the same partner, aggregates only those, and takes a link out +of the group when the partner stops answering. -The switch runs LACP in software on the 8051. Frames are the Slow Protocols -group: destination `01:80:C2:00:00:02`, ethertype `0x8809`, subtype 1. The -implementation covers the parts that decide whether an aggregate forms: -the Receive machine and the Selection and Mux machines of 802.3ad clause 43, -`recordPDU` and `update_NTT`, and the mis-cabling protection that refuses to -aggregate links whose partners differ. It does not implement the Churn Detection -machines or the Marker protocol, and the Mux machine is the coupled variant, so -collecting and distributing are enabled together rather than in sequence. - -Up to `LACP_NUM_LAGS` (4) groups can be placed under LACP; the rest stay static. +The switch runs LACP in software on the 8051. LACP frames belong to the Slow +Protocols, with destination `01:80:C2:00:00:02` and ethertype `0x8809`; the +subtype of LACP frames is 1. The implementation has the Receive, Selection and +Mux machines of 802.3ad clause 43 and does not aggregate links whose partners +differ. Up to `LACP_NUM_LAGS` (4) groups can run LACP; the others stay static. ## Getting the frames to the CPU -Reserved multicast addresses are handled by the RMA block before the L2 lookup -happens. Each address has its own register and the action field sits in bits -[5:4]: 0 forward, 1 trap to CPU, 2 drop, 3 forward excluding the CPU. The -registers are packed and consecutive by *entry* rather than by address byte, so -the step of 4 counts entries, not bytes of the address: `...:00` sits at -`0x4ecc` and `...:02` at `0x4ed4`, but `...:08` sits at `0x4ee0`. - -``` -#define RTL837X_RMA0_CONF 0x4ecc /* 01:80:C2:00:00:00, entry 0 */ -#define RTL837X_RMA2_CONF 0x4ed4 /* 01:80:C2:00:00:02, Slow Protocols */ -#define RTL837X_RMA_ACT_MASK 0x30 -#define RTL837X_RMA_ACT_TRAP 0x10 -#define RTL837X_RMA_ACT_DROP 0x20 -``` - -The engine puts this address on *trap* while it runs. A trap takes the frame -out of the forwarding path and hands it to the port named in `EXT_CPU_CTRL` -(`0x6724`), whose reset value names no port at all: the destination is meant to -be an *external* CPU on a physical port, which these boards do not have. That -is why the trap first looked like a dead end - the receive counters stayed -frozen while the partner kept sending, and the ingress discard counter grew -instead. `trap_init()` points the register at the CPU port once at startup, and -the same frames then arrive. - -`FORWARD` reaches the CPU too, because the CPU port is in the forwarding -domain, but a forwarded frame is subject to the L2 lookup and on its own floods -to every other port in the VLAN. That leaks a neighbour's LACPDUs to hosts that -must not see them and, with two bonds on one switch, poisons both; a reserved -link-local group must not be forwarded at all. Containing it takes a static L2 -multicast entry with a CPU-only member mask, and because lookups are IVL on -this chip, one entry per distinct pvid over all front panel ports - a VID 0 -entry is never matched, and a port outside the LACP groups needs its pvid -covered as well, or a LACPDU arriving there misses the lookup and is flooded -through its VLAN. - -The trap needs none of that and reaches further, because it acts on the address -before the VLAN lookup and before the ingress filters. An untagged LACPDU is -delivered on a port that admits tagged frames only, where a forwarded one is -dropped by the acceptable-frame-type filter; and it is delivered on a port STP -holds in blocking, so a group converges without waiting out the forward delay. -Both were measured against forward on the same ports of an SWTGW218AS. - -A trapped frame arrives without the VLAN tag the hardware inserts into a -forwarded one, so the LACPDU body sits four bytes earlier. `TRAP_RX_BODY()` -gives the offset of the body for either form, and `lacp_in()` reads the PDU -from there. - -Neighbouring addresses in the same block still differ in other respects - -`...:00` is exempt from the ingress STP-state check and `...:02` is not, which -the trap makes moot only because it acts before that check - so measure before -carrying a conclusion about one of them across to another. - -When LACP is switched off the action goes to `DROP`, which acts before the L2 -lookup, so there is nothing to undo. +While LACP runs, the reserved multicast action for `01:80:C2:00:00:02` is +*trap*, the same mechanism STP uses for its BPDUs, see [stp.md](stp.md). A +trapped LACPDU has no VLAN tag after the RTL tag, and `TRAP_RX_BODY()` gives the +offset of the LACPDU in the received frame. When the last group leaves LACP, +the action goes back to *drop*. ## What has to match across the members @@ -92,42 +36,30 @@ mismatch should be rejected or applied to the whole group. ## Protocol timing -``` -#define LACP_FAST_PERIODIC 0x0032 /* fast TX 1 s; partner expires at 3 s */ -#define LACP_SLOW_PERIODIC 0x05dc /* slow TX 30 s; partner expires at 90 s */ -#define LACP_SHORT_TIMEOUT 0x00c8 /* silent partner dropped after 4 s (802.3ad: 3 s) */ -``` +The timer values are in units of the LACP timer, which steps every four ticks +of the system clock, 50 times per second: -One unit is one step of the LACP timers, taken every four system ticks, so 50 -per second whatever the pace of the main loop; 50 units came out as 1.00 s on -the wire in all 116 intervals of a two minute capture. +| Constant | Value | Time | +|----------------------|-------|------| +| `LACP_FAST_PERIODIC` | 50 | 1 s | +| `LACP_SLOW_PERIODIC` | 1500 | 30 s | +| `LACP_SHORT_TIMEOUT` | 200 | 4 s | -The two transmit periods are worth a word, because they are the one thing here -that is not ours to choose freely. Our own timeouts only decide how patient we -are with a partner that has gone quiet, and being slow there costs nobody -anything. The transmit period is the opposite: it decides when the *partner* -gives up on us. 802.3ad pairs a 1 s fast period with a 3 s short timeout, and a -30 s slow period with a 90 s long timeout, and the ratio is the whole point. +802.3ad specifies a fast periodic time of 1 s with a short timeout of 3 s, and a +slow periodic time of 30 s with a long timeout of 90 s. The switch transmits at +the rate the partner asks for with its TIMEOUT bit. Its own timeout is always +the short one, one second above the standard, so a partner that stops sending +while its link stays up leaves the trunk after 4 s. -Getting that wrong does not break the aggregate, which is what makes it easy to -miss. An earlier version of this code transmitted every 2 to 3 s while the -partner was asking for the fast rate. The links still aggregated and still -carried traffic, but every time a gap crossed 3 s the partner's receive machine -timed us out and recovered: 13 of its 93 LACPDUs came back carrying `EXPIRED` -and reporting our SYNC as cleared. Since a received actor state is recorded -verbatim as the partner state and sent back out, the flap was visible from both -ends, and the bond's churn machines never settled. With the periods above, the -same capture shows no expiry at all and both churn states read `none`. - -Our own timeouts sit one period above the standard's 3 s and 90 s, which -absorbs a single lost LACPDU plus the tick jitter. A partner whose LACP has -stopped while its link stayed up is out of the trunk after that; anything -longer keeps hashing traffic onto a member that no longer agrees to carry it. +The Partner block of a transmitted LACPDU repeats the partner's actor +information as it was last received: system priority, system, key, port +priority, port and state. A Linux partner sets our SYNC only when this block +matches what it sent. ## LACP API ``` -void lacp_init(void) __banked; /* boot init: clear per-LAG state */ +void lacp_init(void) __banked; void lacp_off(void) __banked; void lacp_lag_set(uint8_t lag, __xdata uint16_t ports) __banked; void lacp_in(void) __banked; @@ -193,35 +125,25 @@ On the switch, with those two ports as the group: lag 1 lacp 7 8 ``` -`lacp show` should then give both ports an actor state of `0x3f`, which is -ACTIVITY, TIMEOUT, AGGREGATION, SYNC, COLLECTING and DISTRIBUTING together, a -partner state with SYNC in it, and the partner's system MAC. `lag show` should -list both ports as trunk members, since that reads the hardware rather than the -protocol's opinion of itself. +`lacp show` should then give both ports an actor state of `0x3f` (ACTIVITY, +TIMEOUT, AGGREGATION, SYNC, COLLECTING and DISTRIBUTING), a partner state with +SYNC set, and the partner's system MAC. `lag show` lists the trunk members as +programmed in the hardware and should show both ports. -The host's view is the one worth trusting, in `/proc/net/bonding/bond0`. Both -slaves want the **same `Aggregator ID`**; two different numbers mean the bond -has not aggregated anything and each link is sitting in its own group, which -looks healthy at a glance and is not. Both churn states should read `none`. -`monitoring` that never settles is the signature of our transmit period being -too slow for the rate the partner asked for. +On the host, `/proc/net/bonding/bond0` should show the same `Aggregator ID` for +both slaves; different IDs mean that each link is in a group of its own. Both +churn states should read `none`. -Three things are worth checking past convergence, because each of them has been -broken here at some point: +Further checks: -* `tcpdump -i enp1s0 ether proto 0x8809` on each slave should show the switch's - LACPDUs and that slave's own, and never the other slave's. A sibling leak is - what the trap prevents: the PDU never enters the forwarding path at all. -* No `illegal loopback` in the host's log. -* In a hex dump of a frame from the switch, the three reserved bytes after the - actor state should be zero. They used to carry whatever the transmit buffer - held from the previous frame. +* `tcpdump -i enp1s0 ether proto 0x8809` on each slave shows the LACPDUs of the + switch and of that slave, never those of the other slave. +* The host log has no `illegal loopback`. +* The three reserved bytes after the actor state in a LACPDU from the switch + are zero. -Pulling one slave down and putting it back is the cheapest failure test. The -group narrows to the surviving member and comes back within tens of seconds, -and traffic across the transition loses a few percent at the edges rather than -half of everything, which is what it would lose if the ASIC kept hashing onto -the dead member. +When one slave goes down, the group narrows to the remaining member, and the +slave rejoins after it comes back. ## Known limits diff --git a/doc/link_aggregation.md b/doc/link_aggregation.md index b725e419..90fc7637 100644 --- a/doc/link_aggregation.md +++ b/doc/link_aggregation.md @@ -227,3 +227,7 @@ iperf Done. ``` As you can see, the total throughput was 4.71 GBit/sec which is close to the maximum possible with a single 5GBit link. + +## LACP + +Dynamic aggregation with LACP is described in [lacp.md](lacp.md).