EtherChannel, Bond, Team, LAG: How Link Aggregation Actually Works

Every few years comes some variation of the same ticket. A server team stands up a new backup target, we hand them four 10 gig ports in an LACP bundle, and everything comes up clean. Four members, one logical interface, 40 gigabits of headroom on the capacity spreadsheet. Then the first full backup kicks off and runs at 10 gig, flat and unmoving all night. The ticket says the network is slow, the graphs show one member pegged and three idle, and somebody wants to know why they bought 40 gig and got 10.

The obvious answer is the boring one. A single flow rides a single member, so one backup stream was never going to top 10 gig no matter how many links sat behind it. Nothing on that switch is misconfigured, and anybody who’s read a show etherchannel already knows this much. The question worth chasing sits underneath it. Which member does a given flow land on, and who decides? The standard won’t say. It’s firm that a single conversation can’t be split across links, then leaves the entire job of spreading conversations across them to the vendor. That gap is where the interesting failures come from, the three tier designs that dutifully load-balance every flow straight onto the same link, and the host operating systems that never agreed to follow the rules in the first place.

This post kicks off a new series, Network Protocol Deep Dive. One protocol at a time, mechanics instead of recipes. Link aggregation earns the debut slot because everybody runs it and a surprising share of what engineers “know” about it is folklore. This part covers the single switch fundamentals… naming, negotiation, and hashing. Part 2 stretches one bundle across two switches, where things get genuinely weird.

Seven names for the same thing

EtherChannel, port-channel, LAG, bond, team, aggregated Ethernet, trunk. Depending on the flavor of switching in front of you, the same 802.3ad bundle answers to any one of them. All describe one idea. Take N physical links between two devices, present them as a single logical link, and spread traffic across the members. The vocabulary is a mess because every vendor and every operating system named it independently, and the names stuck.

Cisco alone uses three terms for one construct, at three different layers. EtherChannel is the technology name, and it predates Cisco. It came from Kalpana, the company that built the first Ethernet switch, acquired by Cisco in 1994. The brand survived Fast EtherChannel and Gigabit EtherChannel and outlived the speeds. Port-channel is the logical interface object the OS creates (interface Port-channel1). Channel-group is the physical interface command that binds a member port into the bundle. The three way split trips up every newcomer, and it’s mostly consistent across IOS and NX-OS once it clicks.

Outside Cisco, the IEEE standard says link aggregation, and a bundle is a Link Aggregation Group, LAG. Linux calls it bonding. Windows calls it teaming. Juniper calls it aggregated Ethernet, so the interface is ae0 instead of Po1. And then there’s the landmine… HP has historically called the bundle a trunk. To anyone raised on Cisco gear, a trunk is an 802.1Q VLAN tagging port, a completely different feature, and Inigo Montoya had the right response ready. You keep using that word. I do not think it means what you think it means. I’ve watched a Cisco person and an HP person “agree” on a trunk config over the phone and build two different things, so spell out which meaning you’re using before anyone types a command.

There is no LACP RFC

I keep running into “the LACP RFC” in design docs and interview answers, and it doesn’t exist. Link aggregation is an IEEE standard, published by the Institute of Electrical and Electronics Engineers as 802.3ad in 2000 as part of the Ethernet family. In 2008 it was moved out of 802.3 entirely and republished as 802.1AX-2008, essentially unchanged. It’s a nerdy little detail, but it’s the reason aggregation is bridging and not routing. Aggregation sits above the MAC, so the 802.1 working group took jurisdiction over the same protocol. Most of us still say 802.3ad out of habit, and the Linux kernel agrees, bonding mode 4 is literally named 802.3ad.

There’s a cleaner way to remember which standards body owns what, and it comes down to layers. The IEEE runs Layers 1 and 2, the physical and data link world, the cabling and the frames and everything that happens before a packet has an IP address. The IETF, the Internet Engineering Task Force, picks it up at Layer 3 and carries it through Layer 7, and that’s where RFCs live. IP, TCP, BGP, the routing you spend your day thinking about. LACP, the Link Aggregation Control Protocol, works inside a single broadcast domain and never climbs past Layer 2, so it was always going to carry an IEEE number instead of an RFC. Go looking for the LACP RFC and you’re one floor too high.

Two revisions followed, in 2014 and 2020, each refining the mechanics without rewriting the core. The standard settled early and barely moved since. Everything that bites you lives in what the vendors bolted on top.

802.1AX pins down more than most people think:

• The LACPDU format. The LACP Data Unit, the on wire control frame both ends exchange.
• The actor/partner state machine. How the two ends discover each other and agree to bundle.
• Identity. System priority plus a system MAC form the System ID. An operational key marks which ports are allowed to aggregate together. Port priority and port number identify each member.
• The in order guarantee. All frames of a given conversation must be delivered in order. This single sentence shapes everything about load balancing below.
• Timers. Two PDU rates, fast (1 second) and slow (30 seconds, the default). An end is declared gone after three missed PDUs, so failure detection takes 3 seconds on fast timers and 90 on slow.
• The Marker protocol. A flush mechanism for draining in flight frames from a member before moving a conversation to a different member, so you can rebalance without reordering. Operator lore holds that almost nobody implements or uses it, which makes it the forgotten half of 802.3ad.
• Churn detection. State machines that flag when the two ends keep renegotiating instead of settling. It’s purely diagnostic, and mostly unread.

The actor/partner machinery is the part you’re staring at in show lacp output when a bundle won’t form. Every LACPDU carries two blocks of state. The actor block is the sender describing itself. The partner block is the sender describing what it currently believes about the other end. Each side keeps transmitting until my view of you matches your view of you. Once the views agree, ports get selected into the aggregator and move to collecting/distributing, and you have a bundle.

When the two ends disagree about selection, the numerically better system priority + MAC pair wins control. Keys answer a quieter question, which ports on this box are even allowed to bundle. A port whose operational key doesn’t match gets left out no matter what the cabling says.

Now for what the standard refuses to specify. The one thing it’s firm about is that a distribution algorithm can’t reorder the frames inside a conversation, and can’t duplicate them either. Which algorithm you actually use to spread traffic across the links, it leaves entirely open. One rule of requirement, the entire algorithm left to the vendor, and that omission explains the ticket at the top of this post and about half of all link aggregation pain.

Active, passive, desirable, auto, and the mode named on

On Cisco gear there are three ways to bring up a bundle, and the taxonomy generalizes to everyone else:

PAgP (Port Aggregation Protocol) predates the standard. Cisco shipped working aggregation years before 802.3ad existed, and PAgP is the fossil of that head start. It survives because it can carry Cisco only state that a standards protocol can’t, a trick that matters for the multi chassis machinery. Cisco’s own guidance for any server or non-Cisco switch is to use LACP, and for anything new I’d extend that to Cisco-to-Cisco as well.

The failure matrix for the negotiated modes is at least honest. Passive plus passive never forms (neither side initiates), auto plus auto same, and PAgP facing LACP never forms because the two protocols have nothing to say to each other. Each of those failures is visible… the bundle refuses to come up and you go investigate.

Mode on fails worse, because static mode sends no control PDUs at all. Pair it with active or passive and the negotiated side waits politely forever, so no bundle forms. Pair on with on and it works… right up until a miscabled member or a config mismatch, and with no protocol running to notice, the reward is a forwarding loop or a silent black hole. LACP’s entire job is refusing to bundle when something is inconsistent. Mode on keeps the speed and throws away the safety net. I stopped deploying it years ago unless a device gives me no other option.

One more negotiation footgun. On Nexus 9000s, lacp suspend-individual is default behavior. A member port that hears no LACPDUs gets put in a suspended state, which is sensible loop protection. Now rack a server that PXE (Preboot Execution Environment) boots. Preboot firmware speaks exactly zero LACP, so the switch suspends the port, and the server can’t reach DHCP or TFTP. The “new server is DOA” ticket lands on the network team, of course. The fix is no lacp suspend-individual on server-facing port-channels so the members fall back to forwarding as individual links, and NX-OS 10.2(1)F added a narrower no lacp suspend-individual pxe that lets exactly one member go individual for the boot while the rest stay suspended, which keeps most of the loop protection. On a vPC it has to go on both legs.

The hash is none of the standard’s business

Back to the server team and their 10 gig backup on a 40 gig bundle.

Re-read the quoted sentence from the standard. The only requirement on frame distribution is that a conversation must never be reordered. The universal way implementers deliver that promise is pinning. Each conversation, however the platform defines one, is assigned to exactly one member link and stays there. Nobody stripes a flow across members, because striping is exactly the reordering the standard forbids.

Follow the pinning logic one step and the ticket resolves itself. A single flow can never move faster than the one member it’s pinned to. A 4x10G port-channel is a 40G pipe in aggregate, across many flows, and a 10G pipe for any individual backup stream, vMotion, or iSCSI session. The workloads most likely to prompt “we need a bigger pipe” show up as one giant flow, so more members change nothing for them. Faster members are the fix.

Every vendor picks its own fields

Since the standard is silent, each implementation chooses which header fields feed the pinning decision… source MAC, destination MAC, IPs, L4 ports, various XOR combinations. Defaults vary by platform and by generation, so the honest summary is rough. Classic Catalyst IOS leaned on MAC based hashing, modern IOS-XE platforms default toward src-dst-IP, and NX-OS defaults to src-dst-MAC for non-IP traffic and src-dst-IP for IP, with an option to fold in L4 ports. On classic IOS and NX-OS alike the load-balance method is a device global setting (per module on the Nexus 7000). You don’t get to choose it per port-channel.

Richer inputs spread better only if the traffic has entropy to offer. A thousand client connections spread beautifully once L4 ports join the hash. A storage replication session between one pair of IPs still pins to one member no matter how fancy the inputs get.

And the two ends of a bundle hash independently, a detail that catches even experienced engineers off guard. LACP negotiates membership, never distribution. Nothing in the LACPDU carries a hash algorithm, so the two ends of one bundle can and often do run different ones, and traffic from A to B can ride member 1 while the replies from B to A ride member 3. Completely legal, and mildly maddening the first time you’re capturing on the member that carries only one direction.

Eight buckets and the powers of two rule

Classic Catalyst hardware turns the vendor freedom principle into visible math. The switch reduces the chosen header fields to a 3 bit result, the Result Bundle Hash (RBH), a value from 0 to 7. Those eight buckets get dealt out among the members. With two members, one owns buckets {0,2,4,6} and the other {1,3,5,7} (the hardware masks are 0x55 and 0xAA if you go digging). Eight buckets divide evenly only when the member count is a power of two:

A 3 member bundle loads two members half again as heavily as the third before your actual traffic mix gets a vote. This is where the old “bundle in powers of two” design rule comes from, straight out of the bucket arithmetic.

Polarization, or every tier picks the same link

Stack bundles in tiers, access to distribution to core, and a subtler failure appears. If every switch runs the same hash algorithm on the same fields, a flow that hashed to bucket 0 downstairs hashes to bucket 0 upstairs too, at every layer. Traffic concentrates onto a subset of links at each tier while their siblings sit idle, and your redundant capacity quietly becomes decorative. Cisco calls this polarization.

The documented fixes attack the “same function, same input” premise. Vary the hash inputs per tier, src-dst-IP at one layer and add L4 ports at the next, so a flow that landed on link 0 below spreads differently above. The hardware fights back too. The Catalyst 6500’s default algorithm deliberately mixes source and destination with unequal weights specifically to resist polarization, and platforms mix in a per device hash seed so identical hardware running identical config still assigns buckets differently at each hop. ECMP (Equal-Cost Multi-Path) suffers the same disease and takes the same medicine, as anyone who’s chased uneven links in a Clos fabric already knows.

The flow pinning model itself is also the assumption cracking wide open in AI back end fabrics right now. A training collective is a handful of GPUs each producing one or two enormous flows at line rate, and a five tuple hash will happily land two 400G elephant flows on the same 400G uplink, stalling the entire job. That’s the backup server ticket with four more zeros on it, and it’s pushing that corner of the industry toward packet spraying and topology tricks instead of per flow hashing. The design side is covered in my writeups of Tyler Conrad’s CHI-NOG 13 talk on AI fabrics and Nokia’s NFD40 session.

Meanwhile, the servers never read the standard

Switches generally want to follow 802.1AX. Host operating systems are another story, and their track record is a twenty year catalog of creative standards evasion.

Linux kernel bonding ships seven modes, and exactly one of them is the standard. Mode 4, named 802.3ad, real LACP with a real switch side LAG. The others range from harmless to actively hostile.

The hostile one is mode 0, balance-rr, which round-robins packets across members. It’s the only mode that stripes a single flow, and therefore the only one that breaks the standard’s in order promise. Packets of a TCP flow arrive out of order, the receiver reads reordering as loss, duplicate ACKs fly, the congestion window collapses, and your “faster” bond delivers worse TCP throughput than a single link. The kernel documentation warns about this in plain language, describing “the peer systems receiving packets out of order, causing TCP/IP’s congestion control system to kick in.” It’s fine for a UDP benchmark, but it’s a footgun anywhere near production TCP.

Mode 2, balance-xor, is the static LAG version of the hashing we covered above, and it needs a static channel-group configured on the switch. People forget the switch half, and the resulting MAC chaos is a preview of the next section. For modes that hash, xmit_hash_policy picks the fields. layer2 by default, layer2+3 to add IPs, layer3+4 to let separate connections to the same peer use separate members (the kernel doc flags that last one as not fully 802.3ad-compliant around IP fragments).

Twenty years of lying to ARP

The stronger story is modes 5 and 6, because they belong to a lineage that predates them by a decade.

Before Windows Server 2012, Windows had no native NIC teaming at all. Teaming was the NIC vendor’s driver suite, Intel’s PROSet with ANS (Advanced Network Services) or Broadcom’s BASP (Broadcom Advanced Server Program), an intermediate driver wedged under TCP/IP presenting a virtual adapter. Both vendors offered a mode that load-balanced received traffic with zero switch configuration, and both pulled it off with the same sleight of hand. One team IP address, but every physical NIC keeps its own MAC, and the driver uses ARP to decide which clients learn which MAC.

Intel called it Adaptive Load Balancing with Receive Load Balancing, and Intel’s documentation describes the ANS driver using ARP to direct incoming traffic to a particular port. Broadcom’s Smart Load Balancing documentation spells the mechanism out with no embarrassment at all. The team “responds to different client ARP requests using the different MAC addresses of the teamed NICs,” and when load skews, the driver fires gratuitous ARPs on a client by client basis to re-point them at a different NIC’s MAC.

That exact trick is alive today as Linux bonding mode 6, balance-alb. The kernel documentation says it outright. The bonding driver “intercepts the ARP Replies sent by the local system and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server.” Client A learns eth0’s MAC for the server’s IP. Client B learns eth1’s. Inbound load spreads across NICs, and the switch was never consulted. (Mode 5, balance-tlb, is the same family with transmit balancing only, all inbound traffic still lands on one member.)

The bill for all of this gets paid by the switch. MAC learning is built on the assumption that a MAC address lives behind one port. An ALB family team sources frames from multiple ports using multiple MACs and moves clients between them whenever it rebalances, so the switch watches MACs hop from port to port. The gentle outcome is MAC flap log spam and unstable CAM (Content-Addressable Memory, the switch’s MAC table) entries. With port-security or MAC-move limits configured, it’s err-disabled ports and dropped traffic. My favorite documented case involved a Linux box whose only misbehaving frame was LLDP (Link Layer Discovery Protocol)… the daemon sourced it every 30 seconds from the bond’s MAC instead of a slave’s, which flapped the switch entry hard enough to cause intermittent connectivity. Intel shipped this trick in the early 2000s, Broadcom carried it through the BASP era, and Linux maintains it in every current kernel.

Windows walked away on purpose

Server 2012 finally shipped native teaming, LBFO (Load Balancing and Failover), with a genuine LACP mode, and for a few years the sane Windows answer was an LACP team into a switch LAG. Then Microsoft changed direction. Server 2016 introduced SET, Switch Embedded Teaming, which folds teaming into the Hyper-V virtual switch itself, up to eight identical NICs, and it’s the required teaming for Storage Spaces Direct, Azure Stack HCI, and RDMA converged designs. SET supports switch independent mode only, with no LACP and no static LAG. Microsoft’s own wording is that SET “supports only switch-independent configuration,” which means the switch ports facing a SET host stay plain ports with no bundle on them at all. The old path is being bricked up behind it as well. Attaching a Hyper-V vSwitch to an LBFO team was formally deprecated in Server 2022 (Microsoft announced it back in 2020, and a PowerShell -AllowNetLbfoTeams override kept it limping along), and in Server 2025 the override is obsolete and the bind fails outright. LBFO itself still works for everything that isn’t Hyper-V.

On the platforms Microsoft cares most about, the sanctioned design has no link aggregation protocol on the wire at all. The host pins each virtual NIC to a physical port and handles failover itself. To be fair, SET behaves far better than its ARP rewriting ancestors, per vPort MAC pinning means a given MAC stays put on one port until a failover happens, so it doesn’t grind the switch’s MAC table the way ALB does. It’s the same switch independent philosophy, implemented by someone who’d read the incident reports.

The host side arc bends one direction. Red Hat deprecated its alternative teaming stack (libteam) in RHEL 9 and removed it in RHEL 10, kernel bonding won that fight, and Microsoft dropped LACP from its flagship teaming entirely. The server world has decided it would rather manage aggregation itself than coordinate with the switch. As the guy on the switch side of that arrangement, I’ve made my peace with the MAC pinning versions… the ARP rewrite versions I’ll keep escalating.

The part where one switch stops being enough

Everything in this post assumed one box on each end of the bundle, which is the world the base standard describes. The obvious next demand breaks the assumption. If I’m bundling four links for redundancy, why do all four terminate on a single switch that can still fail as a unit?

Spreading one LAG across two chassis is where vPC, MLAG, MC-LAG, VLT, and their cousins live. The whole family is held together by a shared system ID trick, two switches agreeing to impersonate one LACP system, plus a pile of proprietary state synchronization to keep the impersonation consistent. It also comes with the best plot twist in this entire topic… the IEEE standardized multi chassis aggregation years ago, and nearly the whole industry shipped proprietary code instead. I’ve been poking at this problem since the early Nexus vPC days in 2011, and it has earned its own post. The multi chassis machinery, the peer-link failure modes, and the precise boundary of what interoperates across vendors are all waiting there.

What’s the worst link aggregation ticket you’ve fielded, and which of the seven names did it arrive under? Tell me in the comments, especially if it wore a name I didn’t list.

No vendor had a hand in this post and nobody paid for it. The opinions here are mine, and so are any errors. For more, please read my full disclaimer.