Link Aggregation Deep Dive, Part 2: The Multi-Chassis Lie

Log into a Linux server that’s dual-homed to a pair of data center switches and check its bond. cat /proc/net/bonding/bond0 prints exactly one partner System ID. As far as that server knows, both of its uplinks terminate on a single switch. They don’t. They land on two separate chassis with separate power feeds, separate supervisors, and separate configurations, and those two boxes work continuously to keep the server from finding out.

I’ve been complicit in this since 2011, when I moved a core to a pair of Nexus 7010s running vPC and spent the next few months writing up the features as I tripped over them. Fifteen years later, every vendor still sells the same trick under a different name, and the failure modes I got paged for back then are the same ones people are getting paged for today.

This is Part 2 of the link aggregation deep dive. Part 1 covered the single-box fundamentals. LACP negotiation, the hash, and the ways hosts wander off the standard. This half is about what happens when the bundle spans two switches, why every mainstream implementation of that idea starts with a forged identity, and the strange fact that the IEEE standardized a proper way to do it… which nearly everyone ignored.

Why the standard’s model stops at one box

The 802.1AX standard builds everything on the Actor/Partner model. Each end of a link is a System, identified by a System Priority plus a MAC address. The selection logic that decides which ports may aggregate together answers one question per port. Which of my links connect to the same partner System? Different partner System IDs never share an aggregation. A LAG, in the standard’s base model, terminates on exactly one system at each end.

That model works until you ask the obvious redundancy question. A bundle protects you against a link dying, and does nothing for the switch dying. So you dual-home the server, one uplink to each of two switches, and the LACP math breaks. Two switches means two MACs, which means two partner System IDs, and the server’s bond refuses to aggregate the pair. You get two one-link bundles instead of one two-link bundle, and something (spanning tree, the bond’s failover logic, your own sense of defeat) parks one of them in standby. The redundancy you cabled for collapses back into active/standby.

There are two ways out. Make the two switches into one system, or keep them as two systems that lie about it. The industry built both, under at least a dozen brand names.

One lie, many accents

Whatever the brand name, the opening move is identical. Both switches advertise a single shared LACP System ID southbound. The downstream device runs bog-standard 802.1AX, sees one partner on both links, and bundles them exactly like the standard says it should. The spoof weaponizes the standard’s own selection logic.

Each vendor forges the signature a little differently. Cisco vPC derives a reserved MAC in the 00:23:04:ee:be:xx range keyed to the vPC domain ID. Arista’s MLAG System ID is, in their own documentation’s words, “the MLAG domain’s MAC address, automatically derived when the MLAG forms,” and it matches the bridge MAC of neither peer. Cumulus/NVIDIA carves its clagd-sys-mac out of a reserved 44:38:39:ff:00:00 through 44:38:39:ff:ff:ff range. Juniper MC-LAG skips derivation entirely and has you configure the shared lacp system-id on both peers by hand.

A shared identity is the easy part. Any parent who’s split up two kids to ask who broke the lamp knows the lie was never the hard part. Getting both stories to match, question after question, is where it falls apart. Two switches face that interrogation every time a peer reboots, an upgrade lands, a link flaps, or somebody fat fingers a config, and the rest of this post is about how each pair keeps its story straight.

The three camps

Strip off the brand names and every multi-chassis implementation I know of is one of three control plane designs.

1. One control plane. Merge the boxes into a single logical device. One brain, one config, one management IP. Cisco VSS (Virtual Switching System) and StackWise Virtual, HPE Comware’s IRF (Intelligent Resilient Framework), Juniper Virtual Chassis, Aruba VSF (Virtual Switching Framework), Extreme SummitStack. IRF stacks up to nine of HPE’s fixed port switches, and cross member LAG comes free, because as far as the one control plane is concerned it’s just a bundle whose ports happen to live in different sheet metal.
2. Two control planes kept in sync. Both boxes keep their own brain and gossip state to each other over a dedicated link. Cisco vPC, Arista MLAG, Juniper MC-LAG, Dell VLT (Virtual Link Trunking), Aruba VSX (Virtual Switching Extension), Cumulus CLAG (Cumulus’s name for MLAG), Extreme MLAG, SONiC MCLAG. Hard two-node limit, everywhere.
3. No peer link at all. Let BGP EVPN coordinate the switches through the routed fabric. Ethernet Segments and ESIs (Ethernet Segment Identifiers), per RFC 7432. The camp everything is slowly migrating toward.

The fork shows up inside vendor catalogs, because several vendors sell more than one camp. Juniper offers Virtual Chassis (one brain) and MC-LAG (two brains). HPE is the best illustration in the industry. IRF on Comware merges switches into one control plane, while VSX on AOS-CX keeps two switches deliberately separate, with independent routing protocols and independent software upgrades. Same company, opposite answers, because the trade is real. One brain is operationally simple and shares its fate, the Bad Boys arrangement… we ride together, we die together. Bad boys for life, right up until the stack link drops and they both decide they’re Mike Lowrey. Two brains upgraded independently can diverge and disagree.

Camp one: one brain, one blast radius

VSS fused two Catalyst 6500s into a single virtual switch. One supervisor runs active, the other hot-standby via SSO (Stateful Switchover), connected by the VSL (Virtual Switch Link), a special port-channel carrying control traffic and inter-chassis data. To every neighbor it’s one switch with one config. Downstream bundles that land on both chassis are Multichassis EtherChannels (MECs), and they work because only one control plane answers LACP.

StackWise Virtual is the same idea re-implemented on IOS-XE for the Catalyst 9000s, with the SVL (StackWise Virtual Link) running over front-panel links since the 9400/9500/9600 have no rear stacking ports.

The failure that keeps camp one operators up at night is dual-active. If the VSL or SVL dies completely, the standby chassis can’t distinguish “my link to the active died” from “the active died.” So it goes active. Now two chassis are alive on the network with the same IP addresses, the same router IDs, and the same LACP System ID, and neither knows the other exists. A single L3 identity, claimed twice, is the catastrophic version of split-brain.

The guardrail is dual-active detection, and the mechanisms differ by platform in a way people constantly get wrong. VSS offered three. Enhanced PAgP (Port Aggregation Protocol) through a shared downstream MEC neighbor (the downstream switch relays “I can still see your other half”), L3 BFD on a dedicated inter-chassis link, and Fast Hello. StackWise Virtual kept enhanced PAgP and Fast Hello and dropped BFD, and its default recovery action is a reload of the chassis that loses the argument. The enhanced PAgP option also explains why Cisco kept a proprietary aggregation protocol alive two decades after LACP shipped. PAgP frames carry Cisco-specific TLVs, including the dual-active detection trailer, and a standards protocol has nowhere to put that.

IRF has the same disease and its own medicine, MAD (Multi-Active Detection). Every architecture that collapses N boxes into one brain ends up with some flavor of out-of-band tiebreaker plus a loser that shuts itself down.

Camp two: two brains and a gossip channel

The two-brain camp is where most of the world’s data center access lives, so it deserves the longest look. Conveniently, every member of this camp has the same anatomy. A peer link for state sync, a separate keepalive to distinguish peer-death from link-death, a role election, and an orphan-port sob story. Each vendor reinvented all four.

vPC, the one I have scar tissue from

vPC keeps two independent Nexus control planes on purpose. That was a genuine philosophy fight inside Cisco’s own portfolio, VSS collapsing two boxes into one brain while vPC kept two and synced only enough to fake one LACP partner. vPC won the data center because two brains means upgrading one Nexus at a time without dropping the pair.

The peer-link is a normal port-channel between the peers. It carries the sync protocol, Cisco Fabric Services (CFS), which keeps MAC tables, IGMP snooping state, and ARP in step, plus BUM traffic (broadcast, unknown unicast, and multicast) and anything destined for orphan ports. The peer-keepalive is a separate routed heartbeat, UDP 3200 by default at roughly one-second intervals with a five-second timeout, that carries no data at all. Its only job is answering one question. Is my peer dead, or is the peer-link just down? The forwarding plane is active-active, both peers forwarding vPC traffic all day, with the primary/secondary roles mattering almost exclusively at failure time.

Configuration consistency is enforced in two tiers. Type-1 parameters (STP mode, MTU, port type) suspend the vPC on mismatch. Type-2 parameters get logged while the vPC keeps forwarding, and the allowed-VLAN list is Type-2. Mismatch it and the mismatched VLANs get pruned while everything else keeps running. That pruning behavior has eaten entire change windows while everyone stared at the port-channel status insisting it looked fine.

Peer-link down with keepalive still up. The secondary concludes the peer is alive, suspends its own vPC member ports and SVIs to prevent a loop, and the primary carries everything. Correct behavior, with one brutal side effect. Any device single-attached to the secondary (an orphan port) just lost its only path. Orphans are the recurring casualty of vPC failures, which is the argument for dual-homing everything and for vpc orphan-port suspend. The nightmare ordering is the reverse, keepalive first and then the peer-link. Each peer now believes it’s alone, both assume primary, and you have two uncoordinated brains advertising the same LACP System ID. The keepalive exists purely to make that ordering survivable.

If you want the config level walkthrough, my 2011 vPC posts still hold up, peer-gateway quirks and all, written in the same stretch as my introduction to the Nexus 1000V.

Arista MLAG, the same skeleton with different defaults

Arista’s MLAG should feel familiar to anyone who’s run vPC. A peer-link port-channel carries the control session (TCP between the peers) and orphan traffic, a heartbeat runs at a four-second default, and peers continuously sync MAC and state. Split-brain protection is dual-primary detection, which needs an out-of-band path independent of the peer-link, and resolves by having the secondary errdisable its MLAG member ports.

The detail worth filing away is reload-delay. When a peer reboots, its non-peer-link ports stay errdisabled until the control plane converges, and the folklore number everyone quotes is 300 seconds. The real default is ASIC-family-dependent, anywhere from 300 seconds on typical fixed switches up to 1800 on Sand modular platforms. If your maintenance window math assumes five minutes and your platform ships thirty, that’s a LONG silent half hour, and plenty of time to explain to the change board why the hitless upgrade has been hitting things since 2am.

Juniper MC-LAG, the one with the reputation

MC-LAG is Juniper’s two-brain answer, coordinated by the Inter-Chassis Control Protocol, ICCP (a TCP control channel between peers that replicates config parameters and forwarding state), plus the Inter-Chassis Link (ICL), the data path between them. It works, and it’s also the implementation operators complain about most, for specific reasons. One letter off from ICP, and after your first split-brain you’ll swear the clowns wrote it.

Start with the split-brain behavior. If the ICCP adjacency drops while both peers are alive, both can revert the shared LACP system-id to their own local defaults. The downstream device accepts whichever system-id shows up first and brings down the links carrying the other one. Read that again. The redundancy machinery reacting to a control-channel blip takes down half the bundle it exists to protect.

The mitigations exist and they’re fiddly. Backup liveness detection adds an out-of-band keepalive over the management network so a peer can tell “ICCP died” apart from “my peer died.” And then there’s prefer-status-control-active, which keeps the standby peer from reverting the system-id during an ICCP failure, and whose documentation warns it’s only safe if ICCP will never go down unless the remote peer itself goes down. A knob whose safety precondition is “this failure never happens independently” is a seatbelt rated for every crash except the one you’re about to have.

Juniper has effectively settled the argument themselves. MC-LAG is unsupported with VXLAN, so in an EVPN-VXLAN fabric you’re using ESI multihoming whether you like it or not. And multi-vendor MC-LAG, where an MX pairs with someone else’s box? That has never existed outside slideware, for reasons the interop section below makes precise.

The quick tour of everyone else

Cumulus/NVIDIA CLAG runs clagd over a peerlink.4094 subinterface, with a backup-ip playing the keepalive role. Two things distinguish it. Roles govern STP BPDU origination while the data plane forwards active-active on both peers, and configuration is deliberately never synced. The docs are blunt that only dynamic state is synchronized, so you hand-mirror both configs and own the drift. NVIDIA now openly positions EVPN multihoming as the replacement, peer-link deleted, more than two switches per redundancy group.

Nokia’s classic MC-LAG on SR OS runs active/standby toward the access node, using LACP to hold only one peer’s links active at a time with the other side parked. If you’ve internalized vPC’s active-active forwarding as “how MLAG works,” Nokia will surprise you. Half your uplinks sit idle on purpose. In exchange, there’s never a question of which peer owns a flow, so failover is a clean handoff from one box to the other instead of two boxes renegotiating who forwards what. Nokia traded bandwidth for predictability, which is about as service provider a trade as it gets. The peers coordinate over an IP session to UDP port 1025 (fixed, not a knob), and Nokia’s fabric-era answer is, again, EVPN multihoming.

Dell VLT, Aruba VSX, Extreme MLAG, and SONiC’s MCLAG all follow the two-brain pattern with their own vocabulary (VLT interconnect, inter-switch link, inter-switch connection, and an ICCP-Lite daemon on TCP 8888, respectively). VSX puts the camp’s whole selling point right in the brochure. Two separate control planes exist specifically so you can upgrade one member while the other keeps forwarding.

Stand back from the zoo and the pattern is hard to unsee. Eight vendors, eight names, eight independent implementations of the same four components, none able to talk to the others. The obvious assumption is that the IEEE never got around to it. Well, that assumption is wrong…

DRNI: the standard existed, and the industry shrugged

Here’s the part of this story almost nobody knows, and the reason I wrote this post.

In 2014 the IEEE standardized multi-chassis link aggregation. 802.1AX-2014 added Clause 9, DRNI (Distributed Resilient Network Interconnect), and if you’ve read this far you already know the parts under different names. The Portal is the MLAG pair, a group of switches that, in the standard’s words, “appears to each of their attached networks to offer a single pathway.” The Intra-Portal Link is the peer link. DRCP (Distributed Relay Control Protocol) is the sync protocol, the IEEE’s answer to CFS and ICCP. The difference is that DRCP is a real, published protocol any vendor could implement, and the 2014 edition allowed up to three switches per Portal where every vendor stopped at two.

H3C implemented it, and through the shared Comware lineage so did HPE, shipping it as DRNI with actual DRCP running between the peers, documented and maintained across multiple Comware releases. Which makes Comware the odd platform out twice over, running IRF for the one-brain camp and the IEEE’s own mechanism for the two-brain camp while everybody else shipped an invented one.

Everybody else means everybody. Cisco, Arista, Juniper, Dell, Nokia, Cumulus, Extreme, the SONiC community… all of them kept their proprietary System-ID spoofs. The IEEE’s 2020 revision of 802.1AX quietly told you how the fight ended. It simplified DRNI from three systems per Portal down to two, citing implementation experience, which is to say the ambitious version had no takers and the standard trimmed itself to match the proprietary protocols already in the field.

Before anyone gets romantic about the holdout, H3C’s DRNI doesn’t interoperate with other vendors either. A standard one company implements is a proprietary protocol with better citations.

The timing explains a lot. Nortel was shipping Split Multi-Link Trunking on the Passport 8600 in 2001, and Cisco’s vPC landed on the Nexus 7000 in 2009, five years before the IEEE published its mechanism. By the time they finally got around to ratifying DRNI, Arista MLAG, Dell VLT, and Juniper MC-LAG were already carrying production traffic. No vendor rewrites a shipping, revenue carrying HA feature to match a standard that arrived thirteen years after the first one shipped.

EVPN ESI: the escape hatch that fails silently

The multihoming standard that won out came from the IETF, in RFC 7432, as a side effect of EVPN.

The model throws away the peer link entirely. The set of links from one host to multiple switches is an Ethernet Segment, identified by a 10-byte ESI configured (or derived) identically on every switch attached to it. Coordination happens in BGP. Type-4 Ethernet Segment routes let the switches discover each other and elect a Designated Forwarder, the only member allowed to send flooded BUM traffic down to the host, so duplicate delivery gets prevented without any peer-link. Type-1 Ethernet A-D routes do double duty at two different granularities. The per-ES flavor gives you mass withdrawal (one withdrawn route pulls an entire failed segment, in place of thousands of individual MAC withdrawals), and the per-EVI (EVPN instance) flavor gives you aliasing (forward toward a switch that never locally learned the MAC). All-active by default, redundancy scoped to the interface, and no two-node limit.

The host still gets told the same old lie, mind you. Southbound it’s still one LACP bundle wanting one partner System ID, and the ESI machinery even formalizes the connection. A Type-1 ESI value (an unfortunate naming collision with the Type-1 route) is auto-derived from the CE’s LACP System MAC plus the port key.

So EVPN ESI is the genuinely multi-vendor answer, and it mostly works. Hear the failure mode before you bet a fabric on it, though. MLAG failures are loud. Ports suspend, peers errdisable, split-brains melt down in ways your NOC cannot miss. EVPN interop failures are silent. The BGP sessions come up clean, the routes are sitting right there in the raw BGP table, and they never make it into the EVPN instance or the MAC table, because route-import policy filtered them without logging a word.

The documented Cisco-Juniper cases are the canonical set. NX-OS auto-derives route targets (RTs) as AS:VNI. Junos doesn’t auto-derive at all and wants explicit RT configuration, so mismatched imports drop routes on receive. Configure identical route distinguishers on two multihoming peers and they’ll discard each other’s Type-1 routes as duplicates, which breaks aliasing with no alarm. And Juniper changed its ESI auto-derivation format along the way. There is literally a Juniper KB article titled, “EVPN AA interoperability issue due to ESI value format update.” All three teach the same field lesson. In a multi-vendor fabric, hand-assign Type-0 ESIs and explicit RTs, so no vendor’s auto-derivation can drift out from under you.

This matters more every year because EVPN is where the industry lives now. The AI fabric conversations I covered at NFD40 treat EVPN as the assumed substrate, and Nokia’s AI-era session was steering everyone toward EVPN multihoming over any two-box construct. The GPU back-end designs opt out of the whole game. As Tyler Conrad laid out at CHI-NOG, every GPU NIC is its own first-class fabric attachment, no MLAG, no bond, nothing multi-chassis at all. The newest networks looked at all of this machinery and decided the cleanest multi-chassis LAG is none.

The exact shape of the interop wall

The interoperability rule fits in two sentences.

East-west never works. You cannot build one multi-chassis pair out of two vendors. A Cisco switch and an Arista switch will never form a vPC/MLAG domain together, because everything between the two chassis (the sync protocol, the consistency checks, the role election, the derivation of the shared System ID) is proprietary and mutually unintelligible. This is a design choice with a business model attached.

North-south always works. Any device speaking plain 802.1AX LACP can dual-home into anyone’s pair. A Linux bond into a VLT pair, an ESXi host into VSX, a Catalyst into Arista MLAG, all fine, every day, everywhere. The pair presents its forged System ID, and the downstream box neither knows nor cares that its partner is two liars in a trench coat.

One asterisk on the always. Across several vendors’ forums there’s a reported bug class where a primary-peer reboot flaps the advertised system-id and bounces the downstream bundle. Remember the two kids and the lamp? This is the part where one says “the dog did it” and the other says “what dog?” The story cracks, and the standard-compliant downstream does the standard-compliant thing and tears your bundle down.

Why didn’t they just use the standard?

Because the checks had already cleared, and standards don’t sell switches. Lock-in does. The engineering cost of converging on DRCP would have bought vendors a feature their customers weren’t asking for and a compatibility story that actively hurts switch sales, since an interoperable pair is a pair your customer can finish with somebody else’s hardware. H3C shipping real DRCP proves the standard was perfectly buildable. The rest of the logos prove nobody with a dominant install base wanted or needed it.

If you skipped straight to the end, here’s the TL;DR. The IEEE defined single-system aggregation, and vendors extended it across two boxes by spoofing the one field the standard trusted. The IEEE then standardized the extension properly, and the industry, one vendor family excepted, declined to adopt it. The escape hatch that finally stuck came from a different standards body solving a different problem, still tells the host the same lie, and fails in silence instead of flames. Every one of those outcomes was a rational decision by someone, and the sum of them is why you’re maintaining a peer-link tonight.

If you’re running any of these pairs, know which camp you’re in, where your keepalive runs, and which of your devices are orphans. And if you’ve got a split-brain war story, put it in the comments. Bonus points if both switches thought they were right. (Oh, and I promise not to tell your change board.)

This one’s a pure deep dive. No event, no briefing, and no vendor involvement, everything here comes from public documentation, the standards themselves, and a long stretch of running this gear myself. The opinions are mine. For more, please read my full disclaimer.

Links and references

• Link Aggregation Deep Dive, Part 1, the single-box fundamentals this post builds on
• IEEE 802.1AX-2014, the link aggregation standard edition that added DRNI (Clause 9)
• RFC 7432, BGP MPLS-Based Ethernet VPN, Ethernet Segments, ESIs, and DF election
• Cisco vPC best practices design guide (PDF)
• Arista EOS user manual, MLAG chapter
• Juniper MC-LAG best practices
• H3C DRNI configuration guide, the one faithful DRCP implementation
• NVIDIA Cumulus Linux, EVPN multihoming