What NFD40 Taught Me About AI Networking

Tony Mattke · 2026.07.07 · 13 min read

I went into NFD40 thinking the AI networking pitches were going to be the marketing layer dressed up in new vocabulary, and I came out convinced it’s a different design problem. Ethernet, BGP, and a lot of the merchant silicon underneath are the same. What changes is the math, the failure modes, and the rules for what counts as “fine.” This post is the wrap-up… the things I walked away with after a week of vendor sessions and hallway conversations.

I opened my Nokia writeup from this same event saying none of my thirty years of networking, leaf-spine fabrics, ECMP, and microsecond-grade financial network design had prepared me for the port math inside an AI data center. A week later, the port math turned out to be the smallest of the recalibrations I had to make.

The math is different

The single hardest thing for a generalist network engineer to internalize is that AI cluster traffic doesn’t look like cloud traffic. The flows aren’t small and numerous, statistical load balancing doesn’t work on them, and jitter tolerance is essentially zero.

AI training does its work in collectives. The most common one is all-reduce, the operation where every GPU in the cluster simultaneously broadcasts its piece of the gradient, then sums everything together so all GPUs end up with the same combined result. All-to-all is similar but bidirectional. Reduce-scatter and broadcast are variants. The detail that matters: when a collective fires, every GPU is talking to every other GPU at the same time, and the entire collective doesn’t finish until the slowest path finishes.

ECMP has always assumed many small flows, and a hash collision doesn’t matter when the flows that collide are short-lived and each far too small to fill the link on its own. In a collective, you have a small number of GPUs each generating one or two enormous flows that burst at line rate. Five-tuple ECMP can absolutely hash two 400G elephant flows onto the same 400G uplink. That one hot spot is enough to stall the entire collective. The GPUs sit idle even though no link is technically saturated according to any metric short of microsecond queue depth.

Upscale’s Aravind Srikumar put it cleanly during their session: “this isn’t head-of-line blocking in the networking sense. The network might experience just one packet drop and recover instantly. But the GPUs cannot proceed because they need consistent state across the collective. The compute stalls even though the network is healthy.”

I’ve spent a career working on networks where a clean fabric meant a happy workload, but today’s AI back-end fabrics sever that link.

The port math is brutal, and it takes its own racks now

Once you accept that you need non-blocking paths between every GPU pair, the port math gets ugly fast.

Nokia’s Patrick McCabe walked through it in their validated-design segment. A single 8-GPU server consumes eight backend leaf ports (one per GPU NIC at 400G), plus two more NICs for frontend and storage connectivity. That’s ten physical switch connections per server before you’ve plugged in a management cable. A 2,048-GPU cluster works out to 256 servers and 2,560 server-facing ports. Manageable. Scale it to 100,000 GPUs and you’re at 12,500 servers and 125,000 server-facing ports, all needing non-blocking paths through a multi-tier fabric, all of which has to deliver 40 Pbps of aggregate southbound bandwidth.

There’s no comfortable oversubscription ratio to fall back on, and no “good enough” load balancing to lean on. When an all-reduce fires across tens of thousands of GPUs, every path matters. The AI back-end side of a data center needs far more switches than the cloud side at equivalent scale, with a different topology, different protocols, and different optics stacked on top of the raw box count.

All those boxes have to go somewhere, and that reshapes the floor plan. One slide reframed how I thought about deployment math for the rest of the week:

A 512-XPU pod sized for a 25.6T fabric, built with OSFP-based switches, needs more switch racks than compute racks. Four racks of XPUs, eight racks of network. The network became the bigger footprint. That’s a sentence you couldn’t honestly say in a cloud data center a decade ago, when network gear was a couple of switches at the top of each rack that nobody budgeted floor space for. In AI back-end fabrics, the network is its own logistical problem. Floor space, cooling, power, the works.

This is the picture sitting behind every optics pitch at the event. Vendors care so much about density because without it, the data center floor plan stops working. You run out of contiguous rack space for the fabric before you run out of demand for GPUs.

Arista’s answer is XPO. The same 512-XPU cluster, rebuilt with XPO switches:

Two switch racks instead of eight. Same compute capacity, roughly half the total floor footprint. This is the visual that sells the optics roadmap to anyone who has to negotiate rack allocation with the data center team.

The math underneath:

Move from a 4OU OSFP chassis to a 1OU XPO switch and you fit 32 switches in a single ORv3 rack at 6.4 Pbps of total capacity. Four times the rack density, and the network footprint compresses back under the compute it serves.

Then I thought about who has to cable that damn thing. Thirty-two 1OU XPO switches stacked in a single ORv3 frame, every port fanning out into the fabric, adds up to a mountain of breakout cables just to make the port counts land. I’d hate to be the tech dressing that rack. (We’re probably headed for breakout-cable hell regardless, once you count the octal OSFP fan-outs, but cramming 32 switches into one frame turns “a lot of cables” into “where do I even start.”)

The whole pitch is operationally driven. You can’t oversubscribe your way out of the port math, and the rack space won’t stretch to cover the difference, so density becomes the only lever left.

Which is why optics suddenly matters again

Optics was the thing nobody could agree on, and that disagreement itself is a tell. The vendors had three competing answers.

Co-packaged optics (CPO) moves the photonics onto the same substrate as the switch ASIC. You eliminate the discrete pluggable module, you save power on the SerDes hops between chip and faceplate, and you get density and efficiency improvements that the press materials love. Cisco demoed it and said volume shipping in the second half of 2026. Multiple vendors have CPO on the roadmap.

The catch came in passing during Vijay Vusirikala’s segment for Arista, and it landed harder than the slide it was on. If one channel out of sixty-four in a CPO switch fails, you replace the entire switch. There is no module to swap. The pluggable model that’s kept optical repairs cheap and boring for twenty years is gone. At hyperscaler scale, operators need a channel failure to cost them a module swap and nothing more, and CPO can’t offer that.

Linear pluggable optics (LPO) is Cisco’s middle ground for the near term. Drop the DSP from the optic, drop the power (under 8W per port versus sixteen), keep the pluggable form factor. It’s a sensible incremental move that trades peak density for a form factor operators already run. Scott Robohn flagged the question that didn’t get a clean answer on stage: where does the signal-conditioning complexity go when you remove the DSP? Probably into the ASIC, but Cisco wasn’t ready to commit to that. The same question is waiting for CPO and will be harder to dodge then.

Extra dense pluggable optics (XPO) is Arista’s answer, built as a multi-source agreement (MSA) so multiple vendors make interchangeable modules. Keep the pluggable model, get most of CPO’s density, and hold onto the RMA workflow and procurement playbook operators already run. It’s a conservative bet, and that’s what makes it a good one.

Nokia stayed studiously neutral. Their stance was “we support OSFP, LPO, XPO, whatever the market lands on.” Reasonable hedge for a vendor who can’t dictate where the optics community ends up.

What I took away: the winning form factor probably isn’t settled yet, and it’ll be decided on serviceability at scale, whether operators can still swap a broken lane instead of writing off a whole switch. Lanes-per-millimeter is the spec everyone quotes. RMA math is what picks the winner.

Scale-up, scale-out, and scale-across are three different problems

The third architectural thing the week clarified for me is that “AI networking” is three different problems wearing one name, and most of the confusion in the room came from people collapsing them together.

Scale-up is GPU-to-GPU within a rack (sometimes across two racks now). The traffic is load/store memory operations against a flat address space. The latency target is sub-microsecond. Topologies are single-stage. Protocols are deliberately stripped down (UALink, ESUN, NVLink). Jitter is what kills you.

Scale-out is GPU-to-GPU across scale-up domains, within the same cluster or data center. The traffic is RDMA memory copies. The latency target is single-digit microseconds. Topologies are multi-tier leaf-spine. Protocols are RoCEv2 with all of PFC, ECN, and DCQCN holding the line. Drop a packet and the whole collective stalls.

Scale-across is GPU-to-GPU between scale-out clusters, typically across data center sites. Traffic still looks like RDMA semantically, but the latency budget jumps from microseconds to tens of milliseconds because physics (and fiber distance) take over from engineering. The silicon shifts from switching-class to routing-class: Nokia pitched Jericho 3 specifically for this tier because the WAN-shaped topologies need deeper buffers and longer-haul features that scale-out switches don’t carry. Throughput is the constraint, because federated training jobs that span sites need to move model state at line rate over distances where round-trip time can’t be engineered away. Scott Robohn’s NFD40 wrap-up made scale-across the centerpiece for a reason. It’s the tier where the next round of differentiation is still open.

These aren’t variants of each other. They’re different problems requiring different silicon, different congestion handling, and different operational models. Upscale was the clearest on the scale-up/scale-out split: “you cannot meaningfully optimize a single switch for both.” Add scale-across and the argument gets stronger. Nokia validated it with hardware, building separate silicon families for each role (Tomahawk 5 and 6 for scale-out, Jericho 3 for scale-across). Arista carved their portfolio along similar lines.

If you’re building an AI cluster and you find yourself buying one switch SKU for everything, you’re probably leaving performance on the table somewhere. Or you’re a small enough deployment that it doesn’t matter. Both are valid answers, as long as you know which one you’re building.

What surprised me

Six things I didn’t expect, in rough order of how much they shifted my mental model.

1. Hardware is commoditized and everyone admits it now. Even Cisco, the historic vertical-integration vendor, ships N9100s with NVIDIA Spectrum-X silicon because customers prefer that bundle. During Arista’s session, a delegate looked at the load-balancing slide and said it could have any vendor’s logo on it. Tom Emmons agreed. Brendan Gibbs enumerated Arista’s hardware differentiators in the opening, but the real differentiators sit in the control plane, observability, and automation now.

2. Multi-tenancy is mandatory even when you have one tenant. Netris made this point and it stuck. When a GPU server fails, you need to move it into an isolated tenant context so it can be safely returned, repaired, and re-inserted without leaking state. If your automation can’t model multi-tenancy from day one, the RMA workflow breaks. Single-tenant AI clusters still need multi-tenant fabric primitives.

3. Data hygiene is the real AI Ops blocker. Three different vendors (Aviz, Selector, NetAI) independently said the same thing: the LLMs and GNNs are the commodity layer. The expensive thing is metadata quality. Interface naming conventions, circuit IDs, BGP descriptions, accurate inventory. Get those right and AI ops works. Skip them and you’re handing the model a corrupted input and asking for magic.

4. The CPO failure math compounds in a way the density slides hide. Sixty-four optical channels per switch means a per-channel failure rate rolls up into a much higher per-switch RMA rate. At hyperscaler scale, that compounding is the number operators budget around, and it rarely shows up next to the lanes-per-millimeter figure everyone leads with.

5. The credible vendors admitted what they don’t do. Upscale opened their session with “we haven’t announced any products yet.” NetAI flatly told us they only do networking, not storage or database RCA. Netris said 40% of the clusters across their customer base are still on InfiniBand, so Ethernet hasn’t swept the backend yet. The vendors who refused to oversell their scope earned more trust than the ones with the slickest decks.

6. The interesting fights have moved to the operational layer. The optics RMA story, the multi-tenancy primitive, the metadata hygiene problem. These are all operations problems. The protocols (UEC, UALink, ESUN, RoCEv2) are converging fast and getting boring, while the operational arguments are where the real disagreements live now, and where the next decade of vendor differentiation gets fought.

The takeaway

AI networking is a separate design problem with separate constraints, and the vendors who treated it that way landed better than the ones who tried to reuse their existing pitch slides. The math is different, the port counts are bigger, the failure modes are categorically new, and the optics situation is wide open.

If you’re a senior engineer trying to figure out which side of this category your career sits on, stop asking “do I know AI networking yet?” Ask “do I understand why my old assumptions don’t work, and what set of new assumptions to replace them with?” That’s the recalibration NFD40 forced on me, and it was worth the week.

Disclosure: I attended Networking Field Day 40 as a delegate. My travel and accommodations were covered, but I was not compensated for this post and the opinions are my own. For more, please read my full Tech Field Day disclaimer.

Links & Resources

Networking Field Day 40

What NFD40 Taught Me About AI Networking

The math is different

The port math is brutal, and it takes its own racks now

Which is why optics suddenly matters again

Scale-up, scale-out, and scale-across are three different problems

What surprised me

The takeaway

Links & Resources

Other Delegate Posts

Tags :

Related Posts

Upscale AI at NFD40: The Pitch Before the Product

Selector at NFD40: Show Your Work

Networks Are Graphs. NetAI Acts Like It.