Matt Griswold Is Building the IX He Argued Against (CHI-NOG 13)

In 2015, Matt Griswold was fresh off launching United IX here in Chicago, his own gear, an investor behind him, and he spent that era arguing a commercial, privately owned exchange was the right way to build one. Eleven years later he came back to CHI-NOG and took the other side. “I spent a lot of time arguing why commercial private was the way to go in 2015. I think I was right at the time. I’m wrong. I would be wrong to do that now.”

He still thinks the 2015 argument was correct in 2015. What changed is the world underneath it… the hardware economics, the shape of an IX, and what ownership does to incentives. He’s also putting the new conclusion into production. Open Pipes, his nonprofit umbrella for community owned exchanges, has a fabric racked and lit in Chicago, and the second half of the talk was a build report.

What made commercial right in 2015

The IX we’d all sketch on a whiteboard today, one fabric stretched across multiple data centers in a metro, was considered a European model back then. Griswold’s recollection is that the US had maybe one or two, and the received wisdom was uniform. It wouldn’t work here, nobody would want it, and you could never pay for the fiber.

Hardware was the other half. White box switching didn’t exist as a practical option, so you couldn’t stand up a fabric for five figures the way you can now, and when the gear costs that much, buying it yourself and skipping the committee is the rational move. Convincing a community to fund an unproven model in a market that had already decided it wouldn’t work… he chose the investor instead, and United IX got built.

Both of those constraints are gone. The fabric got cheap, the multi site model won so thoroughly that we forgot it was ever contested, and enough exchanges exist that nobody needs the value proposition explained anymore.

Ownership changes, incentives follow

Cheap hardware fixed everything except ownership. A commercial exchange works while the operator and the members want the same things, and Griswold’s warning is about what happens later. Ownership changes, and the new owner knows exactly how sticky an IX is. The stickiness is deliberate. An exchange you could walk away from tomorrow wouldn’t be doing its job. It lives in cross connects and BGP sessions, and just as much in the peering relationships, capacity planning, and operational trust that let a network plan on the exchange still existing a year out. A new owner can lean on all of that. Raise pricing because leaving hurts, quietly stop supporting the thing, or have no idea how to run an exchange at all. The first move is Ferengi Rule of Acquisition #1… “Once you have their money, you never give it back.”

He drew one boundary around this. A large pure play exchange like DE-CIX, who he pointed out in the room, could change ownership tomorrow without much risk, since the IX is the entire business and what benefits the members benefits the owner. His concern is the single site commercial exchange, or the IX run as a data center side project, where the exchange’s health is somebody’s line item instead of somebody’s mission.

And rebuilding after an exchange rots is expensive, because starting one is genuinely hard. “Nobody wants to be the first network on an IX. Somebody wants to be the second network on an IX. That’s just called a PNI.” Getting from zero to a real fabric takes convincing, favors, logistics, and then delivery, because networks leave the moment you stop delivering. Lots of exchanges have started, fewer have made it through that.

When United IX launched, AMS-IX came into the market shortly after, spent millions by his telling, and went around town telling everyone to connect to them instead of him. His response was to tell everyone to connect to both, and he still gives that advice. Two ports on the same exchange is fake redundancy. One port on each of two exchanges gets you full diversity across companies, heavy peer overlap, maintenance windows that mostly stop mattering, and a few unique peers on each side.

One 501(c)(3), a hundred exchanges

The Open Pipes design is the part I hadn’t heard anywhere else. Instead of every community exchange incorporating its own nonprofit and filing its own taxes, Open Pipes is built as a global umbrella. One 501(c)(3) that could hold a hundred exchanges, one IRS Form 990 filing for all of them, with the legal and tax overhead paid once. People building exchanges get to build exchanges. He’s watched a handful of IXes get torn apart by doing the nonprofit part wrong, and this is his answer to that failure mode.

Each exchange runs as a self governing committee, the way nonprofit committees already work. Volunteers or nominees with board approval, but the committee picks its own chair and runs its own operation, with the board deliberately kept out of the way except for money decisions and budgets. Open Pipes provides software and legal scaffolding without telling any community where its exchange should be or how to run it. The structure also locks the assets in place. Surplus stays in the fabric as upgrades and redundancy, cost recovery only, and there’s no wind down path while the fabric is being used. The scenario where a new owner inherits an exchange and lets it rot can’t happen, because there’s no ownership to transfer.

There’s a line in Where Wizards Stay Up Late, the book about how the internet actually got built, that compares technological development to building a cathedral… each new person lays a block on the old foundations, and everything ends up tied to everything else. An IX is about as literal a version of that as networking gets, and Open Pipes is the rare case where the cathedral belongs to the people laying the blocks instead of whoever holds the deed.

He was honest about how early this is. The 501(c)(3) was just filed, charitable status included, and while his lawyer likes the odds because the mission is helping communities self-govern, “whether or not we get it, I don’t know.” The member voting and governance structure isn’t finalized either. “We’re waiting to see how the nonprofit works out” was his own hedge, so for now the governance model is a plan without a track record.

The build at 350 E Cermak

The build is going in a few blocks from where Griswold lived for most of his career, and he couldn’t resist the detour. By his telling, the NLNOG infrastructure project was born at Pepper Canister, a bar a few blocks from the venue that’s gone now, during NANOG67 in 2016… he and Job Snijders sketched it out while joking that between them they had access to something like 90% of the internet’s infrastructure servers and probably shouldn’t be sitting in the same room. They announced it that September and had it running in Amsterdam.

The equipment was racked in December, then the build lost several months to fiber problems between the second and fifth floors, resolved only a week before the conference. Four sites so far. Two floors of 350 E Cermak, 427 S LaSalle, where Jay from South Front donated both dark fiber and a switch, and an NYI facility at 800 W. Jorie Blvd out in Oak Brook coming online on more donated dark fiber and space. He was upfront that today’s topology is a hub with plenty of single points of failure, the economical way to start, with quotes already specced to complete the mesh when growth justifies it.

The gear choice is the interesting engineering decision. A 400G Juniper core, with PTX10Ks at the edge specifically because they hold full tables. Routing at the edge of an IX sounds like heresy until he explains the plan. Connect to one exchange, hit a community, and land on a peer at another exchange with a single AS hop in between. He called it almost a meta IX, and it’s mostly built, stalled on the same fiber problem as everything else.

Out of band access is a tidy open source stack I’m tempted to copy for my own lab. A small GL.iNet OpenWRT router at every pop, uplinked over LTE, Wi-Fi, or Ethernet, phoning home over WireGuard, with custom serial dongles fanning out to console ports and an isolated management switch behind it. The whole kit is tiny and USB powered. The rest is his FullCtl automation on one side and NetBox for everything else, with bidirectional sync between them.

Then the services, which is where he picked an old fight on purpose. At United IX he offered transit over VLANs on the exchange, and had people screaming at him that he was competing with his customers. “I never had a single customer complaint.” Customers grabbed transit for out of band and thanked him for it, so the new exchange will offer it again, along with cache servers to generate meaningful traffic from day one. He’d just learned at the Global Peering Forum that SFMIX has been running caches successfully for a while, which he took as confirmation.

The same two peers, eleven years apart

Somebody in the audience asked who he convinced to peer first, eyeballs or content. Both, and it was the same two networks that launched United IX. Akamai and WiscNet, the research and education network for Wisconsin’s universities and schools. Hoyos Consulting and South Front are connected, and OCI just signed on as the first two site peer, taking 100 gigs at both 427 LaSalle and 350 E Cermak.

Griswold reversed himself on ownership, the fabric now costs five figures, the nonprofit paperwork is sitting at the IRS, and the first two networks to show up were the same two that showed up in 2015. The peers apparently trust the person more than the corporate structure, which is either reassuring or exactly the key person risk the nonprofit is supposed to eliminate, and we won’t know which for a few years. If you’ve got a network in Chicago, he’s taking peers now, and he asked the room directly. The buildings involved are ones I drive past, so I’ll be keeping an eye on this one from close by.

Disclosure: I attended Matt Griswold’s session at CHI-NOG 13 in Chicago. CHI-NOG didn’t comp my registration or travel. Nobody at Open Pipes or 20C bought me anything. The opinions here are mine. For more, please read my full disclaimer.