False Immunity: Long Prefixes That Bypass ROV (CHI-NOG 13)

Brad Gorman’s ARIN workshop covered the version of routing security CHI-NOG 13 was teaching as the floor. ROAs you can trust, ASPA shipping, BGPSec quietly retired. The takeaway that morning was that origin validation works, and the 60% global ROA coverage stat means real attacks are being stopped. Real progress.

Then I sat down for Bryton Herdes’ talk the next day, and that’s the one I couldn’t stop thinking about on the drive home.

The title was “False immunity: long prefixes that bypass ROV.” Bryton is a principal network engineer at Cloudflare. The talk is, basically, “here is a way the 60% number is lying to you, and here is the quad1 hijack we ate because of it.” If the ARIN workshop was the floor, this was the talk where somebody pointed out the sinkhole in the corner.

While no one, including Herdes, would say ROV isn’t necessary, his story proves that it also isn’t enough.

The assumption that breaks

The thing every routing-security pitch leans on, including the one from Brad’s ARIN workshop, is that big networks are running ROV. Not just publishing ROAs, enforcing them. The pitch goes: 60% of prefixes have valid ROAs, the tier ones run origin validation, so if you sign your ROA correctly your prefix should be protected from the obvious origin hijacks.

Bryton’s framing on this is something like, you put a really nice lock on a really nice gate. The gate looks secure. But the fence on either side of the gate stops two feet short of the ground. Anything short enough to crawl walks right around the lock and you never notice, because the gate is still locked.

In ROV terms, the gate is /24 and shorter. The crawl-under is /25 through /32.

How a /32 hijack ate quad1

Cloudflare announces 1.1.1.0/24 from AS13335. The ROA is signed for AS13335 with max length /24. Nothing more-specific than /24 is supposed to exist for that prefix anywhere on the internet, period. If something does, it’s wrong by definition.

So when alerts went off saying AS3356 (Lumen in the US, Colt in Europe, Cirion in LATAM, all the same ASN) could no longer reach 1.1.1.1, but could reach 1.1.1.2 and 1.1.1.3 just fine, the shape of the problem already told you what it was. Somebody was announcing a more-specific to that single address. The looking glass confirmed it. There was an active 1.1.1.1/32 in AS3356 with an AS path of 8452 36992. Neither of those is Cloudflare.

AS36992 is Etisalat Egypt. They were the originator. AS8452 is Telecom Egypt, their upstream, who accepted the route. AS3356 then accepted it from Telecom Egypt and propagated it to its customer cone, which is enormous. Every Lumen, Colt, and Cirion customer doing recursive DNS against 1.1.1.1 was effectively having their quad1 traffic black-holed through Egypt.

From a ROV-strict-mode perspective, this route is invalid in every way you can be invalid. Wrong origin AS. Longer than the max length on the ROA. Either check, in isolation, should have caught it. Both checks together should have made it impossible.

So the obvious first question Cloudflare asked AS3356 was, did you forget to install a filter on the Telecom Egypt port? Did you turn ROV off entirely? Neither. The answer Lumen gave was worse than either, because it was deliberate.

They run ROV. They run it on everything up to and including /24. For anything longer than /24 they fall back to IRR-only filtering, no RPKI check. And 1.1.1.1/32, being a /32, is longer than /24, so it never went through the RPKI validator at all. It went through the IRR policy. Telecom Egypt’s AS-SET happens to include Cloudflare’s AS13335. The IRR policy said yes, so the route got installed and quad1 got hijacked.

Why anyone would do this

The “why” is the part that makes this hard. Lumen wasn’t being lazy. They were supporting a real customer ask.

There’s a legitimate traffic-engineering pattern where a customer with multiple uplinks announces a more-specific /32 (or /128, or whatever) to steer that specific destination over a specific port. The covering /24 still floats normally. The more-specific is just a steering signal that says, “for this particular IP, come in here.” On v4 in particular, where address space is tight and individual addresses sometimes carry disproportionate traffic, this isn’t theoretical. Customers ask for it. Some of them pay for it.

The shape of it looks pretty mundane in isolation:

That config is a Junos policy snippet. The first term does the RPKI invalid check, and the action looks correct. Reject and mark invalid. The problem is hiding in the match. route-filter 0.0.0.0/0 upto /24. Anything longer than /24 doesn’t match this term at all, falls through, and lands in the IRR-only acceptance term further down. That’s the bypass. It’s one line of policy. It supports the TE feature their customers wanted. It also accepts an Egyptian ISP’s /32 hijack of quad1 without complaint.

Here’s the part that should bother everyone reading along. The mechanism isn’t exotic, and it’s not a bug. It’s a policy choice somebody made on purpose to keep a paying TE feature working. Once you see how easy that policy looks, you start wondering how many other networks have something that looks just like it.

How many other networks have something that looks just like it

Bryton wondered the same thing. So he tested it.

The methodology is clean. Sign a ROA authorizing one AS, max length /24. Then originate the /32 from a different ASN. Make sure the IRR data still resolves the prefix to a valid AS-SET, so anyone doing IRR-only filtering would accept it. Then advertise that intentionally-invalid /32 to one tier one peer at a time and check whether the route gets installed and forwarded. Repeat for IPv6 with a /128.

Strict ROV networks should drop the route on sight. Bypass networks would accept it.

Results, for the tier ones he could test from a Cloudflare-shaped peering position:

Of the fifteen tier ones tested, two failed the test. AS3356 (Lumen, Colt, Cirion) accepted the long-prefix invalid for IPv4 but not IPv6, which lines up neatly with the theory that the TE-bypass feature was built for v4 customer demand and never extended to v6. AS6461 (Zayo) accepted both. Everyone else enforced ROV the whole way down to /32 and /128, which is the result you want.

Two out of fifteen sounds small until you remember those two account for an enormous share of internet egress, and the impact of a hijack accepted at that tier propagates to every downstream customer cone hanging off them. The traffic-engineering value at the customer relationship level is real but bounded. The blast radius when this gets abused is the customer cone of the entire ASN.

So what’s the answer

This is the part where I have to admit there isn’t a clean one. Bryton put four options on the slide and each one has a real cost.

Option one: don’t offer this TE method. Just stop accepting prefixes longer than /24. The bypass goes away because the feature it was built for goes away. Operationally this is clean and security-strong. Commercially it’s a problem, because the customers using long-prefix TE are paying for it and they don’t want to hear “no.” For a tier one whose value proposition includes flexibility for enterprise and content-heavy customers, this is a harder sell than it sounds.

Option two: require customers to sign ROAs with maxLength all the way down to /32. The customer keeps the TE feature. The provider’s ROV check accepts the more-specific because the ROA explicitly authorizes it. This one actually scales. Yes, it goes against RFC 9319’s “minimal ROAs” guidance, and yes, a permissive maxLength does open you up more to forged-origin sub-prefix hijacks. But the customer was already announcing a /32 in production. The cat is out of the bag. The signed ROA at least closes the door on the unrelated-hijacker case, which is the one that ate quad1. If you genuinely need long-prefix TE, this is the price.

Option three: a brand-new RPKI object specifically for blackhole/discard semantics. Bryton calls this DOA, Discard Origin Authorization. The draft is in front of SIDROps now under draft-spaghetti-sidrops-rpki-doa. The idea is fine. The deployment story is grim.

A new RPKI object means the RIRs have to act as trust anchors for it, the validators have to support it, RTR has to ship the new object type, the router vendors have to consume it, and operators have to write policy around it. That’s the same multi-year adoption tail we just sat through for ROAs, and that’s if the standard lands cleanly. It’s the right architectural answer. It’s not a thing you can deploy this quarter.

Option four: “loose” ROV. Validate origin only, ignore maxLength, but only for blackhole-community-tagged routes. BIRD already implements this with an ignore max length switch on a separate validation session. The DE-CIX route servers have it deployed. It works. It’s also, as Bryton put it, kind of queasy, because the slippery slope is real. The whole reason maxLength enforcement matters is to prevent the forged-origin sub-prefix attack. If your operator turns the switch on for blackholes today and then someone copies that pattern to the wrong policy term tomorrow, you’ve thrown out half of RFC 9319 for everybody. The mitigation works only if the people implementing it are disciplined about scoping it to BLACKHOLE-community routes and nothing else.

If you’re tracking my read on these: option two for TE, option four for blackholes if you can scope it tightly, option three someday when it ships. Option one is honest but it doesn’t survive a sales conversation.

RTBH is the same problem in a different costume

Worth calling out, because the second half of the talk basically argues that remote-triggered blackhole routes are this same hole in a different shape.

RTBH routes are how a customer under a DDoS attack tells their provider “drop traffic to this IP for me.” The customer announces a /32 (or /128) of their own space with a BLACKHOLE community attached, the provider null-routes it on receipt, and the rest of the customer’s network keeps working. The whole point is that the announcement is super-specific, because nobody wants to blackhole more of their addresses than they have to.

Almost nobody runs RPKI validation on RTBH routes. Most providers IRR-filter them and call it done. Bryton showed a Cloudflare prefix being remote-triggered blackholed by a hijacker (AS55720, not Cloudflare) on a tier one’s network, with traffic actually getting discarded. The signed /19 from AS13335 is sitting right there in the repository. It just doesn’t get consulted, because RTBH is one of the few places where the industry agreed long ago to wave RPKI off and trust IRR.

DOA exists as a draft for this reason, and it’s why loose ROV is even on the table. The legitimate-need-creates-attack-surface tension is the same tension as the TE case. Customers genuinely need the feature, the way it got built is incompatible with strict validation, and something has to give.

The thing that actually matters

The number to remember from the ARIN piece was 60%. The number to remember from Bryton’s talk is two-of-fifteen. Both are true at the same time. Coverage is up and to the right. Enforcement is not uniform, and the gaps are exactly where customers are paying providers to leave them open.

What kept me up after the conference wasn’t the technical detail. It was that the routing-security industry talks about ROV as if it’s a binary. You’re either running it or you aren’t. The 60% stat gets cited as if it means 60% of traffic is protected. It doesn’t. It means 60% of prefixes are covered by ROAs. The enforcement layer underneath that is a per-operator policy choice, and at least two tier ones who show up in the “we run ROV” column have a documented carve-out big enough to drive an Egyptian /32 through.

If you’re a Cloudflare customer, you didn’t get a notification when this hijack happened. You probably noticed quad1 looked slow for a bit if you noticed at all. We’re encouraging the whole industry to declare victory on origin hijacks. The data says we shouldn’t yet.

A few things I’d actually do if you operate a network downstream of someone large:

• If your traffic-engineering setup uses sub-/24 announcements, sign your ROAs all the way down to whatever maxLength you actually announce. RFC 9319’s minimal-ROA advice was written for the common case. Long-prefix TE isn’t the common case. Cover what you announce.
• If you’re a transit operator running ROV up to /24 and IRR-only beyond that, you already have the hole AS3356 had. RPKI never sees your long prefixes, so the IRR filter is the only thing deciding whether a customer can announce a more-specific for someone else’s space. Audit that policy on every port allowed to carry long prefixes. An AS-SET that expands to include Cloudflare, Google, or any major content network the customer doesn’t serve shows you how much of someone else’s space is exposed through you.
• If you’re using RTBH from a provider, ask them whether their RTBH path validates anything other than IRR. The answer is almost certainly no, but the conversation is worth having, because the volume of operators thinking about this is currently very small and that’s how slow change happens.

Bryton ended the talk with a soft pitch for the DOA draft (Discard Origin Authorizations, draft-spaghetti-sidrops-rpki-doa), which Job Snijders co-authors, a signed RPKI object for authorizing RTBH-style discard routes. It’s worth reading if you’re in the policy-implementation weeds, but it doesn’t solve the problem this quarter. The thing that solves the problem this quarter is the conversation between transit operator and customer, where the customer signs a long-maxLength ROA and the operator stops carving out the IRR-only escape hatch.

That conversation happens one peering meeting at a time. Which is why it’s worth writing about. Coverage doesn’t equal enforcement. Enforcement doesn’t equal correctness. The hole that ate quad1 is still there in at least one network and probably more we haven’t found yet.

Disclosure: I attended Bryton Herdes’ talk at CHI-NOG 13 in Chicago. CHI-NOG didn’t comp my registration or travel. Cloudflare didn’t buy me anything. The slide images are from Bryton’s published deck. The opinions and editorial framing here are mine. For more, please read my full disclaimer.