Audit the Failover Before It Fails: Proving Your Backup Circuit Isn't a Mirror of Your Primary
Photo: Asurnipal, CC BY-SA 4.0, via Wikimedia Commons
Every enterprise network architect has seen the diagram: a primary internet circuit drawn in one color, a backup circuit in another, and a confident note that failover is automatic. The diagram implies diversity. The invoice confirms two providers. But when a backhoe severs a bundle two blocks from the data center and both circuits drop simultaneously, the diagram turns out to be fiction.
Redundancy that exists only on paper is one of the most expensive illusions in American enterprise networking. Organizations routinely pay for five to seven years of a backup circuit that shares the same physical conduit, the same central office, or even the same last-mile fiber as the link it is supposed to replace. The fix is not buying a third circuit. It is auditing the two you already have — with the same rigor you would apply to a security control.
Why Diverse Providers Often Mean Nothing
The intuition that two carriers equal two paths breaks down the moment you look at how middle-mile infrastructure is actually built in the United States. Regional fiber is consolidated into a small number of conduit systems, and long-haul providers lease lateral capacity from the same handful of construction firms. Two national carriers may hand traffic off at the same central office, traverse the same river crossing, or share a single conduit bank installed a decade ago by a competitor that has since been acquired twice.
The problem is compounded by how circuits get sold. A carrier's sales engineering team answers the diversity question with a standard assurance: paths are "logically diverse." Logically diverse means different IP addresses and different BGP sessions. It says nothing about whether both strands of glass run through the same 4-inch conduit under the same street.
Traceroute Forensics: Reading the Path in the Wild
The first audit technique costs nothing. Run traceroutes from your edge to well-known targets during normal operations, then repeat from the failover circuit after forcing it active. You are not looking for identical hops — providers will never share last-mile routers — but for converging paths.
Watch for:
- Shared intermediate hops. If both traceroutes pass through the same metro aggregation point, the same exchange point, or the same city-pair handoff, you have a logical choke point even if the fiber itself is separate.
- Geographic bottlenecks. Hop hostnames frequently embed POP codes and city names. Two circuits that both funnel through, say, the same regional interconnection facility in Atlanta or Dallas will fail together in a metro-scale event.
- Latency signatures. If the failover circuit's round-trip time converges toward the primary's within two or three hops, both paths likely share transport infrastructure.
None of this is conclusive on its own. But convergence patterns give you the questions to ask next.
Conduit Records and the Physical Truth
Logical audits end where the street begins. For circuits that terminate in buildings you control, request conduit and pathway records from your building management or, in owned facilities, from your own as-built documentation. Then ask each carrier for the physical route from your demarc to their nearest POP, down to street level.
Carriers will resist. Route maps are treated as proprietary and, in some cases, security-sensitive. This is precisely why the request matters: a carrier confident in its path diversity can describe it in general terms — direction of egress, major crossings, central office of record. A carrier that cannot answer at all is telling you something.
Where possible, walk the egress. In many commercial buildings across the US, all carriers enter through a single telecom room and a single riser, then exit through one conduit bank to the street. If both of your circuits leave the building through the same manhole, your redundancy ends at the patch panel.
The Provider Diversity Questionnaire
Formalize the audit with a written questionnaire, issued to both providers annually. At minimum, require answers to:
- The central office or wire center serving each circuit, and whether they differ.
- Physical entry pathway into the customer building, including conduit bank identifiers where assigned.
- Direction of egress from the building and the first physical splice point.
- Known shared infrastructure with other carriers, including leased laterals and subsea, bridge, or tunnel crossings.
- Scheduled maintenance windows and the process for coordinating them so both circuits are never taken down concurrently.
Item five is the one most organizations forget. Two physically diverse circuits can still share a maintenance calendar, and an uncoordinated carrier upgrade is the quiet killer of dual-ISP designs.
Making Diversity a Contractual Obligation
An audit tells you the current state. Contract language protects the future. When negotiating or renewing, push for provisions that:
- Define path diversity explicitly, requiring separate central offices, separate building entry points, and no shared conduit segments between the demarc and the first metro POP.
- Require notification of route changes. Carriers re-route traffic routinely for construction and capacity. You need written notice before your "diverse" circuit migrates onto your primary's conduit.
- Attach remedies. A service-level agreement that pays out when both circuits fail in a common-mode event converts diversity from a sales promise into an enforceable obligation.
- Reaffirm diversity annually, with the questionnaire results appended as exhibits rather than exchanged as email.
Test It Like You Mean It
Finally, the only audit that truly matters is the one you run with traffic on the line. Schedule a failover test during a maintenance window: pull the primary circuit, run production or realistic synthetic load across the backup, and measure throughput, session stability, and DNS behavior. A failover circuit that has never carried real traffic is not redundancy. It is a hypothesis.
The organizations that survive metro-scale outages are rarely the ones with the biggest circuit budgets. They are the ones that treated path diversity as a verifiable claim — and demanded the evidence.