SDNGNet All articles
Wide Area Networking

Audit the Failover Before It Fails: Proving Your Backup Circuit Isn't a Mirror of Your Primary

SDNGNet
Audit the Failover Before It Fails: Proving Your Backup Circuit Isn't a Mirror of Your Primary

Photo: Asurnipal, CC BY-SA 4.0, via Wikimedia Commons

Every enterprise network architect has seen the diagram: a primary internet circuit drawn in one color, a backup circuit in another, and a confident note that failover is automatic. The diagram implies diversity. The invoice confirms two providers. But when a backhoe severs a bundle two blocks from the data center and both circuits drop simultaneously, the diagram turns out to be fiction.

Redundancy that exists only on paper is one of the most expensive illusions in American enterprise networking. Organizations routinely pay for five to seven years of a backup circuit that shares the same physical conduit, the same central office, or even the same last-mile fiber as the link it is supposed to replace. The fix is not buying a third circuit. It is auditing the two you already have — with the same rigor you would apply to a security control.

Why Diverse Providers Often Mean Nothing

The intuition that two carriers equal two paths breaks down the moment you look at how middle-mile infrastructure is actually built in the United States. Regional fiber is consolidated into a small number of conduit systems, and long-haul providers lease lateral capacity from the same handful of construction firms. Two national carriers may hand traffic off at the same central office, traverse the same river crossing, or share a single conduit bank installed a decade ago by a competitor that has since been acquired twice.

The problem is compounded by how circuits get sold. A carrier's sales engineering team answers the diversity question with a standard assurance: paths are "logically diverse." Logically diverse means different IP addresses and different BGP sessions. It says nothing about whether both strands of glass run through the same 4-inch conduit under the same street.

Traceroute Forensics: Reading the Path in the Wild

The first audit technique costs nothing. Run traceroutes from your edge to well-known targets during normal operations, then repeat from the failover circuit after forcing it active. You are not looking for identical hops — providers will never share last-mile routers — but for converging paths.

Watch for:

None of this is conclusive on its own. But convergence patterns give you the questions to ask next.

Conduit Records and the Physical Truth

Logical audits end where the street begins. For circuits that terminate in buildings you control, request conduit and pathway records from your building management or, in owned facilities, from your own as-built documentation. Then ask each carrier for the physical route from your demarc to their nearest POP, down to street level.

Carriers will resist. Route maps are treated as proprietary and, in some cases, security-sensitive. This is precisely why the request matters: a carrier confident in its path diversity can describe it in general terms — direction of egress, major crossings, central office of record. A carrier that cannot answer at all is telling you something.

Where possible, walk the egress. In many commercial buildings across the US, all carriers enter through a single telecom room and a single riser, then exit through one conduit bank to the street. If both of your circuits leave the building through the same manhole, your redundancy ends at the patch panel.

The Provider Diversity Questionnaire

Formalize the audit with a written questionnaire, issued to both providers annually. At minimum, require answers to:

  1. The central office or wire center serving each circuit, and whether they differ.
  2. Physical entry pathway into the customer building, including conduit bank identifiers where assigned.
  3. Direction of egress from the building and the first physical splice point.
  4. Known shared infrastructure with other carriers, including leased laterals and subsea, bridge, or tunnel crossings.
  5. Scheduled maintenance windows and the process for coordinating them so both circuits are never taken down concurrently.

Item five is the one most organizations forget. Two physically diverse circuits can still share a maintenance calendar, and an uncoordinated carrier upgrade is the quiet killer of dual-ISP designs.

Making Diversity a Contractual Obligation

An audit tells you the current state. Contract language protects the future. When negotiating or renewing, push for provisions that:

Test It Like You Mean It

Finally, the only audit that truly matters is the one you run with traffic on the line. Schedule a failover test during a maintenance window: pull the primary circuit, run production or realistic synthetic load across the backup, and measure throughput, session stability, and DNS behavior. A failover circuit that has never carried real traffic is not redundancy. It is a hypothesis.

The organizations that survive metro-scale outages are rarely the ones with the biggest circuit budgets. They are the ones that treated path diversity as a verifiable claim — and demanded the evidence.

All Articles

Related Articles

Whose Riser Is It Anyway? The Hidden Network Risks of Leasing Space in Shared-Infrastructure Buildings

Whose Riser Is It Anyway? The Hidden Network Risks of Leasing Space in Shared-Infrastructure Buildings

The $18,000 Handshake: What Really Happens Between the Street and Your Switch

The $18,000 Handshake: What Really Happens Between the Street and Your Switch

A Million Square Feet, One Controller: Inside the Private 5G Buildout Reshaping American Distribution Centers

A Million Square Feet, One Controller: Inside the Private 5G Buildout Reshaping American Distribution Centers