Green Lights, Dead Sessions: The Commissioning Gap That Sinks Hospital Network Go-Lives
Photo: hospital server room network switches technician, via static.wixstatic.com
At 6:00 a.m. on a Monday in March, the physicians' tower at a mid-sized community hospital in the Midwest opened for its first patient admissions. The network team had spent eighteen months on the build: new core, new access layer, redundant uplinks, a clean design signed off by three vendors. Every switch port had passed acceptance testing. Every VLAN responded to ICMP. The dashboard was a wall of green.
By 7:30 a.m., clinicians couldn't open charts. The electronic health record (EHR) sessions were timing out in waves — fine for ten seconds, then frozen for twenty. IT had a network that passed every test it was given and failed the only test that mattered.
This is the commissioning gap, and it shows up everywhere from hospitals to distribution centers to corporate campuses: the space between a network that is reachable and a network that is usable under production load.
What Acceptance Testing Actually Measures
Traditional acceptance testing is built around reachability and redundancy. Can every device get a DHCP lease? Do the failover links converge? Does the core pass a routing audit? These are necessary checks, and skipping them is malpractice. But they share a fundamental limitation: they validate the network at rest.
A hospital wing at 6:00 a.m. is not at rest. Within an hour, hundreds of workstations, workstations-on-wheels, infusion pumps, badge readers, imaging modalities, and guest devices authenticate simultaneously. EHR sessions open persistent connections to backend servers. Roaming laptops hop between access points mid-shift. None of that behavior appears in a ping sweep.
The gap isn't a design failure in most cases. It's a testing failure — the network was never asked to do the job before opening day.
The Three Tests That Should Happen Before Cutover
Load simulation with production-like traffic. Modern traffic generators can replay captured flow data from an existing facility, scaled to the new wing's expected census. For the hospital scenario, that means replaying EHR session patterns, imaging transfers (a single CT study can move hundreds of megabytes), and VoIP streams concurrently. The goal is not to hit a bandwidth number — it's to expose queueing behavior, buffer contention, and session-establishment latency under concurrency. A 10-gigabit uplink can still crawl if microburst traffic from fifty simultaneous chart opens saturates a shallow egress queue.
Configuration and policy audits against intent. In software-defined environments, the control plane's declared intent and the data plane's actual behavior can diverge quietly. Automated config auditing — comparing rendered device configurations against the high-level policy model — catches the QoS policy that never propagated to the third floor, the ACL that silently blocks the EHR application's secondary port, the MTU mismatch on a DCI link that fragments large transfers. These are the defects that ping tests cannot see and that users discover painfully.
Flow analysis during a staged occupancy. The most effective commissioning programs don't cut over an entire wing at once. Move one department, then watch NetFlow or IPFIX telemetry for a week. Real user traffic reveals asymmetries, hairpinning through a firewall, and application traffic landing on paths the design never anticipated. Every anomaly found during staged occupancy is an incident that never reaches a clinician.
Why the Gap Persists
If the fixes are well understood, why do go-lives keep failing this way? Schedule pressure is the honest answer. Commissioning sits at the end of a construction timeline, and the days reserved for load testing are the easiest ones to surrender when a build runs late. Testing at production scale also requires tooling and traffic models that many organizations haven't invested in — a one-week rental of a traffic generator feels expensive until you price a stalled opening day in cancelled procedures.
There's also an organizational blind spot: acceptance testing is often scoped by the installer, whose contractual obligation ends at reachability and standards compliance. Performance under load belongs to no one's line item.
Making Ramp Testing a Deliverable
The organizations that avoid this pattern treat commissioning as a formal phase with named deliverables: a load test report at projected peak concurrency, a config-versus-intent audit with zero unexplained deviations, and a flow baseline captured during staged occupancy. Network automation platforms make this repeatable — the same intent model that drives configuration can drive validation, turning pre-cutover testing from a manual scramble into a pipeline.
The tower in our Midwest hospital recovered by mid-morning once engineers found a QoS misclassification starving EHR traffic during bursts. The fix took forty minutes. The reputation cost lasted considerably longer.
Green lights tell you the network is alive. Only load tells you whether it can work for a living — and that question deserves an answer before the first badge swipe, not after.
