This morning, at around at 9:30 AM Pacific Daylight Time, a construction crew working on a stormwater treatment project in NPI’s hometown of Redmond, Washington accidentally caused a major telecommunications outage for nearly six thousand customers of Frontier Communications (including yours truly) when they dug in the wrong place and severed vital fiber and copper wires.
The outage, which is still ongoing, has resulted in a total, catastrophic failure of Frontier’s Internet, television, and phone services in the Redmond area. Affected households and businesses are not able to connect to the Net, watch any FiOS channels, or place calls (even emergency calls) through Frontier’s infrastructure.
Frontier has technicians onsite and is working as fast as it can to restore service. A Frontier spokeswoman said the construction crew that was at fault caused “considerable damage”. There is no estimate of when service might be back up yet, but Frontier says its own people are working “nonstop”. Earlier this afternoon they were pulling materials needed for the repairs, according to Frontier’s Twitter feed.
The City of Redmond has also acknowledged the outage on its website and on Twitter. City personnel are working with Frontier to figure out what went wrong and how to get service back up for the thousands of people affected.
The outage is more than an inconvenience, particularly for those without cellular phone service, since it prevents them from calling 911 to summon first responders in the event of an emergency. The outage has stretched on for eight hours already, and it might not be resolved for many more hours.
No doubt the city and Frontier will investigate, and try to figure out how this happened. If the construction company is at fault (and it sounds like they were) there should be appropriate consequences.
But regardless of what the investigation uncovers, this and other incidents demonstrate that our critical infrastructure is more vulnerable than we might think.
In this case, one misguided construction crew was able to knock out Internet, phone, and television services for a sizeable fraction of the population of Washington’s nineteenth largest city merely by digging in the wrong place.
It’s sobering to think about what might happen to our critical infrastructure in the event of a natural disaster like an earthquake, or a human-caused calamity, like a terrorist attack. We are not as prepared for such events as we ought to be.
To minimize telecommunication service disruptions, we should look at improving redundancy along with implementing better shielding to protect our core fiber optic cables from being cut, either accidentally or purposefully.
Washington does participate in the Call Before You Dig initiative, as do Montana and Oregon, but that effort could use more awareness.
A large amount of traffic flows over the publicly and privately owned trunk lines in this region and across the United States. Their integrity matters.
At present, I am staying connected to the Internet through NPI’s wireless provider, so it’s business as usual. But if we didn’t have that redundancy, it wouldn’t be.
Redundancy is a very good thing, and can guard against all sorts of problems… from data loss and hard disk failure to connectivity disruptions. We need more of it.
I don’t think better shielding would have helped. When you have a huge backhoe tearing through asphalt and concrete not much is going to stop it from tearing through encasement and such.
They’re supposed to have redundancy: Verizon (from whom Frontier bought the physical plant) said they had a ring of fiber connecting the local hubs to the exchange in Redmond. (Local hub: anonymous small building, eg, west side of Redmond-Woodinville Road opposite NE 116th. Exchange: 148th Av NE in Redmond, just north of Grass Lawn Park.)
The purpose of a ring (in contrast to a linear scheme or a star connection) is to allow two ways for the fiber to connect between the hubs and the exchange. A ring can’t be disconnected by cutting one place: your conversation can go the other way to the hub.
Possibly, this event disconnected the Frontier exchange from other exchanges. Whichever, it shows really insufficient traffic engineering. As Andrew said, lack of proper infrastructure.
In contrast to roads and bridges for which we pay indirectly, Frontier customers pay directly for the infrastructure. They’ve let us down.
I think the report of 6000 customers affected is low. (They did qualify it: 6000 Redmond customers.) I’ve heard the outage spreads from Renton to Woodinville, excepting localities that are served by other providers.
I was affected by this. However, only FIOS TV was affected, FIOS internet still worked. So something was correct with their network planning.
Second, this was city construction which was planned months, perhaps years in advance. They should have known all of the dig points by heart. The City and the contractor should share MOST of the blame for this, not Frontier. Frontier just lays cables where they are permitted to.
Frontier purchased the trunking lines from Verizon’s ground services, and nothing is different. The lines are the same. If this were still branded as Verizon, the same result would have happened.
[…] the past few days, many homes and businesses in the greater Redmond area have been without phone, Internet, and television service after a construction crew working on a stormwater treatment project sliced through fiber and copper […]