Incident report - Outage George Street to Ultimo

[matt]

On Monday the 5th of January at 15:24:33 the main high speed redundant path between the George Street Data center and Global Switch Ultimo was interrupted.
The loss of this circuit and its redundancy caused a major network outage of voice,data and hosting services.

At 15:32:05 service was restored by switching to a reduced capacity backup fiber between the two locations.

Post incident investigation indicated that a configuration error was made on the core switching system which caused the normally fully redundant system to lock out both fibers within the system completely blocking the path.

The errant configuration has been removed on the primary system and it will be restored to operational service during a planned hazard at 18:00

A post incident meeting will be held to improve cross checking procedures around the core Ethernet trunk network.

[matt]

The planned hazard has been completed and the full primary capacity and redundancy has been restored.