Unplanned outage Friday 10th August 2018

Here we will post network problems, Planned & Unplanned downtime as well as restoration times and other network issues.

Unplanned outage Friday 10th August 2018

Postby matt » Mon Aug 13, 2018 11:36 am

Unplanned outage Friday 10th August 2018

Known issues.
17:03:00 Management systems detected loss of connectivity to a TOR switch in Global switch L4 Mycloud rack.
17:04:14 Management systems detected a restart of the L2GNF node 1 at the George Street Data centre. - Redundant switch still operating however, convergence took a few minutes.
17:15:20 Managment systems detected another restart of the L2GNF node 1. Staff disabled the switch to stop further restarts from affecting users.
17:20:10 l2gnf stabilised restoring most affected users.
17:23:00 Enginers detect a failed SFP+ on a CWDM system between MMR3 GSU and L4 TOR Switch. Which disrupted traffic on adjacent channels on a CWDM mux

Summary
A faulty SFP+ in a CWDM system that connected a compute area of the network located on Level 4 of GSU Ultimo to MMR3 at GSU disrupted connectivity of adjacent channels on that system. Multiple loop conditions were detected across all channels in the mux. Spectrum Stuff remotely powered down the laser in the faulty SFP+ removing loops and restoring services on the mux.

Post-incident actions
Staff have logged a case with the software vendor of the switch in George Street to try and understand why a loop further down the network caused multiple switch restarts.
User avatar
matt
Site Admin
 
Posts: 322
Joined: Thu Apr 09, 2009 11:44 am
Location: George Street Sydney

Return to Service Availability & Announcements

Who is online

Users browsing this forum: No registered users and 1 guest

cron