Unplanend Event Monday Feb 25th

Here we will post network problems, Planned & Unplanned downtime as well as restoration times and other network issues.

Unplanend Event Monday Feb 25th

Postby matt » Tue Feb 26, 2019 11:05 am


Known issues.

14:30:03 An engineer working on the network modernisation project. Inadvertently put a new path into service between GSU and the George Street DC
14:45:22 Remote out of band communication was established to the affected switches and the normal path was restored. Restoring service.

Summary
An engineer working on the network modernisation project, inadvertently allowed traffic to flow over a new circuit that was in the process of being commissioned to replace the old George Steet Global Switch Pipe Room link. Temporary allowing a loop to block both circuits between George Street and the Global switch Pipe Room.

This disrupted connectivity with some customers on the legacy part of the network. Engineers then accessed configuration via out of band channels and reset the link restoring services. However, some bridges inside customer network equipment failed to correctly reset and a small amount of manually rebooting some customer CPE was required to restore service.

Connectivity was also lost to some Perth based customers as a Perth edge router stopped routing traffic on some vlan's. The problem required a reboot of the Perth edge router to restore traffic at approximately 15:50 Eastern time.


Post-incident actions

Staff have re-scheduled more hazardous work outside Business hours.

Network Modernisation project

The Network Modernisation project has been ongoing for approximately 8 months and aims to move all users from older network components to the new Software-defined Layer 2 Global Network Fabric (L2-GNF) other key aims is to move all customer edge services to 10+ Gigabit-capable equipment and to depreciate the George Street Data Center due to the inability to receive diesel fuel since the inception of the Sydney Light Rail project in George Steet.
User avatar
matt
Site Admin
 
Posts: 325
Joined: Thu Apr 09, 2009 11:44 am
Location: George Street Sydney

Return to Service Availability & Announcements

Who is online

Users browsing this forum: No registered users and 0 guests

cron