Incident Report - Unplanned outage - core-global-1

Here we will post network problems, Planned & Unplanned downtime as well as restoration times and other network issues.

Incident Report - Unplanned outage - core-global-1

Postby matt » Fri Mar 10, 2017 9:11 am

Incident Report - disruption to some internet connectivity. core-global-1

On Thursday the 9th of March 2017 at approximately 16:20 our monitoring system detected poor performance of a key internet edge router core-global-1.
Investigations showed that a large broadcast storm was effecting the performance of the control plane of the router which was causing BGP flapping to several peers. Resulting in loss of internet access to some customers.

At 16:30 the 40G/bit trunk to the peer where the broadcast storm originated was disabled and normal processing ability of core-global-1 was restored.
At 16:40 all sessions on core-global-1 had converged and were operating normally.

Further investigations confirmed that the peer network were aware of a large scale broadcast storm during the time of the outage. Operators of the peer network have put in place countermeasures to insure the storm does not reoccur. Spectrum staff will schedule a hazard period shortly to enable counter measures which will insure an effected peer network does not impact our network

Backup systems operated normally within the outage however as the router had not failed they were not called upon to take over routing load.

We do not expect a recurrence of this incident.
User avatar
matt
Site Admin
 
Posts: 325
Joined: Thu Apr 09, 2009 11:44 am
Location: George Street Sydney

Return to Service Availability & Announcements

Who is online

Users browsing this forum: No registered users and 1 guest

cron