Incident report - Core IPv4 internet.

Here we will post network problems, Planned & Unplanned downtime as well as restoration times and other network issues.

Incident report - Core IPv4 internet.

Postby matt » Thu Nov 05, 2015 2:58 pm

At 02:40 PM on Thursday the 5th of November Network monitoring indicated that there was OSPF flapping on a router within the core IP network that connect AAPT internet and some other smaller AAPT services into the core. The result of this flapping was very small intermittent outages as systems switched between the primary and backups.

Staff diagnosed the problem to what appeared to be a software fault within the OSPF fabric of the core router and the decision was made at take the router out of service to diagnose the problem. At 3:00 The router was taken off line and a 30 second hard outage occurred while redundant systems took over.

With the router out of service the OSPF process was restarted and the flapping condition abated. After 10 minutes of monitoring we returned the router to service without incident.

We will continue to monitor the router but believe the cause was the result of a software bug in the OSPF subsystem of the router.

During the outage some Internet connected and other sites connected via AAPT would have been offline. There was also around a 20 minute period where there may have been some slowdown.

Voice, Private network and peer services were not effected by this outage.
User avatar
matt
Site Admin
 
Posts: 325
Joined: Thu Apr 09, 2009 11:44 am
Location: George Street Sydney

Return to Service Availability & Announcements

Who is online

Users browsing this forum: No registered users and 1 guest

cron