Incident report - Hight Packet Loss

Here we will post network problems, Planned & Unplanned downtime as well as restoration times and other network issues.

Incident report - Hight Packet Loss

Postby matt » Mon Dec 01, 2014 2:32 pm

At 00:00 on Saturday the 29th of November 2014. High packet loss was reported within some parts of the Spectrum Co-Location Network.

On investigation Spectrum Networks staff isolated the problem to be a unicast flood to all ports on a switching segment that was leaking backup traffic into an area of the network that did not have capacity for that volume of traffic. As a result that flooded segment exhibited a some packet loss and increased latency.

At approximately 02:00 Spectrum Staff completed a roll back of all provisioning task's within the layer 2 network from Friday the 28th's work logs. At that time the unicast flood was no longer detectable on the network.


At 00:00 on Sunday the 30th of November 2014. A repeat of the high packet loss event was detected. Spectrum staff isolated the problem to be a fault in a edge switches forwarding information database. After applying some fault finding techniques it was decided to halt the backup process within the cluster network. Which abated the flood.

As a result of the unexplained error on the switches FIB. Spectrum Staff removed the incident switch from operation and moved the mycloud backup network to a newer part of the network.

A further test of the backup process showed no further unicast flood. We will close the case if no further flooding activity is detected on the network at 00:00 Tuesday 2nd.

Matt
User avatar
matt
Site Admin
 
Posts: 325
Joined: Thu Apr 09, 2009 11:44 am
Location: George Street Sydney

Return to Service Availability & Announcements

Who is online

Users browsing this forum: No registered users and 1 guest

cron