Incident report - Intermittent packet loss through george-v

Here we will post network problems, Planned & Unplanned downtime as well as restoration times and other network issues.

Incident report - Intermittent packet loss through george-v

Postby matt » Thu Feb 25, 2016 3:59 pm

Incident report - Intermittent packet loss through george-vrouter-1

At Approximately 7.55 AM on Thursday the 25th of February Spectrum staff were alerted to a high level of packet loss on an MPLS router (george-vrouter-1). This router is responsible for switching packets for some of the customers in the George Street Data center as well as some customers managed MPLS clouds.

At 8.00 AM the fault was escalated to a senior engineer at Spectrum.

At 8.30 AM the fault was isolated to the george-vrouter-1 and the root cause appeared to be a very high packet throughput which exceeded the routers design capacity. Unfortunately due to the very high level of packets the routers management system was either not responsive or very slow to respond slowing progress in isolating the source of the high packet load.

At 9.00 AM Staff disconnected circuits from george-vrouter-1 and systematically re-established links to determine the source of the high level of packets. Most services being restored by 9.30 AM with all services other then the high packet service being restored by Approximately 10.30 AM

Spectrum staff determined that a customer server had been compromised with Malicious software and was participating in a denial of service attack. The customers server was isolated from the network and moved to an isolated lab for testing.

The issue was fully resolved at 10.30AM and Spectrum do not expect a re-occurrence of the issue.

Customers can request their segment be moved to different part of the network if need be.
This outage falls within the Spectrum Standard SLA.
User avatar
matt
Site Admin
 
Posts: 325
Joined: Thu Apr 09, 2009 11:44 am
Location: George Street Sydney

Return to Service Availability & Announcements

Who is online

Users browsing this forum: No registered users and 1 guest

cron