Incident report - Slow response times - high packet loss

[matt]

At approximately 3:17pm on Monday the 29th of June a line card in the router edge-george-1 in the George Street data centre stopped communicating with the backplane of the router.

At 3:18pm the router attempted to reset the card automatically which failed. At this time automated systems alerted Spectrum Networks staff of a problem with the router edge-george-1

At 3:23pm the router again attempted to reset the line card which again failed at that time the router marked the line card as faulty.

At 3:33 Spectrum staff were alerted to the fact that routing speed across edge-george-1 was being affected. Spectrum staff determined that the failing line card had failed in such a mode as to effect all services on the router.

At 3:45 Spectrum staff initiated a reload of the D-CEF. (Distributed Cisco Express forwarding) across the router, which restored it’s routing speed to normal.

At 3:50 Spectrum Staff established that edge-george-1 was responding correctly and begin migrating 3 services from the failed line card to a new location on the network.

At 4:15 all services were restored.


Spectrum staff determined that a failed line card caused an unexpected condition within the router’s software and as a result forwarding speed was affected. A reset of the Routers database corrected the problem.

A planned event (hazard) will follow to remove the failed line card.

Matt.

[matt]

An Emergency planned Hazard is scheduled for 7AM on Tuesday the 30th of June to remove a failed line card from edge-george-1 .
As the card can be hot-swapped. We do not expect an outage and as such this is a Hazard only.

Matt.