Data Center Upstream Issue

Incident Report for Redtail Technology

Postmortem

Incident Summary

On July 30, 2016 around 0522AM ET, QTS OSC started receiving isolated reports from customers concerning internet connectivity issues at our Sacramento, CA Data Center. QTS OSC Engineers started troubleshooting with our upstream provider. At around 946AM ET, additional customers started to call QTS OSC concerning internet connectivity issues. At approximately 1013AM ET, QTS OSC Engineers determined that there was an upstream provider issue. QTS OSC escalated this issue to QTS Network Engineering team and by 1130AM ET they performed a manual fail over to QTS redundant upstream provider to restore services to all the impacted customers.

Root Cause

QTS upstream provider was performing maintenance and made an unannounced change to their routing configuration.

Resolution

QTS Network Engineers rerouted all traffic to QTS redundant upstream provider. The Upstream provider rolled back their change, which restored services for their customers.

Future Measures

Upstream provider maintenance notifications were being sent to an incorrect distribution list. QTS is working with the upstream provider to ensure that they update QTS on all of their scheduled maintenances that could potentially impact QTS.

QTS is performing additional training from lessons learned to QTS engineers to decrease mean time to resolution

Posted Sep 13, 2016 - 09:42 PDT

Resolved

The reported issue with our Data Center has been resolved. All systems are a go. We sincerely apologize for any inconvenience it may have caused.

Posted Jul 30, 2016 - 09:34 PDT

Investigating

Users are experiencing issues accessing all applications within Redtail's product portfolio. Currently one of our Data Centers is experiencing an issue with an upstream provider. We are in communication with them and hope to have a resolution as soon as possible.

Posted Jul 30, 2016 - 07:57 PDT