Home Webmail Support Shop Contact Us

RCA for Feb 12th San Antonio Network Outage

Root Cause Analysis (RCA) - Feb 12th - San Antonio Network Outage

Axcess Internet Services, Inc.® utilizes space in the PEER 1 Hosting/Serverbeach Datacenter in San Antonio, Texas.  PEER 1 Hosting/Serverbeach currently uses two unique transport providers for their circuits running between San Antonio and Dallas. The network outage that we experienced on February 12th 2013, uncovered the fact that both transport providers use a common third party, long-haul provider for the underlying fiber infrastructure.

The underlying provider was performing a planned fiber maintenance in the Waco, Texas area that ran from 00:00CT to 06:00CT the morning of February 12th.  During that maintenance, both backbone circuits that run between San Antonio and Dallas were taken offline; one for 3 hours and 30 minutes and the other for 5 hours and 13 minutes.  While both circuits were down simultaneously for 3 hours and 30 minutes, clients hosted on our servers in the San Antonio Datacenter would have seen an outage.

These transport circuits were initially ordered as diverse paths, which means that no single maintenance should be able to impact both at once. However, it was discovered during this incident that one of the providers was recently sold to another company and physical circuit path routing may have been changed without our knowledge.

Initial diagnosis of this event took longer than normal as Axcess Internet®, PEER 1 Hosting/Serverbeach was not notified of this planned maintenance. PEER 1 Hosting/Serverbeach was also performing non-impacting port reconfiguration work in the same area of the network as the fiber maintenance, so this complicated the initial diagnostics. In light of this issue PEER 1 Hosting/Serverbeach are taking the following steps:

  1. They have already been in the process of replacing two fiber providers in the San Antonio to Dallas corridor for several months now. A new deal has been signed already with a separate third provider, and has been in the deployment phase.  The end result will be a redundant circuit ring through one provider and separate redundant circuit through another.  PEER 1 Hosting/Serverbeach is taking the appropriate precautions to ensure that these new circuits will run on completely separate paths.
  2. They are working with both current providers to obtain more information about how these previously diverse circuits were diverted to be in shared paths and a full history of circuit re-locations.
  3. How the transition of the account was handled when one provider changed ownership and why they were not properly informed of the planned maintenance.  They are also following up with the second provider regarding their lack of notification.
  4. They are obtaining confirmation from both our current providers that there will not be any work on our circuits for the next 8 weeks.
  5. They are also performing a PEER 1 network wide audit of the current fiber paths being used throughout the transport links to ensure full network redundancy is available in all locations.

Axcess Internet® will also be auditing our escalation lists and contact detail for all transport/transit vendors. PEER 1 Hosting/Serverbeach has apologized to us and we are in turn apologizing to you.

Axcess Internet® greatly appreciates your patience and continued support and we want to assure you that we are taking steps to ensure that this type of issue will not occur again.

Sincerely,
J. A. Restucci
CEO, Network Operations
Axcess Internet Services, Inc.®


Posted Feb 19 2013, 03:23 PM by Axcess Corporation™
Copyright © 2016 Axcess Corporation™ | Phone: 1.866.4AXCESS | Fax: 1.509.837.4011
Mobile Site | Copyright Notice | Privacy Statement