Did Poor Change Management Contribute to the AT&T Wireless and McDonald’s Outages?

In the past few months, both AT&T Wireless and McDonald’s experienced system outages that affected customer service.

On February 22, AT&T encountered widescale network outages throughout its U.S. service areas.

Other providers that use AT&T cell towers were also affected, including Cricket Wireless, Consumer Cellular and Straight Talk Wireless. Customers reported issues making calls and sending text messages.

Then, on March 15, McDonald’s reported global outages to its point-of-sale system, with  employees noting an inability to accept orders, open cash registers or process payments. The outage caused many restaurants to close.

Both companies stated that the outages were due to a software update and configuration change to their respective systems, and not caused by bad external actors in a cyberattack, despite media reports. In other words, the software update encountered complications. McDonald’s attributed the change to a third-party service provider.

If the outages were caused by poor change management practices, both AT&T and McDonald’s should evaluate their procedures. Even if they have strong processes in place, it’s important to recognize that change management needs to be followed consistently and throughout all types of changes. Even seemingly straightforward or small changes could have significant consequences.

Best Practices to Limit Software Change Issues

Effective change management is crucial to minimizing the significant potential negative impact changes could have when pushed to production. The following list identifies some key recommendations for any organization looking to improve its change management process:

• An Emergency Change Process - This should be in place for those changes that require fast or immediate implementation into production. This allows organizations to expedite the approval and implementation of urgent changes while still following the organizational change management process, ensuring that changes are documented and tracked throughout.
• Building in Adequate Network Redundancy, which ensures that if a change does go wrong and affects the overall network, network redundancy will automatically take over, minimizing downtime and maintaining service availability.
• Change Testing - All changes should undergo review within a separate test environment prior to migration into production. Types of tests include quality assurance, user acceptance, regression, security (code reviews, dynamic/static scans, fuzz testing) and configuration, which ensure the change meets quality standards and performs as intended. It helps the team identify any bugs, errors or unexpected outcomes before they impact the users or system, as well as any security risks. Testing should also follow segregation of duties and be performed by an independent resource.
• Deploying Changes on a Predefined Schedule - These allows organizations to implement changes during low-traffic periods, which can minimize service disruptions for users or customers. It can also reduce the risk of errors or failures by allowing for ample planning and testing. In addition, companies should keep up with critical updates and patches based on the vendor’s recommended schedule.
• Implementing a Change Advisory Board - A board consisting of various leaders throughout the organization who can assist in making informed decisions about proposed changes by considering the risks and impact of the change.
• Implementing a Change Management Policy - Policies provide guidance during software changes, outline the requirements of the change process and ensure adherence to all necessary steps.
• Implementing a Ticket System - Systems like Jira, ServiceNow, Cherwell, etc., can provide a centralized location to track all change requests from initiation to resolution while maintaining documentation of the change, including description, testing, approvals and prioritization.
• Keeping Servers Updated - Ensure servers have not reached their end of life. Unsupported servers pose a significant security risk due to their no longer receiving important security updates/patches. Reliability and performance are also compromised with out-of-date/unsupported software.
• Limiting Administrator Access/Assigning Unique and Separate Accounts - This allows organizations to reduce the risk of unauthorized changes. Unique accounts enable easier tracking and auditing of changes that occur.
• Maintaining Segregration of Duties - This is a crucial control in the change management process, meaning that the user(s) who developed the change(s) should not have access to migrate those changes to the production environment. A change log should be reviewed monthly if segregation of duties conflicts exist.
• Monitoring Third-party/Offshore Locations - If third parties or service providers are involved with implementing changes or providing support for systems, it’s important to monitor their performance and compliance to the change management process to ensure risks are mitigating with outsourcing and that changes are implemented effectively and securely.
• Obtaining Final Approvals - Changes should have final approval by management prior to being deployed into a production environment. This ensures that the change has undergone review and validation, thus reducing the risk of a company deploying untested or harmful changes into production.
• Obtaining Initial Approvals - All proposed changes should have proper management approval to ensure they align with strategic objectives and operational needs.
• Set and Monitor Change Management KPIs - This ensures that organizations will assess the effectiveness and efficiency of the change management process, help identify improvements areas and optimization opportunities.
• Strong Password Controls for Administrator Access - Utilize a password vault for administrator generic accounts to help manage passwords and reduce the risk of weaker password control for administrators.

About Schneider Downs Risk Advisory

Our team of experienced risk advisory professionals focus on collaborating with your organization to identify and effectively mitigate risks. Our goal is not only to understand the risks related to potential loss to the organization, but to drive solutions that add value to your organization and advise on opportunities to ensure minimal disruption to your business.

Explore our full Risk Advisory Service offerings or contact the team at [email protected].