The Hardware Failure That Took Down The Tokyo Stock Exchange

On October 1, the Tokyo Stock Exchange (TSE) was halted for a complete day due to an issue IT professionals tend to overlook, a piece of hardware.

A crucial data storage and distribution device dubbed “Arrowhead” had malfunctioned, and the automatic backup failed to initiate. Arrowhead is the heartbeat of the TSE, giving and taking commands, routing data and, most importantly, monitoring trades. Without it, the exchange was forced to shut down for a full day, its longest sustained outage since 1999.

What happened?

The Arrowhead system is a hardware and software suite developed by Fujitsu that consists of two shared disk devices. On the day in question, the primary disk “Number 1 shared disk” encountered a memory error. When this occurred, the secondary device “Number 2 shared disk” should have automatically taken over in a failover procedure – essentially a handshake – to seamlessly keep processes functioning as normal. But this was not the case, since a forced manual failover needed to occur and that would have required a restart of the entire system, which was out of the question since orders, trades and data were already beginning to backlog. TSE officials made the decision to halt trading and resume operations the next day.

What could have prevented this?

  • Testing recovery strategies – Testing recovery strategies and documenting results can often aid in the ability to recover in scenarios where time is of the essence. During the testing of recovery strategies, critical recovery gaps can be identified, addressed and resolved. Conducting testing scenarios better prepares an organization and can help greatly reduce the impact of downtime when dealing with a real-life disruption.

Additional Backup and Recovery Considerations

  • Configuring backups – Configuring backup jobs are vital to all types of businesses. Backing up device configurations and data allows organizations to be resilient to disruptive events and occurrences. The ability to recovery quickly and completely is invaluable.
  • Monitoring backups – While the configuration of backups is critical, what good is backup data if it fails to actually back up successfully? Monitoring backup jobs is equally as important, as it ensures data is in a healthy state and ready for recovery.
  • Restoration testing of backups – A crucial part of successful data backups is testing the ability to restore that data. Restoration testing will help ensure data and systems are able to be recovered as intended. Businesses should conduct backup restoration testing frequently to confirm that they are prepared for an unscheduled outage or disaster-type event similar to what was faced by the TSE.

If you have questions about your backup and recovery strategies or your disaster recovery plan, please connect with us. We would welcome a discussion.

You’ve heard our thoughts… We’d like to hear yours

The Schneider Downs Our Thoughts On blog exists to create a dialogue on issues that are important to organizations and individuals. While we enjoy sharing our ideas and insights, we’re especially interested in what you may have to say. If you have a question or a comment about this article – or any article from the Our Thoughts On blog – we hope you’ll share it with us. After all, a dialogue is an exchange of ideas, and we’d like to hear from you. Email us at contactSD@schneiderdowns.com.

Material discussed is meant for informational purposes only, and it is not to be construed as investment, tax, or legal advice. Please note that individual situations can vary. Therefore, this information should be relied upon when coordinated with individual professional advice.

© 2020 Schneider Downs. All rights-reserved. All content on this site is property of Schneider Downs unless otherwise noted and should not be used without written permission.

our thoughts on

Successfully Transitioning Significant Processes During Business Succession
Fraud Risk Assessments – What your Organization Should Consider
Prop 24 passes in California What is it? What does it mean for privacy?
Ransomware Attack Disrupts Popular Sports Gambling Sites
Reshoring and Manufacturers
The Hardware Failure That Took Down The Tokyo Stock Exchange

Register to receive our weekly newsletter with our most recent columns and insights.

Have a question? Ask us!

We’d love to hear from you. Drop us a note, and we’ll respond to you as quickly as possible.

Ask us

contact us

Map of Pittsburgh Office
Pittsburgh

One PPG Place, Suite 1700
Pittsburgh, PA 15222

contactsd@schneiderdowns.com
p:412.261.3644     f:412.261.4876

Map of Columbus Office
Columbus

65 East State Street, Suite 2000
Columbus, OH 43215

contactsd@schneiderdowns.com
p:614.621.4060     f:614.621.4062

Map of Washington Office
Washington, D.C.

1660 International Drive, Suite 600
McLean, VA 22102