This success story overviews how a major American bank used Cutover automated runbooks to successfully execute an IT disaster recovery failover procedure to meet regulatory requirements.
The challenge: Testing data center failover
A major American bank wanted to test that they could successfully execute a major data center failover event and get all of their 863 critical applications back online within a set time. They also needed to be able to credibly attest to their ability to do this to the regulator.
Better preparation for data center failover testing
Around 2,500 people at the bank were involved in the process, with 1,863 of them executing tasks in Cutover automated runbooks. Cutover’s Client Success Team conducted 25 hours of training at the bank over two weeks to ensure preparation for the data center failover testing event.
Strategic preparation for data center isolation
Every participant could be contacted via SMS through Cutover. This meant that across a 16-hour data center testing event, they didn’t have to spend the entire time on an open bridge waiting to be told to start their task as they had done previously. Instead, they could step away and do other things or rest until they received the notification that they were needed.
Mobilizing a cross-functional team for seamless execution
Using Cutover also allowed the bank to prepare for a key person not being available, as each task had three contacts assigned to it and if the first person was unable to respond when needed, the secondary and tertiary contacts were in the system ready to go - so there was no need to waste time searching a phone directory or spreadsheet.
Creating the data center runbook
The bank then built the runbooks for the data center failover which involved 863 applications and 13 different lines of business. Each line of business had its own dashboard so they could easily filter for the relevant tasks and information.
Achieving a seamless data center failover
Due to the scale and complexity of the data center failover event, it was also very useful that Cutover enabled users to easily filter the runbook to quickly find their relevant tasks, saving time and ensuring nothing was missed.
Flawless execution and results
In total, the event took 16 hours and 22 minutes to complete. The Command Center team that organized the data center failover event felt that everything went flawlessly when managing with Cutover.
Validating the data center failover plan and runbook
Any minor hiccups with users were swiftly resolved and despite it being a complex and high-risk operation, all the applications were successfully failed over. The team is already using Cutover to analyze and plan for the next event, using the insights provided by the platform from this event to improve for next time.
Key success factors
To following success factors contributed to the successful data center failover:
- Strong data center failover plan converted into an executable runbook
- Close collaboration and communication between team members
- Comprehensive training and thorough testing of the failover plan
- Effective orchestration of all data center failover tasks
“Using Cutover, we are now able to easily orchestrate large events like this one. The platform also helps us gain insights and find areas for improvement for future events.”
- Command Center
Let Cutover runbooks help you with your own data center failover plan
Contact Cutover to learn how the Collaborative Automation platform can streamline failover testing and IT disaster recovery execution.
