The challenge: Testing data center failover
A major American bank wanted to test that they could successfully fail over a major data center and get all of their 863 critical applications back online within a set time. They also needed to be able to credibly attest to their ability to do this to the regulator.
Better preparation for data center isolation
Around 2,500 people at the bank were involved in the process, with 1,863 of them executing tasks through Cutover. Cutover’s Client Success Team conducted 25 hours of training at the bank over two weeks to ensure preparation for the event.
The bank then built the runbooks for the failover which involved 863 applications and 13 different lines of business. Each line of business had its own dashboard so they could easily filter for the relevant tasks and information.
Every participant could be contacted via SMS through Cutover. This meant that across a 16-hour event, they didn’t have to spend the entire time on an open bridge waiting to be told to start their task as they had done previously. Instead, they could step away and do other things or rest until they received the notification that they were needed. Using Cutover also allowed the bank to prepare for a key person not being available, as each task had three contacts assigned to it and if the first person was unable to respond when needed, the secondary and tertiary contacts were in the system ready to go - so there was no need to waste time searching a phone directory or spreadsheet.
Due to the scale and complexity of the event, it was also very useful that Cutover enabled users to easily filter and find their relevant tasks in the runbook, saving time and ensuring nothing was missed.
Flawless failover
In total, the event took 16 hours and 22 minutes to complete. The Command Center team that organized the event felt that everything went flawlessly when managing with Cutover. Any minor hiccups with users were swiftly resolved and despite it being a complex and high-risk operation, all the applications were successfully failed over.
The team is already using Cutover to analyze and plan for the next event, using the insights provided by the platform from this event to improve for next time.
“Using Cutover, we are now able to easily orchestrate large events like this one. The platform also helps us gain insights and find areas for improvement for future events.”
- Command Center
