I've always loved a trick play, whether in football or soccer (football, right?) and so my interest was piqued when seeing this article which talked about how trick plays are conceived and then executed.
I, like a good Manchester United fan, don't live in the locale of my favorite team, the Patriots, but have followed them for years. In recent times, it's been interesting to see the difference that a quarterback makes to the overall team's success (more on that another time), but also the willingness of a team to try things that their opponents aren't expecting. The Philly Special will live long in the memory of Eagles fans, as well as Patriots fans, for what it ultimately meant for the winners of Super Bowl LII.
One sentence in the article I mentioned above from Josh McDaniels particularly stood out.
"I say to our players, 'if your heartbeat goes up when we call this, we haven't run it enough in practice.'”
I guess I hadn't always appreciated the work that goes into trick plays, thinking (wrongly) that they are a play of last resort. That perhaps the players and the coaches are reluctant to execute a play with a higher degree of risk, and therefore perhaps with a greater chance of failing to complete. Certainly, there are risks in calling such a play, but it's interesting to consider what we might learn from the approach that the Patriots (and others) take to prepare themselves for an unusual scenario.
So, let's consider what this might mean for all things operational resilience - what relevance does a football team practicing and executing trick plays have, relative to how an organization prepares for and executes successfully when something unforeseen occurs? Here are my top 5 takeaways:
- Unexpected events will happen
There's a reason trick plays are effective - they're an unexpected way of advancing the team down the field, typically because the team is in an unexpected situation. Perhaps they're at fourth down but still needing yards, as they're behind on points. Football teams are not anticipating that they will need a fourth down conversion each time they need to advance the ball ten precious yards.
In the world of IT, where ever-increasing complexity is the order of the day, there is an ever-increasing chance of an unexpected event. How you respond to those events will dictate how well your business can demonstrate its ability to weather a storm. In other words, how resilient is your organization when faced with what looks like an impossible scenario?
- Consider, and plan for, scenarios outside of the norm
The Patriots, like other teams in the NFL, plan for a whole raft of scenarios. We've seen those laminated, highlighted, detailed lists of plays that coaches have constructed for all manner of scenarios. The quarterback often has an equivalent list on their wrist, such that they can translate the instruction from the coach into actionable plays. This list of plays has clearly been constructed over time, based on knowledge captured about the best way their team can execute plays, but also what they need to execute to overcome the defense of the opposition. Interestingly, McDaniels calls out the role that every player on the field must execute for the play to work successfully - and it's not just the job of the coaches to do this - they welcome input from players into what they think will work for the team as a whole.
In an IT organization (and what business doesn't now depend on technology?), we know that incidents will happen. That unexpected turn of events that causes you, as the service owner, or accountable executive, to have to drop everything and deal with the situation in front of you. How are you going to deal with that server that's failed? Or that mainframe that requires its workload to be failed over to its backup counterpart because of an issue with the production system? Perhaps you have to plan for, and evidence to regulators, your ability to deal with a total loss scenario - the loss of your primary data center, or issues with your primary AWS region.
Organizations demonstrate effective resilience by planning for and rehearsing scenarios such as the above, and more. They detail the steps they're going to take to recover from any impact, the people that will respond and take ownership of recovery, and the roles they're going to take in doing so.
- Reps are key or rehearse, rehearse, rehearse
In the article McDaniels says this:
"I usually tell the guys that we're going to rep them in practice enough and rep them long enough, sometimes for weeks, to the point where none of us feel like it's a trick," explained McDaniels. "I think once we practice it enough and, a lot of times for me, we have to see a bad look and then make a good decision in practice, not make the play into a turnover in practice and see the reaction from the players when those things happen. Once we cross over that line and feel comfortable with the decision making based on who's going to be touching the ball then I don't really feel like it's a huge risk because I trust the guys, I trust what we're doing and we've seen it on tape and we've got a lot of examples of it in practice."
Practice is clearly important in any sport. McDaniels calls out the need to repeat the play, the approach, again and again, to a point where it becomes second nature for the players and the coaches. Such that where you or I might marvel at the uniqueness of the play we're seeing in front of us, the players are treating it as just one more play.
Cutover customers use our platform to rehearse their response to a particular scenario - they do this again and again, such that a highly complex event, like a data center failover, can be orchestrated in a matter of hours. An event might have hundreds of participants, all with their own specific roles and parts to play in demonstrating that an organization can deal with a total loss scenario.
IT organizations do this to build muscle memory and ensure that when they come to the time of execution, when an incident has been declared, that their "heartbeat doesn't go up" when executing a response, because they've practiced and rehearsed sufficiently.
- Clear communication is paramount
Those that watch football on a regular basis will appreciate that when you're playing at your competitor's stadium with all the attendant, partisan support of those home fans, hearing calls from the coach, and then from quarterback to offense, can be particularly challenging. So much so that teams will resort to hand signals at the line of scrimmage to effectively communicate when a quarterbacks voice cannot be heard.
In executing a trick play, clear communication is important - in practice, everyone needs to understand what their particular role is, who's going to run what route, who's going to throw the football, and who the target receiver is. That clear communication has to move over into game day, and if voice communication is difficult, then signaling via a different mechanism and relying on muscle memory is key for successful execution.
How IT organizations respond to an event or incidents often dictates how successful they are. Not only do individuals need to understand their role, but they also need to clearly communicate with each other as that response is executed. Organizations need a variety of cues - spoken, written, and other visualizations, to gain clear context and situational awareness.
- Understand your role and that of others on the team for the play you're executing
I confess, being a Brit, at times I still have to work out who's doing what in a particular football team! Being used to an 11 man team (and subs) playing soccer, I marvel at a 53 man football team, each member of that team ready to do the role they've been hired and trained for on game day. Each understands what their specific role is on the team, as well as what their specific role is in a given play for a given scenario.
The interesting thing about trick plays, though, is that often, roles are switched. The quarterback becomes a receiver, that receiver that played quarterback in high school now throws the ball to the "converted" quarterback. But the key here (as mentioned above) is that they all understand what their roles are for that particular scenario, irrespective of the role they may have been signed to!
When executing a complex activity such as a datacenter failover, each participant has to understand their role, the activities they're executing, and the sequencing of that activity. One of our core principles at Cutover is that everyone should ‘see’ (literally!) all activity on a runbook so that they understand the impact of one task vs another, and everyone understands the context of their activity in light of the bigger picture.
Bringing this all together, how does Cutover help?
- Plan your response Through the use of runbooks that detail a planned, canned, or unplanned response to a particular scenario or set of scenarios, customers are able to build a comprehensive library.
- Cater for a variety of scenarios Runbooks can be crafted for a variety of scenarios, including how to respond to a DDOS attack, a datacenter failure, or to provide a set of instructions as to how to fail a particular service from one location to another.
- Rehearse to build muscle memory Cutover facilitates rehearsal of runbooks that can be dynamically edited in real time on the fly, as well as providing a detailed evidentiary trail and feedback as to how one event performed vs another.
- Communicate to keep your team informed In-built communication features, as well as integration to third-party tools such as Slack, Microsoft Teams, or Mattermost allow for timely, clear, and effective communication.
- Real-time visibility During execution, customers can understand how long an activity has taken vs the forecast for that same activity, with appropriate, real-time, custom dashboarding, providing a common view for all to understand how an event is progressing.
- Understand the context of your work in the light of others Participation in the theater of orchestration, in Cutover, allows all to see their allocated work and what that means in the context of the larger picture.
The moral of the story? The next time you see a trick play on a Sunday afternoon, consider all that's gone into it - the planning, the rehearsing, the communication, the clear definition of roles - all so that execution of that play proceeds without a hitch.
Find out more about how you can use Cutover for technology resilience.