August 27, 2024
…By Paul O’Dell, CPP Principal/Sales Director
Many of you know already that I am a huge fan of Stephen Covey and the principles he laid out in his landmark book, “The 7 Habits of Highly Effective People.” Back in 2021, I wrote about a concept that came out of that book which we’ve adopted as a core value of our company: “High tech without high touch doesn’t work.” That statement has never been more relevant and cautionary as it has been in the aftermath of the CrowdStrike incident on July 19, 2024.
Now that the dust has settled, companies have taken a sobering look at the enormous impact a bad piece of code – hidden in an automated software update from the security vendor CrowdStrike – can have on such a huge swath of organizations across the globe. Delta Airlines measured the impact at $500 million (and growing); the total lost to the Fortune 500 was estimated at $5.4 billion.
And now the reckoning begins. IT leaders were stunned to realize how much exposure they had by trusting their major technology vendors with access to their business-critical systems at any time to make automated updates. Most of the damage was done to companies who were heavily dependent on Microsoft Windows; an estimated 8.5 million devices were initially struck, with cascading impact growing exponentially throughout organizations’ IT systems. (Delta reported that their IT department had to work tirelessly for days to manually reset 40,000 servers.)
How could it have been avoided?
Here’s where the “high touch” part comes in. At CPP Associates, we recognize the pivotal role of the human element in ensuring that technology solutions are effectively and responsibly implemented. The CrowdStrike outage is a compelling reminder of the need to balance the allure of automation and other sophisticated (and quickly evolving) technology like AI with good, old fashioned human knowledge, reasoning, and problem-solving to evaluate, prevent and address potential risks.
While cloud computing offers businesses unprecedented scalability, flexibility, and cost-efficiency, it is vital to acknowledge that technology alone cannot guarantee success. The human touch — comprising smart, skilled professionals who can critically assess and manage these technologies — is indispensable. Part of being wise is considering the alternatives to a “cloud only” approach — like hybrid cloud or multi-cloud — as a hedge against the risk of major outages beyond your control. No longer is “easy and fast” the primary factor in mapping out your cloud strategy. Costs have becoming more relative over the last few years. Now RISK is being weighed heavily as well.
The CrowdStrike outage, which resulted in significant disruptions and financial losses, highlights the importance of rigorous testing and quality assurance. Although automated updates are convenient, they can introduce vulnerabilities if not thoroughly vetted. This incident has prompted many IT executives to adopt a more cautious approach, emphasizing the need for thorough testing before deploying updates in production environments. Many industry analysts are doubling down on the importance of risk mitigation techniques such as “canary deployments” — preliminary rollouts under controlled conditions prior to broader deployments. No doubt, there will continue to be much examination and discussion – by some very smart human beings – about how to avoid a potentially catastrophic incident in the future.
At CPP Associates, we firmly believe that the synergy between high tech and high touch is the key to successful technology implementation. Our dedicated team of experts meticulously ensures that every technological solution is not only cutting-edge but also reliable and secure. By combining advanced technology with human oversight and expertise, we can deliver solutions that adhere to the highest standards of quality and performance.
Contact us today if you would like an independent evaluation of your potential vulnerabilities and an action plan to mitigate them.