Microsoft was just handed a two-piece combo of failures over the past twelve hours as it reported an Azure failure just a few hours before a separate CrowdStrike security update forced Windows devices around the world into Blue Screen of Death (BSOD) boot loop.
While the CrowdStrike update that rendered banks, airlines, broadcasting, hospitals, schools, and other organizations inert over the past twelve hours has understandably sucked up much of the news oxygen, Microsoft also suffered an Azure outage hours before the third-party security software provider decided to pile on.
As the Azure outage is a Microsoft issue, the company has issued a workaround for customers still experiencing issues independent of the greater CrowdStrike problem. As benign as it may sound, Microsoft suggests that customers simply turn off Azure virtual machines, and then restart them until they are back to normal.
We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines. Several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.
The manual reboot process seems straightforward for anyone who backed their system up before 1900 UTC yesterday. However, for others, the process could require that the OS disk be repaired offline as well as seeking out Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys and delete this as well before bringing OS disk back online to avoid any complications caused by the additional CrowdStrike problem.
Speaking of the CrowdStrike problem, Microsoft’s brand took a much larger black eye when the widely used third party security software company pushed an update late yesterday that sent thousands of Windows machines into a BSOD boot loop. The Texas based company services roughly 29,000 customers of which more than five hundred lands under the Fortune 1000 qualifier, which means the outage inducing update issued yesterday took out the Federal Aviation Administration (FAA), banks, broadcasting, supermarkets, schools, automotive manufacturing, and sectors of governments internationally.
The specifics of the CrowdStrike update issue are being reported as untested kernel-level driver from the systems platform Falcon Sensor that’s sent Windows based devices into a boot loop.
While some systems are back up and the grounding order from the FAA has recently been lifted, CrowdStrike has issued a workaround for systems still impacted by the previous update.
CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted.
George Kurtz – CrowdStrike CEO
We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.
As a result, both CrowdStrike and Microsoft have taken losses in the market, (as pissed off traders systems come back online) with the former’s stock losing 21 percent of its value or roughly $16 billion while Microsoft shed some $60 billion or 2.5 percent of its stock price.
More damaging than the financial hit is the brand gut punch Microsoft just took from the two outages. While one was self-inflicted, the much larger one was due to a third party and will have some affected organizations questioning Windows as primary solution in the future, as well as probably triggering some regulatory investigations intended to mitigate potential future occurrences of similar magnitude.
Microsoft may survive the day with some hotfixes from itself and CrowdStrike but when an outage can take out half the world from flying, shopping, commerce, governing, providing healthcare or education, and more, there will be tough questions and tougher decisions being made in the future.
