IISF Release. Date: 19-July-2024
What happened?
The issue originated from a bug in a single content update to Microsoft Windows hosts by Cybersecurity company CrowdStrike. This update to their anti-virus product "Falcon sensor" rolled out last night, caused business systems of many organisations the world over to crash and experience the infamous “blue screen of death” (BSOD), leading to widespread disruption to connected business and peripheral information systems. The ‘Blue Screen of Death’ or BSOD loop on Windows is a system crash, where the Windows kernal can no longer operate.
CrowdStrike’s CEO, George Kurtz, confirmed that the problem was caused by a "defect" in the most recent update to Falcon sensor.
Crowdstrike Falcon Anti-virus control panel UI
The outages varied in duration depending on the industry and region. Most sectors experienced disruptions for several hours, with some simpler services being restored within a few hours after the issue was identified and a correction was deployed by CrowdStrike. However, other larger industries, like airlines, banking and healthcare, face longer recovery times due to the complexity of their systems and the nature of their deployment.
Ireland, for reasons relating to our smaller economy and general inertia to new technology adoption has meant that Crowdstrike product and services has not been as widely taken up; so aside from Ryanair reporting major disruption,other impact in the Republic seemed to be relegated to some services of the Road Safety Authority (RSA) resulting in a small number of National Car Test Service (NCTS) centres not being able to carry out tests or contacting affected customers. There were reports that Leap card top up apps were not functioning for a while, forcing customers to topup over the phone instead.
Root Cause and Response
Microsoft’s Windows remains one of the most used operating systems powering the core of the world’s business IT infrastructure. Anti-virus software, by virtue of it's critical function is embedded in nearly all modern computer systems and has kernal access normally not granted to third party business software. In addition, to bypass the normal testing/checking/validation cycle normally required by Microsoft of new kernal driver updates, Crowdstrike engineered a method whereby they could use their existing validated drivers to load and run external code in the kernal bypassing the same validation oversight. Secondly, Crowdstrike's Falcon sensor software and support systems have been adopted to a very high level within larger businesses worldwide.
"Disastrously, that access, both from its adoption by big business to the kernal access affoarded to it's anti-virus drivers created an unprecedented vulnerability that got exposed dramatically today."
While Crowdstrike quickly issued information on a workaround and later deployed replacement updates, (to prevent any more systems becoming affected) IT teams around the world are struggling to address the task of restoring those locked systems and recover from backlogs, dataprocessing delays, manual changes and transactions impacted during the outage; something predicted to take many days, and possibly weeks as CrowdStrike has advised customers that affected machines need to be booted into “safe mode”, and then a specific .sys file(s) deleted manually to restore function. (managed modern IT systems using encrypted backup deployment technology may even take longer)
CrowdStrike_BSOD_Loop_Issue.pdf (ncsc.gov.ie).
Read Crowdstrikes Statement on Falconcontent Update for windows Hosts
References
(1) CrowdStrike outage sparks global chaos with airline, banking and other disruptions. https://www.msn.com/en-us/money/companies/crowdstrike-outage-sparks-global-chaos-with-airline-banking-and-other-disruptions/ar-BB1qg2I0.
(2) Live updates: Microsoft outage disrupts global IT; Airlines ground .... https://www.nbcnews.com/news/world/live-blog/live-updates-it-outage-flights-banks-businesses-microsoft-crowdstrike-rcna162669.
(3) Global IT outage live: Computer havoc caused by Crowdstrike outage .... https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960.
(4) Mass worldwide IT outage affects airlines, media and banks - BBC News. https://bing.com/search?q=todays+global+IT+outages.
(5) Internet outage latest | Airlines, businesses hit by global technology disruption. https://www.msn.com/en-us/travel/news/internet-outage-latest-airlines-businesses-hit-by-global-technology-disruption/ar-BB1qhgeJ.
(6) Faulty CrowdStrike update causes major global IT outage, taking out banks, airlines and businesses globally. https://www.msn.com/en-us/money/companies/banks-airlines-brokerage-houses-report-widespread-outages-across-the-globe/ar-BB1qg1pm.
If you are interested in finding out more about the IISF, or would like to attend one of our Chapter Meetings as an invited guest, please contact the
IISF Secretary:
By email:
secretary@iisf.ie
By post:
David Cahill
Information Security
GPO, 1-117
D01 F5P2
Enhance your Cybersecurity knowledge and learn from those at the coalface of information Security in Ireland
Invitations for Annual Sponsorship of IISF has now reopened.
Sponsors are featured prominently throughout the IISF.IE website, social media channels as well as enjoying other benefits Read more