×

Irish Information Security Forum

Anti-virus update causes Global IT outage

IISF Release. Date: 19-July-2024

 

A significant global IT outage on July 19, (2024), disrupted multiple sectors, including airlines, transport networks, stock exchanges, banks, retail on the high street, medical and emergency services. This incident, linked to a bug in a update made to a anti-virus software that is popular with big firms has thrown into the spotlight a huge vulnerability across the world's interconnected digital infrastructure.

 

What happened?


The issue originated from a bug in a single content update to Microsoft Windows hosts by Cybersecurity company CrowdStrike. This update to their anti-virus product "Falcon sensor" rolled out last night, caused business systems of many organisations the world over to crash and experience the infamous “blue screen of death” (BSOD), leading to widespread disruption to connected business and peripheral information systems. The ‘Blue Screen of Death’ or BSOD loop on Windows is a system crash, where the Windows kernal can no longer operate. 

 

BSOD loop Blue Screen of Death windows 11

CrowdStrike’s CEO, George Kurtz, confirmed that the problem was caused by a "defect" in the most recent update to Falcon sensor. 
 

Crowdstrike Falcon Anti-virus screen of death update
Crowdstrike Falcon Anti-virus control panel UI

 

Impact on Various Sectors (worldwide)

 

The outages varied in duration depending on the industry and region. Most sectors experienced disruptions for several hours, with some simpler services being restored within a few hours after the issue was identified and a correction was deployed by CrowdStrike. However, other larger industries, like airlines, banking and healthcare, face longer recovery times due to the complexity of their systems and the nature of their deployment.

 

  • Airlines:  Major airlines across the world, (including United, American, Wizz and Ryanair) were forced to ground flights due to the resultant IT outage causing check-in systems worldwide to be disrupted, causing significant delays and cancellations. Additionally airports have also been seeing problems with their booking systems, including the knock on effects to other airlines and support services that were not directly affected by the outage.
  • Financial Institutions: Banks and stock exchanges experienced significant downtime, affecting transactions and trading activities. 
  • Stock Markets: Major indices experienced sharp declines as investors reacted to the disruptions. Tech stocks, in particular, were hit hard due to their reliance on IT infrastructure.
  • Banking Sector: Banks faced operational challenges, leading to delays in transactions and trading activities. This caused some fluctuations in banking stocks. Banks also experienced outages that affected their online services and ATMs.
  • Cryptocurrencies: The outages also affected cryptocurrency exchanges, leading to temporary halts in trading and increased volatility in crypto prices.
  • Media: News organizations and broadcasters faced outages, impacting their ability to deliver content reported network outages, impacting their broadcasting and online services. Sky News, CBBC, MTV and ESPN went off-air.
  • Healthcare: Hospitals and healthcare providers reported issues with patient services and appointment systems. In Germany, the University Hospital Schleswig-Holstein had to cancel elective procedures and outpatient services due to the IT issues. The British National Health Service (NHS) reported disruptions in general practitioner practices, forcing them to revert to paper records and handwritten prescriptions.
  • Retail: Some retail stores couldn’t process payments due to the outages, forcing some to resort to accepting cash only.
  • Transportation: Train services in the UK and other regions experienced serious delays and cancellations.

 

 

Impact in Ireland

 

Ireland, for reasons relating to our smaller economy and general inertia to new technology adoption has meant that Crowdstrike product and services has not been as widely taken up; so aside from Ryanair reporting major disruption,other impact in the Republic seemed to be relegated to some services of the Road Safety Authority (RSA) resulting in a small number of National Car Test Service (NCTS) centres not being able to carry out tests or contacting affected customers. There were reports that Leap card top up apps were not functioning for a while, forcing customers  to topup over the phone instead.

 

 

Root Cause and Response

 

Microsoft’s Windows remains one of the most used operating systems powering the core of the world’s business IT infrastructure. Anti-virus software, by virtue of it's critical function is embedded in nearly all modern computer systems and has kernal access normally not granted to third party business software.  In addition, to bypass the normal  testing/checking/validation cycle normally required by Microsoft of new kernal driver updates, Crowdstrike engineered a method whereby they could use their existing validated drivers to load and run external code in the kernal bypassing the same validation oversight. Secondly, Crowdstrike's Falcon sensor software and support systems have been adopted to a very high level within larger businesses worldwide.


"Disastrously, that access, both from its adoption by big business to the kernal access affoarded to it's anti-virus drivers created an unprecedented vulnerability that got exposed dramatically today."

 

While Crowdstrike quickly issued information on a workaround and later deployed replacement updates, (to prevent any more systems becoming affected)  IT teams around the world are struggling to address the task of restoring those locked systems and recover from backlogs, dataprocessing delays, manual changes and transactions impacted during the outage; something predicted to take many days, and possibly weeks as CrowdStrike has advised customers that affected machines need to be booted into “safe mode”, and then a specific .sys file(s) deleted manually to restore function. (managed modern IT systems using encrypted backup deployment technology may even take longer)

 

Immediate Workarounds released

Information on the Crowdstrike BSOD workaround was released by the NCSC.

CrowdStrike_BSOD_Loop_Issue.pdf (ncsc.gov.ie). 
Crowdstrike BSOD Loop workaround fix cover

 

 Read Crowdstrikes Statement on Falconcontent Update for windows Hosts

falcon content update for windows hosts

 


References

(1) CrowdStrike outage sparks global chaos with airline, banking and other disruptions. https://www.msn.com/en-us/money/companies/crowdstrike-outage-sparks-global-chaos-with-airline-banking-and-other-disruptions/ar-BB1qg2I0.
(2) Live updates: Microsoft outage disrupts global IT; Airlines ground .... https://www.nbcnews.com/news/world/live-blog/live-updates-it-outage-flights-banks-businesses-microsoft-crowdstrike-rcna162669.
(3) Global IT outage live: Computer havoc caused by Crowdstrike outage .... https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960.
(4) Mass worldwide IT outage affects airlines, media and banks - BBC News. https://bing.com/search?q=todays+global+IT+outages.
(5) Internet outage latest | Airlines, businesses hit by global technology disruption. https://www.msn.com/en-us/travel/news/internet-outage-latest-airlines-businesses-hit-by-global-technology-disruption/ar-BB1qhgeJ.
(6) Faulty CrowdStrike update causes major global IT outage, taking out banks, airlines and businesses globally. https://www.msn.com/en-us/money/companies/banks-airlines-brokerage-houses-report-widespread-outages-across-the-globe/ar-BB1qg1pm.

 



IISF Logo

If you are interested in finding out more about the IISF, or would like to attend one of our Chapter Meetings as an invited guest, please contact the
IISF Secretary:

By email:
secretary@iisf.ie

By post:

David Cahill

Information Security

GPO, 1-117
D01 F5P2

Enhance your Cybersecurity knowledge and learn from those at the coalface of information Security in Ireland

 


Forum SPONSORS 

Invitations for Annual Sponsorship of IISF has now reopened.

Sponsorship of IISF Opportunity
(your logo & profile link here)

 

Sponsors are featured prominently throughout the IISF.IE website, social media channels as well as enjoying other benefits Read more

 

secured by edgescan digital security radar logo

© iiSf. All rights reserved. CRN: 3400036GH  - Privacy  - Sponsorship  - Cybersecurity News Index  - Cybersecurity Resources  - X  - Produced by
LinkedIn Twitter