Test Software Bug and EU Agreements Blamed for Global CrowdStrike Outage

• CrowdStrike’s preliminary report says a bug in the testing software let a faulty update pass validation.
• Problematic content data slipped through the Content Validator and was deployed worldwide.
• Microsoft added they could have prevented the unfortunate incident were it not for the 2009 European agreement.

On Wednesday, a preliminary Post-Incident Review (PIR) was added to the CrowdStrike remediation guide. It states that problematic content data slipped through the testing software on July 19, when a Windows sensor Rapid Response Content update was deployed. This triggered an unexpected exception that resulted in a system crash, displaying the Blue Screen of Death (BSOD).

'Rapid Response Content' gathers telemetry on possible novel threat techniques and is delivered as content configuration updates to the Falcon sensor via the Content Configuration System, the Content Interpreter, and the Sensor Detection Engine.

The Content Configuration System is part of the Falcon platform in the cloud and is used to create Template Instances. These instances go through the Content Configuration System, which includes the Content Validator software, which checks and validates content before being published.

New Template Types are stress-tested for resource utilization, system performance impact, event volume, and more. A specific Template Instance is used to stress-test each Template Type to identify adverse system interactions.

One of the two additional InterProcessCommunication (IPC) Template Instances deployed on July 19 contained problematic content data in Channel File 291. Still, they passed validation due to a bug in the Content Validator, which resulted in an out-of-bounds memory read and triggered an unexpected exception that crashed Windows systems.

These instances were deployed into production after the March testing, Content Validator checks, and previous successful IPC Template Instance deployments.

Microsoft said it might have blocked the faulty CrowdStrike update if not for the 2009 agreement with the European Commission, which allows several security providers to install software at the kernel level to prevent anti-competitive behavior. 

The tech giant has already released an updated recovery tool with two repair options to help IT admins make fixing easier. 

The faulty CrowdStrike update caused mayhem worldwide, resulting in flight and train cancellations and the shutdown of healthcare systems, among other things. Mac and Linux hosts were not impacted – only Windows hosts running sensor version 7.11 and above that were online on July 19 between 04:09 UTC and 05:27 UTC and received the update.