The CrowdStrike Falcon Sensor update for Windows disaster
Intro
References
- CISA, Widespread IT Outage Due to CrowdStrike Update: https://www.cisa.gov/news-events/alerts/2024/07/19/widespread-it-outage-due-crowdstrike-update
- ACSC, Widespread outages relating to CrowdStrike software update: https://www.cyber.gov.au/about-us/view-all-content/alerts-and-advisories/widespread-outages-relating-crowdstrike-software-update?utm_source=linkedin&utm_campaign=jul-2024&utm_medium=social&utm_content=alert-crowdstrike-outage-update
- CrowdStrike Statement on Falcon Content Update for Windows Hosts: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
- Here’s how IT admins are fixing the Windows Blue Screen of Death chaos: https://www.theverge.com/2024/7/19/24201806/microsoft-windows-bsod-pc-crashing-crowdstrike-fix
- Malicious domains exploiting CrowdStrike Outage: https://socradar.io/suspicious-domains-exploiting-the-recent-crowdstrike-outage/
The nerdy stuff
Security agents running on the end point install different components that operate at the user and kernel levels. In Microsoft Windows OS, Kernel Patch Protection (KPP), informally known as PatchGuard, is a feature that prevents patching the kernel. It was first introduced in 2005 with the x64 editions of Windows XP and Windows Server 2003 Service Pack 1. “Patching the kernel” refers to unsupported modification of the central component or kernel of the Windows operating system. Such modification has never been supported by Microsoft because, according to Microsoft, it can greatly reduce system security, reliability, and performance.
The kernel components of end point protection solutions are primarily system drivers able to “listen” to telemetry using what’s called “kernel call-back notifications”. When these drivers are installed they register with the kernel and “subscribe” to different kernel notifications. From that moment on, the kernel will interact with (notify) the driver when some events happen. Different routines running in the kernel manage events such as process creation, network connections, registry operations, etc. The driver will then interact with the user space of the OS.
During the notification process, and although Microsoft doesn’t allow 3rd parties to “patch the kernel”, the kernel can still crash if the driver installed doesn’t “behave” as expected.
For what’s been published so far, the root cause for sending hundreds of thousands of Windows machines to a kernel crash is a driver with bad code (or incomplete). This driver was part of a CrowdStrike Falcon “minor” update and distributed in a similar fashion to “files signature updates”.
(How a “faulty channel file” from Crowdstrike bricked Windows computers everywhere)
For reasons still unknown, the new driver that was included in the update was full of null characters causing the kernel to crash (no code). Lack of routines or functions in the new code deployed after the update returned no results to the kernel routine making the call back.
So, how to remove the bad driver?
From CrowdStrike Statement on Falcon Content Update for Windows Hosts: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
In some articles, workarounds like reboot the system 15 times (??!!) or use AD Group Policies (??) to fix the issue at scale have been published.
When security gets in the way of….security: BitLocker.
In environments with BitLocker, or similar disk encryption solution enabled, the manual process to boot the system and go into safe mode won’t be an easy task.
NIST Guidelines for software Updates
NIST Security and Privacy Controls for Information Systems and Organizations: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf
SI-2(5): Automatic Software / Firmware Updates
Control Statement
The organization installs organization-defined security-relevant software and firmware updates automatically to organization-defined information system components.
Supplemental Guidance
Due to information system integrity and availability concerns, organizations give careful consideration to the methodology used to carry out automatic updates. Organizations must balance the need to ensure that the updates are installed as soon as possible with the need to maintain configuration management and with any mission or operational impacts that automatic updates might impose.
Need Help?
The functionality discussed in this post, and so much more, are available via the SOCFortress platform. Let SOCFortress help you and your team keep your infrastructure secure.
Website: https://www.socfortress.co/
Contact Us: https://www.socfortress.co/contact_form.html
