What The CrowdStrike Incident Taught About Preparedness
When last Friday’s CrowdStrike Incident update caused a massive IT outage, Neal Juern’s day started like many other MSPs’ that day—a scramble to help customers get their systems back up and running.
Thanks to a solid incident response (IR) plan, some creative thinking, and a key vendor relationship, the MSP had most of its affected customers up and running by the end of the day, with some stragglers left on Monday morning. Plus, they took the opportunity to have a post-mortem to apply the lessons learned to become more resilient for any future event, says the CEO of 7tech, a national MPS/MSSP in San Antonio, Texas.
Blue Screen Of Death
The IT outage first came to light about 3:00 am when his overnight techs noticed that their RMM, which they host internally, went offline. Yep, they saw the “blue screen of death.” They quickly figured out it was a CrowdStrike Incident, but because the RMM was down, “we didn’t have full visibility into all of our client systems to know if they were up or down,” Juern recounts. “But once we got that back online, which took about an hour, we started to see, ‘Oh my gosh, this isn’t just us.”
As we all know now, it was an update from CrowdStrike that caused the outage. “Even though they [CrowdStrike] pushed out that bad file, because it doesn’t get rolled to every single system immediately, and they then corrected it, it didn’t impact every system.”
Juern says it did impact about 50% of their customers’ Windows servers and about 20% of their workstations.
Putting The IR Plan Into Action
The manual process of getting their systems back up began around 5:00 am, he says, and by then all staff were following the IR plan.
“We were pleased with the way our CrowdStrike Incident response plan worked,” Juern says. The team was first alerted via a Teams channel. The plan specifies that “nobody comes into our office. They all stay where they’re at [so they] don’t waste any time commuting. They start work early and start working with clients,” he explains. The operations team communicates via a dedicated Teams channel, sharing knowledge of what’s working and what isn’t. Should the Teams channel become unavailable, everybody knows to switch to a channel they built inside RingCentral.
“All our team just put their heads down and, man, they cranked it out pretty fast.”
Why Vendor Relationships Matter
The outage required a manual fix to reboot servers and workstations, and 7tech was well underway doing just that when Juern received a text message from one of his vendor partners, ThreatLocker CEO Danny Jenkins, who knew Juern also used CrowdStrike. He suggested a fix that would be faster than doing it manually—something Juern had been thinking about too, he recalls.
“We both kind of had the same idea, which is, because CrowdStrike and ThreatLocker both start up as what are called kernel drivers, they start up very early in the boot-up process and they also have exclusive access. The idea that I was talking to Danny about was [trying to] have ThreatLocker block access to the bad file for CrowdStrike … and then it will boot up. Then the CrowdStrike could push out that newer file.”
After Jenkins and Juern brainstormed, they had their two top technical people build a storage rule that blocked access to the file, which took a few hours to get right. Then ThreatLocker “published it into their threat ops library where any CrowdStrike partner could then just grab it and apply that rule and get their systems up,” Juern says.
Request A Quote
Your business depends on your IT systems. We’re confident that we are the only network services, computer support, or IT consulting business in Alabama that provides these benefits to your business.