IT WAS A REGEX?!? - Full CrowdStrike Report Released

"We solved it with regex!" .. "Which variant of regex?" .. "What do you mean which variant of regex?"

It's so unfair. When these nerds regex, they get a billion dollars. When I regex all over myself, the cops get called!

CrowdStrike: "We investigated ourselves and found it was actually a boo boo and not an owie. Please leave."

Regex in kernel mode? That somehow sounds like the weirdest thing ever

A regex with over 20 parameters, what could go wrong

Damn 12 pages to say it worked on my machine. Also regex is the root of all evil.

To clarify, CrowdStrike's testing and deployment process specifically for channel 291 was operating on what you'd call the "Hopes and Dreams" algorithm. Got it.

The “real” problem is that someone applied the wrong shirt size to a Jira ticket.

That's a lot of text to say, "We didn't test it prior to deploying to millions of installations worldwide." They only needed to deploy it to an internal instance to prove that it would crash everywhere. In other words, this is why 100% unit test coverage gives managers a false sense of security.

Canaries and staged rollouts - what a novel idea.

Crowdstrike: "We tested it" Skeptical Tester: "According to the code coverage? Was it fuzzed?"

CrowdStrike: "GASLIGHT, TECH JARGON, AND DENY! WE GOING BANKRUPT!!!!"

12 pages to say "we now properly check the length of input"

Them: Everything broke because we didn't test for a 21st variable. Fix: Test for a 21st variable. Upcoming news: Everything broke we didn't test for a 22nd variable.

"I'll use regex here" Now you have 291 problems.

CrowdStrike: It was not a null dereference error! It was an off by one error!

I’ll just build my own regex parser that runs in the kernel, what’s the worst that could happen? 😀

When you distill all of that down, it's a simple data interface error made worse by lapses in good testing procedures. They're deliberately making it sound complicated so that readers will glaze over saying: "Well... I guess it was a really tough issue. Pretty understandable."

Cloudflare + Cloudstrike both got wrecked by Regexes. I can’t wait for Kevin Fang’s video on this

I want to point out here at 4:50 and 8:10. They said they tested the "Template", they did not say they actually tested channel file 291... It really sounds like they did not, in fact, test 291... Unless this is covered later, it doesn't seem like this is a "it worked on my machine", it seems like "well we assumed it would work because we weren't 'changing code'". 21:00 and there we go..... 38:00 so basically yes. "Channel files" were not viewed as being dangerous to update by Crowdstrike because "they're not code", and so they had minimal or no testing. This is astoundingly negligent in my opinion.