• 2 Posts
  • 158 Comments
Joined 1 year ago
cake
Cake day: June 28th, 2023

help-circle





  • Microsoft gave CrowdStrike unfettered access to push an update that can BSOD every Windows machine without a bypass or failsafe in place. That turned out to be a bad idea.

    CrowdStrike pushed an errant update. Microsoft allowed a single errant update to cause an unrecoverable boot loop. CrowdStrike is the market leader in their sector and brings in hundreds of millions of dollars every year, but Microsoft is older than the internet and creates hundreds of billions of dollars. CrowdStrike was the primary cause, but Microsoft enabled the meltdown.


  • Even if that’s the case, how is it Crowdstrike’s place to call these other companies out for claiming something similar will never happen to them?

    I agree completely, which is why I added that last sentence in an edit. This is a bad look for CrowdStrike, even if I agree with the sentiment.

    Thus far, it had only ever happened to CS.

    Everybody fucks up now and then. That’s my point. It’s why you shouldn’t trust one company to automatically push security updates to critical production servers without either a testing environment or disaster recovery procedures in place.

    I doubt you’ll find any software company, or any company in any industry, that has not fucked up something really important. That’s the nature of commerce. It’s why many security protocols exist in the first place. If everyone could be trusted to do their jobs right 100% of the time, you would only need to worry about malicious attacks which make up only a small fraction of security incidents.

    The difference here is that CrowdStrike sold a bunch of clients on the idea that they could be trusted to push security updates to production servers without trsting environments. I doubt they told Delta that they didn’t need DRP or any redundancy, but either way, the failure was amplified by a collective technical debt that corporations have been building into their budget sheets to pad their stock prices.

    By all means, switch from CrowdStrike to a competitor. Or sue them for the loss of value resulting in their fuckup. Sort that out in the contracts and courts, because that’s not my area. But we should all recognize that the lesson learned is not to switch to another threat prevention software company that won’t fuck up. Such a company does not exist.

    If you stub your toe, you don’t start walking on your hands. You move the damn coffee table out of the pathway and watch where you’re walking. The lesson is to invest in your infrastructure, build in redundancy, and protect your critical systems from shit like this.


  • It’s not really criticism, it’s competitors claiming they will never fuck up.

    Like, if you found mouse in your hamburger at McDonald’s, that’s a massive fuckup. If Burger King then started saying “you’ll never find anything gross in Burger King food!” that would be both crass opportunism and patently false.

    It’s reasonable to criticize CrowdStrike. They fucked up huge. The incident was a fuckup, and creating an environment where one incident could cause total widespread failure was a systemic fuckup. And it’s not even their first fuckup, just the most impactful and public.

    But also Microsoft fucked up. And the clients, those who put all of their trust into Microsoft and CrowdStrike without regard to testing, backups, or redundancy, they fucked up, too. Delta shut down, cancelling 4,600 flights. American Airlines cancelled 43 flights, 10 of which would have been cancelled even without the outage.

    Like, imagine if some diners at McDonald’s connected their mouths to a chute that delivers pre-chewed food sight-unseen into their gullets, and then got mad when they fell ill from eating a mouse. Don’t do that, not at any restaurant.

    All that said, if you fuck up, you don’t get to complain about your competitors being crass opportunists.





  • Yeah, a lot of people are (understandably) mad at Crowdstrike right now, but I want to drag some c-suite executives into a conference room and impress upon them the value of allocating budget for test environments and disaster recovery. Banks, airlines, service providers, these aren’t mom-and-pop bakeries and plumbers who don’t have time for all that nonsense. Every service that went down should be looking for the fuckwit in their organization, and they’re probably in the executive lounge. Anyone can make a mistake, but it takes dedication to systematically ignore the best advice of top experts in the field and run your infrastructure on a shoestring budget.