There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

retrospectology ,
@retrospectology@lemmy.world avatar

This is why you create restore points if using windows.

recapitated ,

Clownstrike

lando55 ,

Crowdshite haha gotem

WhatAmLemmy ,

CrowdCollapse

kadotux , (edited )

Here’s the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching “C-00000291*.sys” 4)Boot the system normally

StV2 ,

It’s disappointing that the fix is so easy to perform and yet it’ll almost certainly keep a lot of infrastructure down for hours because a majority of people seem too scared to try to fix anything on their own machine (or aren’t trusted to so they can’t even if they know how)

r00ty Admin ,
r00ty avatar

It might not even be that. A lot of places have many servers (and even more virtual servers) running crowdstrike. Some places also seem to have it on endpoints too.

That's a lot of machines to manually fix.

HaleHirsute ,

They also gotta get the fix through a trusted channel and not randomly on the internet. (No offense to the person that gave the info, it’s maybe correct but you never know)

kadotux ,

Yeah, and it’s unknown if CS is active after the workaround or not (source: hackernews commentator)

letsgo ,

True, but knowing what the fix might be means you can Google it and see what comes back. It was on StackOverflow for example, but at the time of this comment has been taken offline for moderation - whatever that means.

huginn ,

Yeah and a lot of corpo VPNs are gonna be down from this too.

ColeSloth ,

Meh. Even if it bricked crowdstrike instead of helping, you can just restore the file you deleted. A file in that folder can’t brick a windows system.

NaibofTabr ,

This sort of fix might not be accessible to a lot of employees who don’t have admin access on their company laptops, and if the laptop can’t be accessed remotely by IT then the options are very limited. Trying to walk a lot of nontechnical users through this over the phone won’t go very well.

AccountMaker ,

Yup, that’s me. We booted into safe mode, tried navigating into the CrowdStrike folder and boom: permission denied.

Cryophilia ,

Half our shit can’t even boot into safe mode because it’s encrypted and we don’t have the keys rofl

Munkisquisher ,

And people need to travel to remote machines to do this in person

thehatfox ,
@thehatfox@lemmy.world avatar

Might seem easy to someone with a technical background. But the last thing businesses want to be doing is telling average end users to boot into safe mode and start deleting system files.

If that started happening en masse we would quickly end up with far more problems than we started with. Plenty of users would end up deleting system32 entirely or something else equally damaging.

Ookami38 ,

I do IT for some stores. My team lead briefly suggested having store managers try to do this fix. I HARD vetoed that. That’s only going to do more damage.

nyarla ,

Yes but the recovery menu may have been configured to ask for administrative credentials, to prevent unwanted access to the computer, and then fixing the problem would take way longer.

CaptainBasculin ,

A driver failure, yeesh. It always sucks to deal with it.

cheeseburger ,
@cheeseburger@lemmy.ca avatar

I’m on a bridge still while we wait for Bitlocker recovery keys, so we can actually boot into safemode, but the Bitkocker key server is down as well…

gnutrino ,

Gonna be a nice test of proper backups and disaster recovery protocols for some organisations

huginn ,

Chaos Monkey test

WagnasT ,

Man, it sure would suck if you could still get to safe mode from pressing f8. Can you imagine how terrible that’d be?

a_postmodern_hat ,

You hold down Shift while restarting or booting and you get a recovery menu. I don’t know why they changed this behaviour.

Ookami38 ,

That was the dumbest thing to learn this morning.

resin85 ,

Not that easy when it’s a fleet of servers in multiple remote data centers. Lots of IT folks will be spending their weekend sitting in data center cages.

invisiblegorilla ,

Ironic. They did what they are there to protect against. Fucking up everyone’s shit

Telorand ,

Maybe centralizing everything onto one company’s shoulders wasn’t such a great idea after all…

Excrubulent ,
@Excrubulent@slrpnk.net avatar

Wait, monopolies are bad? This is the first I’ve ever heard of this concept. So much so that I actually coined the term “monopoly” just now to describe it.

joostjakob ,

Someone should invent a game, that while playing demonstrates how much monopolies suck for everyone involved (except the monopolist)

KingJalopy ,

And make it so you lose friends and family over the course of the 4+ hour game. Also make a thimble to fight over, that would be dope.

Telorand ,

I mean, I’m sure those companies that have them don’t think so—when they aren’t the cause of muti-industry collapses.

jaybone ,

Yes, it’s almost as if there should be laws to prevent that sort of thing. Hmm

jaybone ,

The too big to fail philosophy at its finest.

StaySquared ,

CrowdStrike has a new meaning… literally Crowd Strike.

nintendiator ,

Since when has any antivirus ever had the intent of actually protecting against viruses? The entire antivirus market is a scam.

ytg ,

>Make a kernel-level antivirus
>Make it proprietary
>Don’t test updates… for some reason??

CircuitSpells ,

I mean I know it’s easy to be critical but this was my exact thought, how the hell didn’t they catch this in testing?

grabyourmotherskeys ,

I have had numerous managers tell me there was no time for QA in my storied career. Or documentation. Or backups. Or redundancy. And so on.

kandoh ,

Move fast and break things! We need things NOW NOW NOW!

thearch ,

Irrelevant but I keep reading “crowd strike” as “counter strike” and it’s really messing with me

umami_wasbi ,

No one bother to test before deploying to all machines? Nice move.

pufferfisherpowder ,

YOLO 🚀🙈

huginn ,

This outage is probably costing a significant portion of Crowd strike’s market cap. They’re an 80 billion dollar company but this is a multibillion outage.

Someone’s getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.

TheBat ,
@TheBat@lemmy.world avatar

Puts on Crowdstrike?

sugar_in_your_tea ,

They’re already down ~9% today:

finance.yahoo.com/quote/CRWD/

So I think you’re late to the party for puts. Smart money IMO is on a call for a rebound at this point. Perhaps smarter money is looking through companies that may have been overlooked that would be CrowdStrike customers and putting puts on them. The obvious players are airlines, but there could be a ton of smaller cap stocks that outsource their IT to them, like regional trains and whatnot.

Regardless, I don’t gamble w/ options, so I’m staying out. I could probably find a deal, but I have a day job to get to with nearly 100% odds of getting paid.

TheBat ,
@TheBat@lemmy.world avatar

You’re going to love this: old.reddit.com/…/crowdstrike_is_not_worth_83_bill…

sugar_in_your_tea ,

Nice. The first comment is basically saying, “they’re best in class, so they’re worth the premium.” And then the general, “you’ll probably do better by doing the opposite of /r/wallstreetbets” wisdom.

So yeah, if I wanted to gamble, I’d be buying calls for a week or so out when everyone realizes that the recovery was relatively quick and CrowdStrike is still best in class and retained its customers. I think that’s the most likely result here. Switching is expensive for companies like this, and the alternatives aren’t nearly as good.

dhork ,

I picked the right week to be on PTO hahaha

CanadaPlus ,

Yep, this is the stupid timeline. Y2K happening to to the nuances of calendar systems might have sounded dumb at the time, but it doesn’t now. Y2K happening because of some unknown contractor’s YOLO Friday update definitely is.

Cornelius_Wangenheim , (edited )

deleted_by_author

    •
    OsrsNeedsF2P ,

    Apparently the slow rollout was skipped (on Crowdstrike’s end) for this

    RaoulDook ,

    That’s true about the test group deployments, but it turned out this one was not an agent update under that control system. It’s a Channel File update that goes out to all endpoints automatically.

    jedibob5 ,

    Reading into the updates some more… I’m starting to think this might just destroy CloudStrike as a company altogether. Between the mountain of lawsuits almost certainly incoming and the total destruction of any public trust in the company, I don’t see how they survive this. Just absolutely catastrophic on all fronts.

    NaibofTabr ,

    If all the computers stuck in boot loop can’t be recovered… yeah, that’s a lot of cost for a lot of businesses. Add to that all the immediate impact of missed flights and who knows what happening at the hospitals. Nightmare scenario if you’re responsible for it.

    This sort of thing is exactly why you push updates to groups in stages, not to everything all at once.

    rxxrc OP ,

    Looks like the laptops are able to be recovered with a bit of finagling, so fortunately they haven’t bricked everything.

    And yeah staged updates or even just… some testing? Not sure how this one slipped through.

    dactylotheca ,
    @dactylotheca@suppo.fi avatar

    Not sure how this one slipped through.

    I’d bet my ass this was caused by terrible practices brought on by suits demanding more “efficient” releases.

    “Why do we do so much testing before releases? Have we ever had any problems before? We’re wasting so much time that I might not even be able to buy another yacht this year”

    GoofSchmoofer ,
    @GoofSchmoofer@lemmy.world avatar

    At least nothing like this happens in the airline industry

    dactylotheca ,
    @dactylotheca@suppo.fi avatar

    Certainly not! Or other industries for that matter. It’s a good thing executives everywhere aren’t just concentrating on squeezing the maximum amount of money out of their companies and funneling it to themselves and their buddies on the board.

    Sure, let’s “rightsize” the company by firing 20% of our workforce (but not management!) and raise prices 30%, and demand that the remaining employees maintain productivity at the level it used to be before we fucked things up. Oh and no raises for the plebs, we can’t afford it. Maybe a pizza party? One slice per employee though.

    Munkisquisher ,

    Yeah saw that several steel mills have been bricked by this, that’s months and millions to restart

    gazter ,

    Got a link? I find it hard to believe that a process like that would stop because of a few windows machines not booting.

    TheBat ,
    @TheBat@lemmy.world avatar

    a few windows machines with controller application installed

    That’s the real kicker.

    drspod ,

    Those machines should be airgapped and no need to run Crowdstrike on them. If the process controller machines of a steel mill are connected to the internet and installing auto updates then there really is no hope for this world.

    TheBat ,
    @TheBat@lemmy.world avatar

    But daddy microshoft says i gotta connect the system to the internet uwu

    IsThisAnAI ,

    What lawsuits do you think are going to happen?

    Nachorella ,

    They can have all the clauses they like but pulling something like this off requires a certain amount of gross negligence that they can almost certainly be held liable for.

    IsThisAnAI ,

    Whatever you say my man. It’s not like they go through very specific SLA conversations and negotiations to cover this or anything like that.

    Cryophilia ,

    Forget lawsuits, they’re going to be in front of congress for this one

    IsThisAnAI ,

    For what? At best it would be a hearing on the challenges of national security with industry.

    RegalPotoo ,
    @RegalPotoo@lemmy.world avatar

    Agreed, this will probably kill them over the next few years unless they can really magic up something.

    They probably don’t get sued - their contracts will have indemnity clauses against exactly this kind of thing, so unless they seriously misrepresented what their product does, this probably isn’t a contract breach.

    If you are running crowdstrike, it’s probably because you have some regulatory obligations and an auditor to appease - you aren’t going to be able to just turn it off overnight, but I’m sure there are going to be some pretty awkward meetings when it comes to contract renewals in the next year, and I can’t imagine them seeing much growth

    Skydancer ,

    Nah. This has happened with every major corporate antivirus product. Multiple times. And the top IT people advising on purchasing decisions know this.

    SupraMario ,

    Yep. This is just uninformed people thinking this doesn’t happen. It’s been happening since av was born. It’s not new and this will not kill CS they’re still king.

    jedibob5 ,

    Don’t most indemnity clauses have exceptions for gross negligence? Pushing out an update this destructive without it getting caught by any quality control checks sure seems grossly negligent.

    Wooki ,

    Testing is production will do that

    TheBat ,
    @TheBat@lemmy.world avatar

    Not everyone is fortunate enough to have a seperate testing environment, you know? Manglement has to cut cost somewhere.

    Blisterexe ,

    Manglement is the good term lmao

    ThrowawaySobriquet ,

    I think you’re on the nose, here. I laughed at the headline, but the more I read the more I see how fucked they are. Airlines. Industrial plants. Fucking governments. This one is big in a way that will likely get used as a case study.

    Cryophilia ,

    The London Stock Exchange went down. They’re fukd.

    rozodru ,
    @rozodru@lemmy.ca avatar

    It’s just amatuer hour across the board. Were they testing in production? no code review or even a peer review? they roll out for a Friday? It’s like basic level start up company “here’s what not to do” type shit that a junior dev fresh out of university would know. It’s like “explain to the project manager with crayons why you shouldn’t do this” type of shit.

    It just boggles my mind that if you’re rolling out an update to production that there was clearly no testing. There was no review of code cause experts are saying it was the result of poorly written code.

    Regardless if you’re low level security then apparently you can just boot into safe and rename the crowdstrike folder and that should fix it. higher level not so much cause you’re likely on bitlocker which…yeah don’t get me started no that bullshit.

    regardless I called out of work today. no point. it’s friday, generally nothing gets done on fridays (cause we know better) and especially today nothing is going to get done.

    Revan343 ,

    explain to the project manager with crayons why you shouldn’t do this

    Can’t; the project manager ate all the crayons

    candybrie ,

    Why is it bad to do on a Friday? Based on your last paragraph, I would have thought Friday is probably the best week day to do it.

    Lightor ,

    Most companies, money included, try to roll out updates during the middle of start of a week. That way if there are issues the full team is available to address them.

    Bell ,

    Don’t we blame MS at least as much? How does MS let an update like this push through their Windows Update system? How does an application update make the whole OS unable to boot? Blue screens on Windows have been around for decades, why don’t we have a better recovery system?

    sandalbucket ,

    Crowdstrike runs at ring 0, effectively as part of the kernel. Like a device driver. There are no safeguards at that level. Extreme testing and diligence is required, because these are the consequences for getting it wrong. This is entirely on crowdstrike.

    bdonvr ,

    The amount of servers running Windows out there is depressing to me

    franklin ,
    @franklin@lemmy.world avatar

    The four multinational corporations I worked at were almost entirely Windows servers with the exception of vendor specific stuff running Linux. Companies REALLY want that support clause in their infrastructure agreement.

    Avatar_of_Self ,

    I’ve worked as an IT architect at various companies in my career and you can definitely get support contracts for engineering support of RHEL, Ubuntu, SUSE, etc. That isn’t the issue. The issue is that there are a lot of system administrators with “15 years experience in Linux” that have no real experience in Linux. They have experience googling for guides and tutorials while having cobbled together documents of doing various things without understanding what they are really doing.

    I can’t tell you how many times I’ve seen an enterprise patch their Linux solutions (if they patched them at all with some ridiculous rubberstamped PO&AM) manually without deploying a repo and updating the repo treating it as you would a WSUS. Hell, I’m pleasantly surprised if I see them joined to a Windows domain (a few times) or an LDAP (once but they didn’t have a trust with the Domain Forest or use sudoer rules…sigh).

    Blackmist ,

    I’ve had my PC shut down for updates three times now, while using it as a Jellyfin server from another room. And I’ve only been using it for this purpose for six months or so.

    I can’t imagine running anything critical on it.

    ccdfa ,

    Windows server, the OS, runs differently from desktop windows. So if you’re using desktop windows and expecting it to run like a server, well, that’s on you. However, I ran windows server 2016 and then 2019 for quite a few years just doing general homelab stuff and it is really a pain compared to Linux which I switched to on my server about a year ago. Server stuff is just way easier on Linux in my experience.

    conciselyverbose ,

    It doesn’t have to, though. Linux manages to do both just fine, with relatively minor compromises.

    Expecting an OS to handle keeping software running is not a big ask.

    sugar_in_your_tea ,

    Yup, I use Linux to run a Jellyfin server, as well as a few others things. The only problem is that the CPU I’m using (Ryzen 1st gen) will crash every couple weeks or so (known hardware fault, I never bothered to RMA), but that’s honestly not that bad since I can just walk over and restart it. Before that, it ran happily on an old Phenom II from 2009 for something like 10 years (old PC), and I mostly replaced it because the Ryzen uses a bit less electricity (enough that I used to turn the old PC off at night; this one runs 24/7 as is way more convenient).

    So aside from this hardware issue, Linux has been extremely solid. I have a VPS that tunnels traffic into my Jellyfin and other services from outside, and it pretty much never goes down (I guess the host reboots it once a year or something for hardware maintenance). I run updates when I want to (when I remember, which is about monthly), and it only goes down for like 30 sec to reboot after updates are applied.

    So yeah, Linux FTW, once it’s set up, it just runs.

    0xD ,

    Well with your level of expertise you should probably not be running anything, to be honest :)

    ji17br ,

    Wow dude you’re so cool. I bet that made you feel so superior. Everyone on here thinks you are so badass.

    Rinox ,

    I dunno, but doesn’t like a quarter of the internet kinda run on Azure?

    atocci ,

    And 60% of Azure is running Linux lol

    Rinox ,

    I guess Spotify was running on the other 40%, as many other services

    pewgar_seemsimandroid ,

    so 40% of azure crashes a quarter of the internet…

    unconfirmedsourcesDOTgov ,

    Where did you think Microsoft was getting all (hyperbole) of their money from?

    boaratio ,

    CrowdStrike: It’s Friday, let’s throw it over the wall to production. See you all on Monday!

    jayandp ,

    https://sh.itjust.works/pictrs/image/ce10cdca-116b-4061-b6c3-3f49780c0551.jpeg

    ^^so ^^hard ^^picking ^^which ^^meme ^^to ^^use

    frezik ,

    When your push to prod on Friday causes a small but measurable drop in global GDP.

    lightnsfw ,

    We did it guys! We moved fast AND broke things!

    Pudutr0n ,
    @Pudutr0n@feddit.cl avatar

    This is a better article. It’s a CrowdStrike issue with an update (security software)

    AnarchistArtificer ,

    I agree that’s a better article, thanks for sharing

    Damage ,

    The thought of a local computer being unable to boot because some remote server somewhere is unavailable makes me laugh and sad at the same time.

    Munkisquisher ,

    A remote server that you pay some serious money to that pushes a garbage driver that prevents yours from booting

    lanolinoil ,
    @lanolinoil@lemmy.world avatar

    yeah so you can’t get Chinese government spyware installed.

    Passerby6497 ,

    Not only does it (possibly) prevent booting, but it will also bsod it first so you’ll have to see how lucky you get.

    Goddamn I hate crowdstrike. Between this and them fucking up and letting malware back into a system, I have nothing nice to say about them.

    Cryophilia , (edited )

    It’s bsod on boot

    And anything encrypted with bitlocker can’t even go into safe mode to fix it

    Passerby6497 ,

    It doesn’t consistently bsod on boot, about half of affected machines did in our environment, but all of them did experience a bsod while running. A good amount of ours just took the bad update, bsod’d and came back up.

    rxxrc OP ,

    I don’t think that’s what’s happening here. As far as I know it’s an issue with a driver installed on the computers, not with anything trying to reach out to an external server. If that were the case you’d expect it to fail to boot any time you don’t have an Internet connection.

    Windows is bad but it’s not that bad yet.

    __init__ ,

    It’s just a fun coincidence that the azure outage was around the same time.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •