There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

veam ,

oh joy. can’t wait to have to fix this for all of our clients today…

iturnedintoanewt ,
@iturnedintoanewt@lemm.ee avatar

You have no idea how much fun its being.

Passerby6497 ,

I’m so tired of all the fun…

kadotux , (edited )

Here’s the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching “C-00000291*.sys” 4)Boot the system normally

StV2 ,

It’s disappointing that the fix is so easy to perform and yet it’ll almost certainly keep a lot of infrastructure down for hours because a majority of people seem too scared to try to fix anything on their own machine (or aren’t trusted to so they can’t even if they know how)

r00ty Admin ,
r00ty avatar

It might not even be that. A lot of places have many servers (and even more virtual servers) running crowdstrike. Some places also seem to have it on endpoints too.

That's a lot of machines to manually fix.

HaleHirsute ,

They also gotta get the fix through a trusted channel and not randomly on the internet. (No offense to the person that gave the info, it’s maybe correct but you never know)

kadotux ,

Yeah, and it’s unknown if CS is active after the workaround or not (source: hackernews commentator)

letsgo ,

True, but knowing what the fix might be means you can Google it and see what comes back. It was on StackOverflow for example, but at the time of this comment has been taken offline for moderation - whatever that means.

huginn ,

Yeah and a lot of corpo VPNs are gonna be down from this too.

NaibofTabr ,

This sort of fix might not be accessible to a lot of employees who don’t have admin access on their company laptops, and if the laptop can’t be accessed remotely by IT then the options are very limited. Trying to walk a lot of nontechnical users through this over the phone won’t go very well.

AccountMaker ,

Yup, that’s me. We booted into safe mode, tried navigating into the CrowdStrike folder and boom: permission denied.

Munkisquisher ,

And people need to travel to remote machines to do this in person

thehatfox ,
@thehatfox@lemmy.world avatar

Might seem easy to someone with a technical background. But the last thing businesses want to be doing is telling average end users to boot into safe mode and start deleting system files.

If that started happening en masse we would quickly end up with far more problems than we started with. Plenty of users would end up deleting system32 entirely or something else equally damaging.

CaptainBasculin ,

A driver failure, yeesh. It always sucks to deal with it.

cheeseburger ,
@cheeseburger@lemmy.ca avatar

I’m on a bridge still while we wait for Bitlocker recovery keys, so we can actually boot into safemode, but the Bitkocker key server is down as well…

gnutrino ,

Gonna be a nice test of proper backups and disaster recovery protocols for some organisations

huginn ,

Chaos Monkey test

WagnasT ,

Man, it sure would suck if you could still get to safe mode from pressing f8. Can you imagine how terrible that’d be?

a_postmodern_hat ,

You hold down Shift while restarting or booting and you get a recovery menu. I don’t know why they changed this behaviour.

Damage ,

The thought of a local computer being unable to boot because some remote server somewhere is unavailable makes me laugh and sad at the same time.

Munkisquisher ,

A remote server that you pay some serious money to that pushes a garbage driver that prevents yours from booting

lanolinoil ,
@lanolinoil@lemmy.world avatar

yeah so you can’t get Chinese government spyware installed.

Passerby6497 ,

Not only does it (possibly) prevent booting, but it will also bsod it first so you’ll have to see how lucky you get.

Goddamn I hate crowdstrike. Between this and them fucking up and letting malware back into a system, I have nothing nice to say about them.

rxxrc OP ,

I don’t think that’s what’s happening here. As far as I know it’s an issue with a driver installed on the computers, not with anything trying to reach out to an external server. If that were the case you’d expect it to fail to boot any time you don’t have an Internet connection.

Windows is bad but it’s not that bad yet.

umami_wasbi ,

No one bother to test before deploying to all machines? Nice move.

pufferfisherpowder ,

YOLO 🚀🙈

huginn ,

This outage is probably costing a significant portion of Crowd strike’s market cap. They’re an 80 billion dollar company but this is a multibillion outage.

Someone’s getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.

Nachorella ,

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I’ll be working from home for a few days before anything is fixed.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

One possible fix is to delete a particular file while booting in safe mode. But then they’ll need to fix each system manually. My company encrypts the disks as well so it’s going to be a even bigger pain (for them). I’m just happy my weekend started early.

Valmond ,

You have ta have access to boot in safe mode too, I guess I can’t on my work pc for example.

What a shitty workaround & might crowd strike burn in hell lol

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

Enjoy your weekend unless you are in IT

rozodru ,
@rozodru@lemmy.ca avatar

that would only work for like low level people’s laptops. apparently if your role requires a more secure machine you also have to deal with bitlocker whiiiiiiich is tied in with crowdstrike soooooo no dice.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

Yeah that would be case in most laptops. So if bitlovker is involved as well what could be the possible fix.

rozodru ,
@rozodru@lemmy.ca avatar

I mean if your IT was smart, IF they were smart, they would have the bitlocker decryptions backed up on like a usb or something. IF you need to access the decryption via microsoft then you’re apparently borked for now.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

That would be funny

sasquash ,

never do updates on a Friday.

rozodru ,
@rozodru@lemmy.ca avatar

yeah someone fucked up here. I mean I know you’re joking but I’ve been in tech for like 20+ years at this point and it was always, always, ALWAYS, drilled into me to never do updates on Friday, never roll anything out to production on Friday. Fridays were generally meant for code reviews, refactoring in test, work on personal projects, raid the company fridge for beer, play CS at the office, whatever just don’t push anything live or update anything.

And especially now the work week has slimmed down where no one works on Friday anymore so you 100% don’t roll anything out, hell it’s getting to the point now where you just don’t roll anything out on a Thursday afternoon.

0x0 ,

And especially now the work week has slimmed down where no one works on Friday anymore

Excuse me, what now? I didn’t get that memo.

meanmon13 ,

Yeah it’s great :-) 4 10hr shifts and every weekend is a 3 day weekend

rozodru ,
@rozodru@lemmy.ca avatar

sorry :( yeah I, at most, do 3 days in the office now. Fridays are a day off and Mondays mostly everyone just works from home if at all. downtown Toronto on Mondays and Fridays is pretty much dead.

Blackmist ,

Yep, anything done on Friday can enter the world on a Monday.

I don’t really have any plans most weekends, but I sure as shit don’t plan on spending it fixing Friday’s fuckups.

spyd3r ,
@spyd3r@sh.itjust.works avatar

Never update unless something is broken.

Toribor ,
@Toribor@corndog.social avatar

This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.

Ookami38 ,

I think vulnerabilities found count as “something broken” and chap you replied to simply did not think that far ahead hahah

huginn ,

For real - A cyber security company should basically always be pushing out updates.

iknowitwheniseeit ,

BTW, I use Arch.

Passerby6497 ,

That’s advice so smart you’re guaranteed to have massive security holes.

robocall ,
@robocall@lemmy.world avatar

Buy the dip!

jecht360 ,
@jecht360@lemmy.world avatar

More like short them. This is going to be devastating for their business. I could see them losing tons of customers.

victorz ,

If these affected systems are boot looping, how will they be fixed? Reinstall?

bevan ,

It is possible to edit a folder name in windows drivers. But for IT departments that could be more work than a reimage

Sylence ,
@Sylence@lemmy.dbzer0.com avatar

There is a fix people have found which requires manual booting into safe mode and removal of a file causing the BSODs. No clue if/how they are going to implement a fix remotely when the affected machines can’t even boot.

letsgo ,

Probably have to go old-skool and actually be at the machine.

Freefall ,

Exactly, and super fun when all your systems are remote!!!

VieuxQueb ,
@VieuxQueb@lemmy.ca avatar

And hope you are not using BitLocker cause then you are screwed since BitLocker is tied to CS.

ChefKalash ,

Do you have any source on this?

Sylence ,
@Sylence@lemmy.dbzer0.com avatar

If you have an account you can view the support thread here: …crowdstrike.com/…/Tech-Alert-Windows-crashes-rel…

Workaround Steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. Boot the host normally.
AnUnusualRelic ,
@AnUnusualRelic@lemmy.world avatar

An offline server is a secure server!

bdonvr ,

The amount of servers running Windows out there is depressing to me

franklin ,
@franklin@lemmy.world avatar

The four multinational corporations I worked at were almost entirely Windows servers with the exception of vendor specific stuff running Linux. Companies REALLY want that support clause in their infrastructure agreement.

Blackmist ,

I’ve had my PC shut down for updates three times now, while using it as a Jellyfin server from another room. And I’ve only been using it for this purpose for six months or so.

I can’t imagine running anything critical on it.

ari_verse ,

A few years ago when my org got the ask to deploy the CS agent in linux production servers and I also saw it getting deployed in thousands of windows and mac desktops all across, the first thought that came to mind was “massive single point of failure and security threat”, as we were putting all the trust in a single relatively small company that will (has?) become the favorite target of all the bad actors across the planet. How long before it gets into trouble, either because if it’s own doing or due to others?

I guess that we now know

SupraMario ,

No bad actors did this, and security goes in fads. Crowdstrike is king right now, just as McAfee/Trellix was in the past. If you want to run around without edr/xdr software be my guest.

ansiz ,

All of the security vendors do it over enough time. McAfee used to be the king of them.

zdnet.com/…/defective-mcafee-update-causes-worldw…

bleepingcomputer.com/…/trend-micro-antivirus-modi…

techradar.com/…/microsoft-releases-fix-for-botche…

ari_verse ,

It’s a fair point but I would rather diversify and also use something that is open / less opaque

richtellyard ,

This is going to be a Big Deal for a whole lot of people. I don’t know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.

RegalPotoo ,
@RegalPotoo@lemmy.world avatar

Big chunk of New Zealands banks apparently run it, cos 3 of the big ones can’t do credit card transactions right now

oderus ,

It was mayhem at PakNSave a bit ago.

emmanuel_car ,

In my experience it’s always mayhem at PakNSave.

deadbeef79000 ,

If anything, it’s probably calmed P’n’S down a bit…

index ,

cos 3 of the big ones can’t do credit card transactions right now

Bitcoin still up and running perhaps people can use that

Sylence ,
@Sylence@lemmy.dbzer0.com avatar

Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!

iknowitwheniseeit ,

My flight was canceled. Luckily that was a partner airline. My actual airline rebooked me on a direct flight. Leaves 3 hours later and arrives earlier. Lower carbon footprint. So, except that I’m standing in queue so someone can inspect my documents it’s basically a win for me. 😆

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •