There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

autotldr Bot ,

This is the best summary I could come up with:

There are reports of IT outages affecting major institutions in Australia and internationally.

The ABC is experiencing a major network outage, along with several other media outlets.

Crowd-sourced website Downdetector is listing outages for Foxtel, National Australia Bank and Bendigo Bank.

Follow our live blog as we bring you the latest updates.

The original article contains 52 words, the summary contains 52 words. Saved 0%. I’m a bot and I’m open source!

dorythefish ,

The original article contains 52 words, the summary contains 52 words. Saved 0%. Good bot!

upside431 ,

Interesting day

richtellyard ,

This is going to be a Big Deal for a whole lot of people. I don’t know all the companies and industries that use Crowdstrike but I might guess it will result in airline delays, banking outages, and hospital computer systems failing. Hopefully nobody gets hurt because of it.

RegalPotoo ,
@RegalPotoo@lemmy.world avatar

Big chunk of New Zealands banks apparently run it, cos 3 of the big ones can’t do credit card transactions right now

oderus ,

It was mayhem at PakNSave a bit ago.

emmanuel_car ,

In my experience it’s always mayhem at PakNSave.

deadbeef79000 ,

If anything, it’s probably calmed P’n’S down a bit…

index ,

cos 3 of the big ones can’t do credit card transactions right now

Bitcoin still up and running perhaps people can use that

I_Miss_Daniel ,

Bitcoin Cash maybe. Didn’t they bork Bitcoin (Core) so you have to wait for confirmations in the next block?

whotookkarl ,
@whotookkarl@lemmy.world avatar

Several 911 systems were affected or completely down too

jedibob5 ,

Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half…

misk ,
@misk@sopuli.xyz avatar

My work PC is affected. Nice!

wreckedcarzz ,
@wreckedcarzz@lemmy.world avatar

Plot twist: you’re head of IT

R00bot ,
@R00bot@lemmy.blahaj.zone avatar

Same! Got to log off early 😎

Munkisquisher ,

Dammit, hit us at 5pm on Friday in NZ

BigRedUndead ,

4:00PM here in Aus. Absolutely perfect for an early Friday knockoff.

Magnolia_ ,

Noice!

solrize ,

Xfinity H&I network it down so I can’t watch Star Trek. I get an error msg connection failure. Other channels work though.

Sylence ,
@Sylence@lemmy.dbzer0.com avatar

Yep, stuck at the airport currently. All flights grounded. All major grocery store chains and banks also impacted. Bad day to be a crowdstrike employee!

iknowitwheniseeit ,

My flight was canceled. Luckily that was a partner airline. My actual airline rebooked me on a direct flight. Leaves 3 hours later and arrives earlier. Lower carbon footprint. So, except that I’m standing in queue so someone can inspect my documents it’s basically a win for me. 😆

victorz ,

If these affected systems are boot looping, how will they be fixed? Reinstall?

bevan ,

It is possible to edit a folder name in windows drivers. But for IT departments that could be more work than a reimage

Passerby6497 , (edited )

Having had to fix >100 machines today, I’m not sure how a reimage would be less work. Restoring from backups maybe, but reimage and reconfig is so painful

EncryptKeeper ,

It’s just one file to delete.

Sylence ,
@Sylence@lemmy.dbzer0.com avatar

There is a fix people have found which requires manual booting into safe mode and removal of a file causing the BSODs. No clue if/how they are going to implement a fix remotely when the affected machines can’t even boot.

letsgo ,

Probably have to go old-skool and actually be at the machine.

Freefall ,

Exactly, and super fun when all your systems are remote!!!

Passerby6497 ,

It’s not super awful as long as everything is virtual. It’s annoying, but not painful like it would be for physical systems.

Really don’t envy physical/desk side support folks today…

VieuxQueb ,
@VieuxQueb@lemmy.ca avatar

And hope you are not using BitLocker cause then you are screwed since BitLocker is tied to CS.

EncryptKeeper ,

You just need console access. Which if any of the affected servers are VMs, you’ll have.

CanadaPlus ,

Yes, VMs will be more manageable.

ChefKalash ,

Do you have any source on this?

Sylence ,
@Sylence@lemmy.dbzer0.com avatar

If you have an account you can view the support thread here: …crowdstrike.com/…/Tech-Alert-Windows-crashes-rel…

Workaround Steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. Boot the host normally.
Passerby6497 ,

I can confirm it works after applying it to >100 servers :/

victorz ,

Nice work, friend. 🤝 [back pat]

CanadaPlus ,

It seems like it’s in like half of the news stories.

Pudutr0n ,
@Pudutr0n@feddit.cl avatar

This is a better article. It’s a CrowdStrike issue with an update (security software)

AnarchistArtificer ,

I agree that’s a better article, thanks for sharing

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

One possible fix is to delete a particular file while booting in safe mode. But then they’ll need to fix each system manually. My company encrypts the disks as well so it’s going to be a even bigger pain (for them). I’m just happy my weekend started early.

Valmond ,

You have ta have access to boot in safe mode too, I guess I can’t on my work pc for example.

What a shitty workaround & might crowd strike burn in hell lol

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

Enjoy your weekend unless you are in IT

rozodru ,
@rozodru@lemmy.ca avatar

that would only work for like low level people’s laptops. apparently if your role requires a more secure machine you also have to deal with bitlocker whiiiiiiich is tied in with crowdstrike soooooo no dice.

alphacyberranger , (edited )
@alphacyberranger@sh.itjust.works avatar

Yeah that would be case in most laptops. So if bitlocker is involved as well what could be the possible fix.

rozodru ,
@rozodru@lemmy.ca avatar

I mean if your IT was smart, IF they were smart, they would have the bitlocker decryptions backed up on like a usb or something. IF you need to access the decryption via microsoft then you’re apparently borked for now.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

That would be funny

catloaf ,

Yeah, most large orgs have a key server, or back up to AD. If you don’t have that, and no recovery key, you’re fucked and that data is gone.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

What if that is running crowdstrike?

catloaf ,

I’ll give you one guess.

(That’s why when I was in charge of that stuff at one company, I had that recovery key printed out and kept separately in a lockbox.)

kadotux , (edited )

Here’s the fix: (or rather workaround, released by CrowdStrike) 1)Boot to safe mode/recovery 2)Go to C:\Windows\System32\drivers\CrowdStrike 3)Delete the file matching “C-00000291*.sys” 4)Boot the system normally

StV2 ,

It’s disappointing that the fix is so easy to perform and yet it’ll almost certainly keep a lot of infrastructure down for hours because a majority of people seem too scared to try to fix anything on their own machine (or aren’t trusted to so they can’t even if they know how)

r00ty Admin ,
r00ty avatar

It might not even be that. A lot of places have many servers (and even more virtual servers) running crowdstrike. Some places also seem to have it on endpoints too.

That's a lot of machines to manually fix.

HaleHirsute ,

They also gotta get the fix through a trusted channel and not randomly on the internet. (No offense to the person that gave the info, it’s maybe correct but you never know)

kadotux ,

Yeah, and it’s unknown if CS is active after the workaround or not (source: hackernews commentator)

letsgo ,

True, but knowing what the fix might be means you can Google it and see what comes back. It was on StackOverflow for example, but at the time of this comment has been taken offline for moderation - whatever that means.

huginn ,

Yeah and a lot of corpo VPNs are gonna be down from this too.

ColeSloth ,

Meh. Even if it bricked crowdstrike instead of helping, you can just restore the file you deleted. A file in that folder can’t brick a windows system.

NaibofTabr ,

This sort of fix might not be accessible to a lot of employees who don’t have admin access on their company laptops, and if the laptop can’t be accessed remotely by IT then the options are very limited. Trying to walk a lot of nontechnical users through this over the phone won’t go very well.

AccountMaker ,

Yup, that’s me. We booted into safe mode, tried navigating into the CrowdStrike folder and boom: permission denied.

Cryophilia ,

Half our shit can’t even boot into safe mode because it’s encrypted and we don’t have the keys rofl

Munkisquisher ,

And people need to travel to remote machines to do this in person

thehatfox ,
@thehatfox@lemmy.world avatar

Might seem easy to someone with a technical background. But the last thing businesses want to be doing is telling average end users to boot into safe mode and start deleting system files.

If that started happening en masse we would quickly end up with far more problems than we started with. Plenty of users would end up deleting system32 entirely or something else equally damaging.

Ookami38 ,

I do IT for some stores. My team lead briefly suggested having store managers try to do this fix. I HARD vetoed that. That’s only going to do more damage.

nyarla ,

Yes but the recovery menu may have been configured to ask for administrative credentials, to prevent unwanted access to the computer, and then fixing the problem would take way longer.

CaptainBasculin ,

A driver failure, yeesh. It always sucks to deal with it.

cheeseburger ,
@cheeseburger@lemmy.ca avatar

I’m on a bridge still while we wait for Bitlocker recovery keys, so we can actually boot into safemode, but the Bitkocker key server is down as well…

gnutrino ,

Gonna be a nice test of proper backups and disaster recovery protocols for some organisations

huginn ,

Chaos Monkey test

WagnasT ,

Man, it sure would suck if you could still get to safe mode from pressing f8. Can you imagine how terrible that’d be?

a_postmodern_hat ,

You hold down Shift while restarting or booting and you get a recovery menu. I don’t know why they changed this behaviour.

Ookami38 ,

That was the dumbest thing to learn this morning.

resin85 ,

Not that easy when it’s a fleet of servers in multiple remote data centers. Lots of IT folks will be spending their weekend sitting in data center cages.

moe90 ,
@moe90@feddit.nl avatar

don’t rely on one desktop OS too much. diversity is the best.

aniki ,

Dont rely on corpo trash at al.

UncleArthur ,

Annoyingly, my laptop seems to be working perfectly.

Valmond ,

That’s the burden when you run Arch, right?

Damage ,

lol he said it’s working

jedibob5 ,

Reading into the updates some more… I’m starting to think this might just destroy CloudStrike as a company altogether. Between the mountain of lawsuits almost certainly incoming and the total destruction of any public trust in the company, I don’t see how they survive this. Just absolutely catastrophic on all fronts.

NaibofTabr ,

If all the computers stuck in boot loop can’t be recovered… yeah, that’s a lot of cost for a lot of businesses. Add to that all the immediate impact of missed flights and who knows what happening at the hospitals. Nightmare scenario if you’re responsible for it.

This sort of thing is exactly why you push updates to groups in stages, not to everything all at once.

rxxrc OP ,

Looks like the laptops are able to be recovered with a bit of finagling, so fortunately they haven’t bricked everything.

And yeah staged updates or even just… some testing? Not sure how this one slipped through.

dactylotheca ,
@dactylotheca@suppo.fi avatar

Not sure how this one slipped through.

I’d bet my ass this was caused by terrible practices brought on by suits demanding more “efficient” releases.

“Why do we do so much testing before releases? Have we ever had any problems before? We’re wasting so much time that I might not even be able to buy another yacht this year”

GoofSchmoofer ,
@GoofSchmoofer@lemmy.world avatar

At least nothing like this happens in the airline industry

dactylotheca ,
@dactylotheca@suppo.fi avatar

Certainly not! Or other industries for that matter. It’s a good thing executives everywhere aren’t just concentrating on squeezing the maximum amount of money out of their companies and funneling it to themselves and their buddies on the board.

Sure, let’s “rightsize” the company by firing 20% of our workforce (but not management!) and raise prices 30%, and demand that the remaining employees maintain productivity at the level it used to be before we fucked things up. Oh and no raises for the plebs, we can’t afford it. Maybe a pizza party? One slice per employee though.

Munkisquisher ,

Yeah saw that several steel mills have been bricked by this, that’s months and millions to restart

gazter ,

Got a link? I find it hard to believe that a process like that would stop because of a few windows machines not booting.

TheBat ,
@TheBat@lemmy.world avatar

a few windows machines with controller application installed

That’s the real kicker.

drspod ,

Those machines should be airgapped and no need to run Crowdstrike on them. If the process controller machines of a steel mill are connected to the internet and installing auto updates then there really is no hope for this world.

TheBat ,
@TheBat@lemmy.world avatar

But daddy microshoft says i gotta connect the system to the internet uwu

IsThisAnAI ,

What lawsuits do you think are going to happen?

Nachorella ,

They can have all the clauses they like but pulling something like this off requires a certain amount of gross negligence that they can almost certainly be held liable for.

IsThisAnAI ,

Whatever you say my man. It’s not like they go through very specific SLA conversations and negotiations to cover this or anything like that.

Cryophilia ,

Forget lawsuits, they’re going to be in front of congress for this one

IsThisAnAI ,

For what? At best it would be a hearing on the challenges of national security with industry.

RegalPotoo ,
@RegalPotoo@lemmy.world avatar

Agreed, this will probably kill them over the next few years unless they can really magic up something.

They probably don’t get sued - their contracts will have indemnity clauses against exactly this kind of thing, so unless they seriously misrepresented what their product does, this probably isn’t a contract breach.

If you are running crowdstrike, it’s probably because you have some regulatory obligations and an auditor to appease - you aren’t going to be able to just turn it off overnight, but I’m sure there are going to be some pretty awkward meetings when it comes to contract renewals in the next year, and I can’t imagine them seeing much growth

Skydancer ,

Nah. This has happened with every major corporate antivirus product. Multiple times. And the top IT people advising on purchasing decisions know this.

SupraMario ,

Yep. This is just uninformed people thinking this doesn’t happen. It’s been happening since av was born. It’s not new and this will not kill CS they’re still king.

jedibob5 ,

Don’t most indemnity clauses have exceptions for gross negligence? Pushing out an update this destructive without it getting caught by any quality control checks sure seems grossly negligent.

Wooki ,

Testing is production will do that

TheBat ,
@TheBat@lemmy.world avatar

Not everyone is fortunate enough to have a seperate testing environment, you know? Manglement has to cut cost somewhere.

Blisterexe ,

Manglement is the good term lmao

ThrowawaySobriquet ,

I think you’re on the nose, here. I laughed at the headline, but the more I read the more I see how fucked they are. Airlines. Industrial plants. Fucking governments. This one is big in a way that will likely get used as a case study.

Cryophilia ,

The London Stock Exchange went down. They’re fukd.

rozodru ,
@rozodru@lemmy.ca avatar

It’s just amatuer hour across the board. Were they testing in production? no code review or even a peer review? they roll out for a Friday? It’s like basic level start up company “here’s what not to do” type shit that a junior dev fresh out of university would know. It’s like “explain to the project manager with crayons why you shouldn’t do this” type of shit.

It just boggles my mind that if you’re rolling out an update to production that there was clearly no testing. There was no review of code cause experts are saying it was the result of poorly written code.

Regardless if you’re low level security then apparently you can just boot into safe and rename the crowdstrike folder and that should fix it. higher level not so much cause you’re likely on bitlocker which…yeah don’t get me started no that bullshit.

regardless I called out of work today. no point. it’s friday, generally nothing gets done on fridays (cause we know better) and especially today nothing is going to get done.

Revan343 ,

explain to the project manager with crayons why you shouldn’t do this

Can’t; the project manager ate all the crayons

candybrie ,

Why is it bad to do on a Friday? Based on your last paragraph, I would have thought Friday is probably the best week day to do it.

Lightor ,

Most companies, money included, try to roll out updates during the middle of start of a week. That way if there are issues the full team is available to address them.

Bell ,

Don’t we blame MS at least as much? How does MS let an update like this push through their Windows Update system? How does an application update make the whole OS unable to boot? Blue screens on Windows have been around for decades, why don’t we have a better recovery system?

sandalbucket ,

Crowdstrike runs at ring 0, effectively as part of the kernel. Like a device driver. There are no safeguards at that level. Extreme testing and diligence is required, because these are the consequences for getting it wrong. This is entirely on crowdstrike.

ililiililiililiilili ,

My dad needed a CT scan this evening and the local ER’s system for reading the images was down. So they sent him via ambulance to a different hospital 40 miles away. Now I’m reading tonight that CrowdStrike may be to blame.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •