There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

rottingleaf ,

Servers on Windows? Even domain controllers can be Linux-based.

OsrsNeedsF2P ,

Old servers. Also Crowdstrike took down Linux servers a few years ago.

AllOutOfBubbleGum ,

I’m a long-time Samba fan, but even I wouldn’t run them as DCs in a production environment.

rottingleaf ,

I meant sssd, that I’ve seen working as a DC, if my memory serves me right. But I’ve never been the person setting up a domain controller, so.

EDIT: No, it doesn’t serve me right, from quick googling sssd can’t into DC’s. Kurwa.

boaratio ,

CrowdStrike: It’s Friday, let’s throw it over the wall to production. See you all on Monday!

jayandp ,

https://sh.itjust.works/pictrs/image/ce10cdca-116b-4061-b6c3-3f49780c0551.jpeg

^^so ^^hard ^^picking ^^which ^^meme ^^to ^^use

verity_kindle ,

Good choice, tho. Is the image AI?

jayandp ,

Not sure, I didn’t make it. Just part of my collection.

frezik ,

When your push to prod on Friday causes a small but measurable drop in global GDP.

lagomorphlecture ,

Definitely not small, our website is down so we can’t do any business and we’re a huge company. Multiply that by all the companies that are down, lost time on projects, time to get caught up once it’s fixed, it’ll be a huge number in the end.

LustyArgonianMana ,
@LustyArgonianMana@lemmy.world avatar

I know people who work at major corporations who said they were down for a bit, it’s pretty huge.

frezik ,

One or two days lost, even if it was 100% of the GDP for those days, would still be less than 1% of GDP for the year.

LustyArgonianMana ,
@LustyArgonianMana@lemmy.world avatar

Actually, it may have helped slow climate change a little

iamtrashman1312 ,

The earth is healing 🙏

For part of today

lightnsfw ,

We did it guys! We moved fast AND broke things!

simplejack ,
@simplejack@lemmy.world avatar

They did it on Thursday. All of SFO was BSODed for me when I got off a plane at SFO Thursday night.

StaySquared ,

Been at work since 5AM… finally finished deleting the C-00000291*.sys file in CrowdStrike directory.

182 machines total. Thankfully the process in of itself takes about 2-3 minutes. For virtual machines, it’s a bit of a pain, at least in this org.

lmao I feel kinda bad for those companies that have 10k+ endpoints to do this to. Eff… that. Lot’s of immediate short term contract hires for that, I imagine.

douglasg14b ,
@douglasg14b@lemmy.world avatar

How do you deal with places with thousands of remote endpoints??

StaySquared ,

That’s one of those situations where they need to immediately hire local contractors to those remote sites. This outage literally requires touching the equipment. lol

I’d even say, fly out each individual team member to those sites… but even the airports are down.

ytg ,

>Make a kernel-level antivirus
>Make it proprietary
>Don’t test updates… for some reason??

CircuitSpells ,

I mean I know it’s easy to be critical but this was my exact thought, how the hell didn’t they catch this in testing?

grabyourmotherskeys ,

I have had numerous managers tell me there was no time for QA in my storied career. Or documentation. Or backups. Or redundancy. And so on.

kandoh ,

Move fast and break things! We need things NOW NOW NOW!

Quill7513 ,

Push that into the technical debt. Then afterwards never pay off the technical debt

user224 ,
@user224@lemmy.sdf.org avatar

Just always make sure you have some evidence of them telling you to skip these.

grabyourmotherskeys ,

There’s a reason I still use lots of email in the age of IM. Permanent records, please. I will email a record of in person convos or chats on stuff like this. I do it politely and professionally, but I do it.

Voroxpete ,

Completely justified reaction. A lot of the time tech companies and IT staff get shit for stuff that, in practice, can be really hard to detect before it happens. There are all kinds of issues that can arise in production that you just can’t test for.

But this… This has no justification. A issue this immediate, this widespread, would have instantly been caught with even the most basic of testing. The fact that it wasn’t raises massive questions about the safety and security of Crowdstrike’s internal processes.

Spaceinv8er ,

So that’s why my work laptop is down for the count today. I’m even getting that same error as the thumbnail picture

CanadaPlus ,

Yep, this is the stupid timeline. Y2K happening to to the nuances of calendar systems might have sounded dumb at the time, but it doesn’t now. Y2K happening because of some unknown contractor’s YOLO Friday update definitely is.

dhork ,

I picked the right week to be on PTO hahaha

thearch ,

Irrelevant but I keep reading “crowd strike” as “counter strike” and it’s really messing with me

retrospectology ,
@retrospectology@lemmy.world avatar

This is why you create restore points if using windows.

Raxiel ,

A lot of people I work with were affected, I wasn’t one of them. I had assumed it was because I put my machine to sleep yesterday (and every other day this week) and just woke it up after booting it. I assumed it was an on startup thing and that’s why I didn’t have it.

Our IT provider already broke EVERYTHING earlier this month when they remote installed" Nexthink Collector" which forced a 30+ minute CHKDSK on every boot for EVERYONE, until they rolled out a fix (which they were at least able to do remotely), and I didn’t want to have to deal with that the week before I go in leave.

But it sounds like it even happened to running systems so now I don’t know why I wasn’t affected, unless it’s a windows 10 only thing?

Our IT have had some grief lately, but at least they specified Intel 12th gen on our latest CAD machines, rather than 13th or 14th, so they’ve got at least one win.

wizardbeard ,

Your computer was likely not powered on during the time window between the fucked update pushing out and when they stopped pushing it out.

Happywop ,

It’s Russia, or Iran or China or even our “ally” Saudi Arabia. So really, it’s time to reset the clock to pre 1989. Cut Russia and China off completely, no investment, no internet, no students no tourist nothing. These people mean and are continually doing us harm and we still plod along and some unscrupulous types become agents for personal profit. Enough.

Eezyville ,
@Eezyville@sh.itjust.works avatar

It was a bad update…

Quill7513 ,

Let’s ignore that it was an American company taking down EVERYONE’S stuff, I guess

Treczoks ,

I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.

wizardbeard ,

It’s not a flaw with Windows causing this.

The issue is with a widely used third party security software that installs as a kernel level driver. It had an auto update that causes bluescreening moments after booting into the OS.

This same software is available for Linux and Mac, and had similar issues with specific Linux distros a month ago. It just didn’t get reported on because it didn’t have as wide of an impact.

aaaaace ,

www.theregister.comhas a series of articles on what’s going on technically.

Latest advice…

There is a faulty channel file, so not quite an update. There is a workaround…

  1. Boot Windows into Safe Mode or WRE.
  2. Go to C:\Windows\System32\drivers\CrowdStrike
  3. Locate and delete file matching “C-00000291*.sys”
  4. Boot normally.
scripthook ,
@scripthook@lemmy.world avatar

crowdstrike sent a corrupt file with a software update for windows servers. this caused a blue screen of death on all the windows servers globally for crowdstrike clients causing that blue screen of death. even people in my company. luckily i shut off my computer at the end of the day and missed the update. It’s not an OTA fix. they have to go into every data center and manually fix all the computer servers. some of these severs have encryption. I see a very big lawsuit coming…

chris ,
@chris@l.roofo.cc avatar

I don’t see how they can recover from that. They will get lawsuits from all around the world.

AWittyUsername ,

I’m never financially recovering from this.

AuntieFreeze ,

Webroot had something similar ish earlier this year. Such a pain.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •