There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

foremanguy92_ ,

Bahaha 😂😂 continue using proprietary software, that’s all you are going to get in addition to privacy issues… Switch to Linux.

Cornelius_Wangenheim ,

A bunch of shitty sysadmins/cybersec people just learned why you don’t blindly deploy new updates to production without testing them first.

I’ve used crowd strike before. It has support for deploying version N to a pilot group, N-1 to the test environment and N-2 to production.

invisiblegorilla ,

Ironic. They did what they are there to protect against. Fucking up everyone’s shit

Telorand ,

Maybe centralizing everything onto one company’s shoulders wasn’t such a great idea after all…

TheReturnOfPEB ,

This is a a ruse to make Work From Home end.

exanime ,

???

How would this have been prevented by working on site?

sudo ,

A crowd strike, you could call it

Weax ,

This is just blatantly incorrect - 99% of these outages are going to be fixed remotely.

yournamehere ,

best day ever. the digitards get a wakeup call. how often have been lectured by imbeciles how great whatever dumbo closed source is. “i need photoshop”, “windows powershell and i get work done”, “azure and onedrive and teams…best shit ever”, " go use NT, nobody will use a GNU".

yeah well, i hope every windows user would be kept of the interwebs for a year and mac users just burn in hell right away. lazy scum that justifies shitting on society for their own comfort. while everyone needs a drivers license, dumb fucking parents give tiktok to their kids…idiocracy will have a great election this winter.

ninekeysdown ,
@ninekeysdown@lemmy.world avatar

So when’s the last time you touched some grass? It’s a lovely day outside. Maybe go to a pet shelter and see some puppies? Are you getting enough fiber? Drinking enough water? Why not call a friend and hang out?

CanadaPlus ,

Yeah, I agree with the sentiment, but this is pretty intense.

cyberfae ,
@cyberfae@lemmy.world avatar

While I get your point on the over reliance on Microsoft, some of us are going to be stuck spending the whole day trying to fix this shit. You could show some compassion.

Monument ,

Honestly kind of excited for the company blogs to start spitting out their disaster recovery stories.

I mean - this is just a giant test of disaster recovery plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

If a company uses IPMI (Called AMT and sometimes vPro by Intel), and their network is intact/the devices are on their network, they ought to be able to remotely address this.
But that’s obviously predicated on them having already deployed/configured the tools.

catloaf ,

On desktops, nobody does. Servers, yes, all the time.

ninekeysdown ,
@ninekeysdown@lemmy.world avatar

Depends on your management solutions. Intel vPro can allow remote access like that on desktops & laptops even if they’re on WiFi and in some cases cellular. It’s gotta be provisioned first though.

catloaf ,

Yeah, and no company in my experience bothers provisioning it. The cost of configuring and maintaining it exceeds the cost of handling failure events, even on large scales like this.

CanadaPlus ,

I’m here right now just to watch it unfold in real time. Unfortunately Reddit is looking juicer on that front.

libreddit.northboot.xyz

v9CYKjLeia10dZpz88iU ,

I mean - this is just a giant test of disaster recovery plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

It seems like it is, I’m not responsible for any computers that had this issue, but I saw this powershell script posted on reddit for a group policy.

Though, I think some systems had more unique problems, I also saw different steps for repairing an Azure VM.

There were also that didn’t understand how to get around Bitlocker, and people on reddit posted solutions for that too.

Though, even with all of this, I was surprised that hospitals had issues. It seems like there’s other issues in deployments, and I saw some people on YC claim this was related to organizations filling checkboxes for regulatory requirements. That they likely had this software because they were concerned with failing an audit. I don’t know if there’s truth to that, but I am surprised there wasn’t more redundancy in critical infrastructure.

edit: I want to stress again that I’m not responsible for any computers that had this issue and haven’t tried to use any of the above solutions myself. I’ve just noticed lots of people still commenting on reddit not understanding that they can fix this issue with one of these 3.

EncryptKeeper ,

Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao

TheBat ,
@TheBat@lemmy.world avatar

How many coffee cups have you drank in the last 12 hours?

Cryophilia ,

I work in a data center

I lost count

TheBat ,
@TheBat@lemmy.world avatar

What was Dracula doing in your data centre?

KingThrillgore ,
@KingThrillgore@lemmy.ml avatar

Because he’s Dracula. He’s twelve million years old.

THE WORMS

szczuroarturo ,

I always wondered who even used windows server given how marginal its marketshare is. Now i know from the news.

Semi_Hemi_Demigod ,
@Semi_Hemi_Demigod@lemmy.world avatar

Did you feel a great disturbance in the force?

kamenoko ,

AWS No!!!

Oh wait it’s not them for once.

recapitated ,

Clownstrike

lando55 ,

Crowdshite haha gotem

iAvicenna ,
@iAvicenna@lemmy.world avatar

https://lemmy.world/pictrs/image/2b3581a1-27d9-473e-b9b9-864dc2c41ba1.jpeg

Nachorella ,

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I’ll be working from home for a few days before anything is fixed.

AnUnusualRelic ,
@AnUnusualRelic@lemmy.world avatar

An offline server is a secure server!

CanadaPlus ,

Honestly my philosophy these days, when it comes to anything proprietary. They just can’t keep their grubby little fingers off of working software.

At least this time it was an accident.

ari_verse ,

It’s a fair point but I would rather diversify and also use something that is open / less opaque

ramble81 ,

We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **

veam ,

oh joy. can’t wait to have to fix this for all of our clients today…

iturnedintoanewt ,
@iturnedintoanewt@lemm.ee avatar

You have no idea how much fun its being.

Passerby6497 ,

I’m so tired of all the fun…

stochastic_parrot ,

“Today”, right. I wish you a good weekend stranger.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •