There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

r00ty Admin ,
r00ty avatar

My favourite thing has been watching sky news (UK) operate without graphics, trailers, adverts or autocue. Back to basics.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

One possible fix is to delete a particular file while booting in safe mode. But then they’ll need to fix each system manually. My company encrypts the disks as well so it’s going to be a even bigger pain (for them). I’m just happy my weekend started early.

Valmond ,

You have ta have access to boot in safe mode too, I guess I can’t on my work pc for example.

What a shitty workaround & might crowd strike burn in hell lol

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

Enjoy your weekend unless you are in IT

rozodru ,
@rozodru@lemmy.ca avatar

that would only work for like low level people’s laptops. apparently if your role requires a more secure machine you also have to deal with bitlocker whiiiiiiich is tied in with crowdstrike soooooo no dice.

alphacyberranger , (edited )
@alphacyberranger@sh.itjust.works avatar

Yeah that would be case in most laptops. So if bitlocker is involved as well what could be the possible fix.

rozodru ,
@rozodru@lemmy.ca avatar

I mean if your IT was smart, IF they were smart, they would have the bitlocker decryptions backed up on like a usb or something. IF you need to access the decryption via microsoft then you’re apparently borked for now.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

That would be funny

catloaf ,

Yeah, most large orgs have a key server, or back up to AD. If you don’t have that, and no recovery key, you’re fucked and that data is gone.

alphacyberranger ,
@alphacyberranger@sh.itjust.works avatar

What if that is running crowdstrike?

catloaf ,

I’ll give you one guess.

(That’s why when I was in charge of that stuff at one company, I had that recovery key printed out and kept separately in a lockbox.)

UncleArthur ,

Annoyingly, my laptop seems to be working perfectly.

Valmond ,

That’s the burden when you run Arch, right?

Damage ,

lol he said it’s working

ramble81 ,

We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **

jedibob5 ,

Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half…

veam ,

oh joy. can’t wait to have to fix this for all of our clients today…

iturnedintoanewt ,
@iturnedintoanewt@lemm.ee avatar

You have no idea how much fun its being.

Passerby6497 ,

I’m so tired of all the fun…

stochastic_parrot ,

“Today”, right. I wish you a good weekend stranger.

Nachorella ,

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I’ll be working from home for a few days before anything is fixed.

solomon42069 ,

Why is no one blaming Microsoft? It’s their non resilient OS that crashed.

blackn1ght ,

Probably because it’s a Crowdstrike issue, they’ve pushed a bad update.

solomon42069 ,

OK, but people aren’t running Crowdstrike OS. They’re running Microsoft Windows.

I think that some responsibility should lie with Microsoft - to create an OS that

  1. Recovers gracefully from third party code that bugs out
  2. Doesn’t allow third party software updates to break boot

I get that there can be unforeseeable bugs, I’m a programmer of over two decades myself. But there are also steps you can take to strengthen your code, and as a Windows user it feels more like their resources are focused on random new shit no one wants instead of on the core stability and reliability of the system.

It seems to be like third party updates have a lot of control/influence over the OS and that’s all well and good, but the equivalent of a “Try and Catch” is what they needed here and yet nothing seems to be in place. The OS just boot loops.

EnderMB ,

It’s not just Windows, it’s affecting services that people that primarily use other OS’s rely on, like Outlook or Federated login.

In these situations, blame isn’t a thing, because everyone knows that a LSE can happen to anyone at any time. The second you start to throw stones, people will throw them back when something inevitably goes wrong.

While I do fundamentally agree with you, and believe that the correct outcome should be “how do we improve things so that this never happens again”, it’s hard to attach blame to Microsoft when they’re the ones that have to triage and ensure that communication is met.

solomon42069 ,

I reckon it’s hard to attach blame to Microsoft because of the culture of corporate governance and how decisions are made (without experts).

Tech has become a bunch of walled gardens with absolute secrecy over minor nothings. After 1-2 decades of that, we have a generation of professionals who have no idea how anything works and need to sign up for $5 a month phone app / cloud services just to do basic stuff they could normally do on their own on a PC - they just don’t know how or how to put the pieces together due to inexperience / lack of exposure.

Whether it’s corporate or government leadership, the lack of understanding of basics in tech is now a liability. It’s allowed corporations like Microsoft to set their own quality standards without any outside regulation while they are entrusted with vital infrastructure and to provide technical advisory, even though they have a clear vested interest there.

lanolinoil ,
@lanolinoil@lemmy.world avatar

banks wouldn’t use something that black box. just trust me bro wouldn’t be a good pitch

catloaf ,

If you trust banks that much, I have very bad news for you.

barsquid ,

AFAICT Microsoft is busy placing ads on everything and screen logging user activity instead of making a resilient foundation.

For contrast: I’ve been running Fedora Atomic. I’m sure it is possible to add some kernel mod that completely breaks the system. But if there was a crash on boot, in most situations, I’d be able to roll back to the last working version of everything.

Pudutr0n ,
@Pudutr0n@feddit.cl avatar

This is a better article. It’s a CrowdStrike issue with an update (security software)

kamenoko ,

AWS No!!!

Oh wait it’s not them for once.

invisiblegorilla ,

Ironic. They did what they are there to protect against. Fucking up everyone’s shit

Telorand ,

Maybe centralizing everything onto one company’s shoulders wasn’t such a great idea after all…

Monument ,

Honestly kind of excited for the company blogs to start spitting out their disaster recovery stories.

I mean - this is just a giant test of disaster recovery plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

If a company uses IPMI (Called AMT and sometimes vPro by Intel), and their network is intact/the devices are on their network, they ought to be able to remotely address this.
But that’s obviously predicated on them having already deployed/configured the tools.

catloaf ,

On desktops, nobody does. Servers, yes, all the time.

ninekeysdown ,
@ninekeysdown@lemmy.world avatar

Depends on your management solutions. Intel vPro can allow remote access like that on desktops & laptops even if they’re on WiFi and in some cases cellular. It’s gotta be provisioned first though.

catloaf ,

Yeah, and no company in my experience bothers provisioning it. The cost of configuring and maintaining it exceeds the cost of handling failure events, even on large scales like this.

CanadaPlus ,

I’m here right now just to watch it unfold in real time. Unfortunately Reddit is looking juicer on that front.

libreddit.northboot.xyz

v9CYKjLeia10dZpz88iU ,

I mean - this is just a giant test of disaster recovery plans. And while there are absolutely real-world consequences to this, the fix almost seems scriptable.

It seems like it is, I’m not responsible for any computers that had this issue, but I saw this powershell script posted on reddit for a group policy.

Though, I think some systems had more unique problems, I also saw different steps for repairing an Azure VM.

There were also that didn’t understand how to get around Bitlocker, and people on reddit posted solutions for that too.

Though, even with all of this, I was surprised that hospitals had issues. It seems like there’s other issues in deployments, and I saw some people on YC claim this was related to organizations filling checkboxes for regulatory requirements. That they likely had this software because they were concerned with failing an audit. I don’t know if there’s truth to that, but I am surprised there wasn’t more redundancy in critical infrastructure.

edit: I want to stress again that I’m not responsible for any computers that had this issue and haven’t tried to use any of the above solutions myself. I’ve just noticed lots of people still commenting on reddit not understanding that they can fix this issue with one of these 3.

umami_wasbi ,

No one bother to test before deploying to all machines? Nice move.

pufferfisherpowder ,

YOLO 🚀🙈

huginn ,

This outage is probably costing a significant portion of Crowd strike’s market cap. They’re an 80 billion dollar company but this is a multibillion outage.

Someone’s getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.

TheBat ,
@TheBat@lemmy.world avatar

Puts on Crowdstrike?

bdonvr ,

Huh, so that’s why the office couldn’t order pizza last night lmfao

moe90 ,
@moe90@feddit.nl avatar

don’t rely on one desktop OS too much. diversity is the best.

aniki ,

Dont rely on corpo trash at al.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •