There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Today I'm grateful I'm using Linux - Global IT issues caused by Crowdstrike update causes BSOD on Windows

This isn’t a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn’t log in.

Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.)

Seems insane to me that one company’s messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

PlexSheep ,

From my understanding, they have some ring 0 thing that fucked up. Could that not in theory happen on our beloved Linux systems? Or does the kernel generally not give that option?

ZugZug ,

Even fucked up mu hospital meals. Fuck windows.

Reddfugee42 ,

Most people are completely oblivious because it only affects people using crowdstrike, which practically excludes general consumers.

resin85 ,

https://lemmy.ca/pictrs/image/753b053b-a0de-4b17-af14-4198b6e2694d.jpeg

Churbleyimyam ,

Me too. I’m also grateful I still accept cash.

monoboy ,

Didn’t Cloudstrike have a bad update to Debian systems back in April this year that caused a lot of problems? I don’t think it was a big thing since not as many companies are using Cloudstrike on Debian.

Sounds like the issue here is Cloudstrike and not Windows.

balder1993 , (edited )

They didn’t even bother to do a gradual rollout, like even small apps do.

The level of company-wide incompetence is astounding, but considering how organizations work and disregard technical people’s concerns, I’m never surprised when these things happen. It’s a social problem more than a technical one.

PlexSheep ,

They didn’t even bother to test their stuff, must have pushed to prod

(Technically, test in prod)

Treczoks ,

Same here. I was totally busy writing software in a new language and a new framework, and had a gazillion tabs on Google and stackexchange open. I didn’t notice any network issues until I was on my way home, and the windows f-up was the one big thing in the radio news. Looks like Windows admins will have a busy weekend.

SquigglyEmpire ,

Only if they manage Crowdstrike systems, thankfully.

bricklove ,

I wanted to share the article with friends and copy a part of the text I wanted to draw attention to but the asshole site has selection disabled. Now I will not do that and timesnownews can go fuck themselves

ArrogantAnalyst ,

It is annoying. Some possible solutions:

On desktop: Using Shift + ALT you often can overrule this and select text anyway.

On mobile: Using the reader mode or the Print preview often works. It does for me on this website.

reddeadhead ,

The firefox reader mode button doesn’t show up for me on that site. I wonder if its just a poor site or if they are intentionally trying to break it.

ArrogantAnalyst ,

Could be both. You can enforce it: addons.mozilla.org/en-US/…/activate-reader-view/

cryoistalline ,

heres the entire article

Latest Crowdstrike Update Issue: Many Windows users are experiencing Blue Screen of Death (BSOD) errors due to a recent CrowdStrike update. The issue affects various sensor versions, and CrowdStrike has acknowledged the problem and is investigating the cause, as stated in a pinned message on the company’s forum.
Who Have Been Affected
Australian banks, airlines, and TV broadcasters first reported the issue, which quickly spread to Europe as businesses began their workday. UK broadcaster Sky News couldn’t air its morning news bulletins, while Ryanair experienced IT issues affecting flight departures. In the US, the Federal Aviation Administration grounded all Delta, United, and American Airlines flights due to communication problems, and Berlin airport warned of travel delays from technical issues.
In India too, numerous IT organisations were reporting in issues with company-wide. Akasa Airlines and Spicejet experienced technical issues affecting online services. Akasa Airlines’ booking and check-in systems were down at Mumbai and Delhi airports due to service provider infrastructure issues, prompting manual check-in and boarding. Passengers were advised to arrive early, and the airline assured swift resolution. Spicejet also faced problems updating flight disruptions, actively working to fix the issue. Both airlines apologized for the inconvenience caused and promised updates as soon as the problems were resolved.
Crowdstrike’s Response
CrowdStrike acknowledged the problem, linked to their Falcon sensor, and reverted the faulty update. However, affected machines still require manual intervention. IT admins are resorting to booting into safe mode and deleting specific system files, a cumbersome process for cloud-based servers and remote laptops. Reports from IT professionals on Reddit highlight the severity, with entire companies offline and many devices stuck in boot loops. The outage underscores the vulnerability of interconnected systems and the critical need for robust cybersecurity solutions. IT teams worldwide face a long and challenging day to resolve the issues and restore normal operations.
What to Expect:

-A Technical Alert (TA) detailing the problem and potential workarounds is expected to be published shortly by CrowdStrike.
-The forum thread will remain pinned to provide users with easy access to updates and information.

What Users Should Do:

-Hold off on troubleshooting: Avoid attempting to fix the issue yourself until the official Technical Alert is released.
-Monitor the pinned thread: This thread will be updated with the latest information, including the TA and any temporary solutions.
-Be patient: Resolving software conflicts can take time. CrowdStrike is working on a solution, and updates will be posted as soon as they become available.

**In an automated reply from Crowdstrike, the company had stated:**CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Symptoms include hosts experiencing a blue screen error related to the Falcon Sensor. The course of current action will be - our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket. Status updates will be posted as we have more information to share, including when the issue is resolved.
For Users Experiencing BSODs:
If you’re encountering BSOD errors after a recent CrowdStrike update, you’re not alone. This appears to be a widespread issue. The upcoming Technical Alert will likely provide specific details on affected CrowdStrike sensor versions and potential workarounds while a permanent fix is developed.
If you have urgent questions or concerns, consider contacting CrowdStrike support directly.

beeng ,

This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

Its true, but otherside of same coin is that with too much solo implementation you lose benefits of economy of scale.

But indeed the world seems like a village today.

Swarfega ,

I’ve just spent the past 6 hours booting into safe mode and deleting crowd strike files on servers.

allywilson ,

Feel you there. 4 hours here. All of them cloud instances whereby getting acces to the actual console isn’t as easy as it should be, and trying to hit F8 to get the menu to get into safe mode can take a very long time.

Swarfega ,

Ha! Yes. Same issue. Clicking Reset in vSphere and then quickly switching tabs to hold down F8 has been a ball ache to say the least!

possiblylinux127 ,

Can’t you automate it?

ArrogantAnalyst ,

Since it has to happen in windows safe mode it seems to be very hard to automate the process. I haven’t seen a solution yet.

Swarfega ,

Sadly not. Windows doesn’t boot. You can boot it into safe mode with networking, at which point maybe with anaible we could login to delete the file but since it’s still manual work to get windows into safe mode there’s not much point

lengau ,

It is theoretically automatable, but on bare metal it requires having hardware that’s not normally just sitting in every data centre, so it would still require someone to go and plug something into each machine.

On VMs it’s more feasible, but on those VMs most people are probably just mounting the disk images and deleting the bad file to begin with.

Swarfega ,

I guess it depends on numbers too. We had 200 to work on. If you’re talking hundreds more than looking at automation would be a better solution. In our scenario it was just easier to throw engineers at it. I honestly thought at first this was my weekend gone but we got through them easily in the end.

Natanael ,

The real problem with VM setups is that the host system might have crashed too

Restaldt ,

Hopefully this will be the straw that breaks this dead camels back.

Microsoft should get buried after this

Nighed ,
@Nighed@feddit.uk avatar

It’s not a Microsoft problem

gentooer ,

It’s a world-depending-on-a-few-large-companies problem

isolatedscotch ,

after reading all the comments I still have no idea what the hell crowdstrike is

TimeSquirrel ,
@TimeSquirrel@kbin.melroy.org avatar

Seems to be some sort of kernel-embedded threat detection system. Which is why it was able to easily fuck the OS. It was running in the most trusted space.

Ok_imagination ,

AV, EDP they offer other solutions as well. I think their main selling point is tamper-proof protection as well.

chameleon ,
@chameleon@fedia.io avatar

Company offering new-age antivirus solutions, which is to say that instead of being mostly signature-based, it tries to look at application behavior instead. If Word was exploited because some user opened not_a_virus_please_open.docx from their spam folder, Word might be exploited and end up running some malware that tries to encrypt the entire drive. It's supposed to sniff out that 1. Word normally opens and saves like one document at a time and 2. some unknown program is being overly active. And so it should stop that and ring some very loud alarm bells at the IT department.

Basically they doubled down on the heuristics-based detection and by that, they claim to be able to recognize and stop all kinds of new malware that they haven't seen yet. My experience is that they're always the outlier on the top-end of false positives in business AV tests (eg AV-Comparatives Q2 2024) and their advantage has mostly disappeared since every AV has implemented that kind of behavior-based detection nowadays.

Tenkard ,

I would be too, except Firefox just started crashing on Wayland all the morning D;

axzxc1236 ,

I am born too late to understand what Y2K problem was, this might be what people thought could happen.

cannedtuna ,

Kinda I guess. It was about clocks rolling over from 1999 to 2000 and causing a buffer overflow that would supposedly crash all systems everywhere causing the country to come to a hault.

Hildegarde ,

Most old systems used two digits for years. The year would go from 99 to 0. Any software doing a date comparison will get a garbage result. If a task needs to be run every 5 minutes, what will the software do if that task was last run 99 years from now? It will not work properly.

Governments and businesses spent lots of money and time patching critical systems to handle the date change. The media made a circus out of it, but when the year rolled over, everything was fine.

cannedtuna ,

We also got the worst version of Windows ever, ME. Tho maybe with all the BS they’ve done with 11 that might change.

zod000 ,

I’m not sure I’d stick to calling it the worst version “ever” since MS is trying really hard to out do themselves.

ikidd ,
@ikidd@lemmy.world avatar

I’d use ME before the adware that is the current version. It wasn’t that bad, it was just Win98 with some visual slop on top that crashed slightly more often.

Aceticon ,

Also a lot of people were “on call” to handle any problems when the year changed, so the few problem that had passed unnoticed when doing the fixed and did pop up when the year changed, got solved a lot faster than they normally would.

caseyweederman ,

And it was okay because a lot of people worked really really hard to make it be okay.

caseyweederman ,

Y2K was going to be the end of civilisation. This was basically done by the time I woke up today.

HumanPenguin , (edited )
@HumanPenguin@feddit.uk avatar

Yep pretty much but on a larger scale.

1st please do not believe the bull that there was no problem. Many folks like me were paid to fix it before it was an issue. So other than a few companies, few saw the result, not because it did not exist. But because we were warned. People make jokes about the over panic. But if that had not happened, it would hav been years to fix, not days. Because without the panic, most corporations would have ignored it. Honestly, the panic scared shareholders. So boards of directors had to get experts to confirm the systems were compliant. And so much dependent crap was found running it was insane.

But the exaggerations of planes falling out of the sky etc. Was also bull. Most systems would have failed but BSOD would be rare, but code would crash and some works with errors shutting it down cleanly, some undiscovered until a short while later. As accounting or other errors showed up.

As other have said. The issue was that since the 1960s, computers were set up to treat years as 2 digits. So had no expectation to handle 2000 other than assume it was 1900. While from the early 90s most systems were built with ways to adapt to it. Not all were, as many were only developing top layer stuff. And many libraries etc had not been checked for this issue. Huge amounts of the infra of the world’s IT ran on legacy systems. Especially in the financial sector where I worked at the time.

The internet was a fairly new thing. So often stuff had been running for decades with no one needing to change it. Or having any real knowledge of how it was coded. So folks like me were forced to hunt through code or often replace systems that were badly documented or more often not at all.

A lot of modern software development practices grew out of discovering what a fucking mess can grow if people accept an “if it ain’t broke, don’t touch it” mentality.

sep ,

Was there patching systems and testing they survived the rollover months before it happened.
One software managed the rollover. But failed the year after. They had quickly coded in an explicit exception for 00. But then promptly forgot to fix it properly!.

nickiam2 , (edited )

I work in hospitality and our systems are completely down. No POS, no card processing, no reservations, we’re completely f’ked.

Our only saving grace is the fact that we are in a remote location and we have power outages frequently. So operating without a POS is semi-normal for us.

Unforeseen ,

I’ve worked with POS systems my whole career and I still can’t help think Piece Of Shit whenever I see it

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •