There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Today I'm grateful I'm using Linux - Global IT issues caused by Crowdstrike update causes BSOD on Windows

This isn’t a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn’t log in.

Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.)

Seems insane to me that one company’s messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

bricklove ,

I wanted to share the article with friends and copy a part of the text I wanted to draw attention to but the asshole site has selection disabled. Now I will not do that and timesnownews can go fuck themselves

ArrogantAnalyst ,

It is annoying. Some possible solutions:

On desktop: Using Shift + ALT you often can overrule this and select text anyway.

On mobile: Using the reader mode or the Print preview often works. It does for me on this website.

beeng ,

This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

Its true, but otherside of same coin is that with too much solo implementation you lose benefits of economy of scale.

But indeed the world seems like a village today.

Swarfega ,

I’ve just spent the past 6 hours booting into safe mode and deleting crowd strike files on servers.

allywilson ,

Feel you there. 4 hours here. All of them cloud instances whereby getting acces to the actual console isn’t as easy as it should be, and trying to hit F8 to get the menu to get into safe mode can take a very long time.

possiblylinux127 ,

Can’t you automate it?

ArrogantAnalyst ,

Since it has to happen in windows safe mode it seems to be very hard to automate the process. I haven’t seen a solution yet.

Swarfega ,

Sadly not. Windows doesn’t boot. You can boot it into safe mode with networking, at which point maybe with anaible we could login to delete the file but since it’s still manual work to get windows into safe mode there’s not much point

Restaldt ,

Hopefully this will be the straw that breaks this dead camels back.

Microsoft should get buried after this

Nighed ,
@Nighed@feddit.uk avatar

It’s not a Microsoft problem

gentooer ,

It’s a world-depending-on-a-few-large-companies problem

isolatedscotch ,

after reading all the comments I still have no idea what the hell crowdstrike is

TimeSquirrel ,
@TimeSquirrel@kbin.melroy.org avatar

Seems to be some sort of kernel-embedded threat detection system. Which is why it was able to easily fuck the OS. It was running in the most trusted space.

Ok_imagination ,

AV, EDP they offer other solutions as well. I think their main selling point is tamper-proof protection as well.

chameleon ,
@chameleon@fedia.io avatar

Company offering new-age antivirus solutions, which is to say that instead of being mostly signature-based, it tries to look at application behavior instead. If Word was exploited because some user opened not_a_virus_please_open.docx from their spam folder, Word might be exploited and end up running some malware that tries to encrypt the entire drive. It's supposed to sniff out that 1. Word normally opens and saves like one document at a time and 2. some unknown program is being overly active. And so it should stop that and ring some very loud alarm bells at the IT department.

Basically they doubled down on the heuristics-based detection and by that, they claim to be able to recognize and stop all kinds of new malware that they haven't seen yet. My experience is that they're always the outlier on the top-end of false positives in business AV tests (eg AV-Comparatives Q2 2024) and their advantage has mostly disappeared since every AV has implemented that kind of behavior-based detection nowadays.

Tenkard ,

I would be too, except Firefox just started crashing on Wayland all the morning D;

axzxc1236 ,

I am born too late to understand what Y2K problem was, this might be what people thought could happen.

cannedtuna ,

Kinda I guess. It was about clocks rolling over from 1999 to 2000 and causing a buffer overflow that would supposedly crash all systems everywhere causing the country to come to a hault.

Hildegarde ,

Most old systems used two digits for years. The year would go from 99 to 0. Any software doing a date comparison will get a garbage result. If a task needs to be run every 5 minutes, what will the software do if that task was last run 99 years from now? It will not work properly.

Governments and businesses spent lots of money and time patching critical systems to handle the date change. The media made a circus out of it, but when the year rolled over, everything was fine.

cannedtuna ,

We also got the worst version of Windows ever, ME. Tho maybe with all the BS they’ve done with 11 that might change.

zod000 ,

I’m not sure I’d stick to calling it the worst version “ever” since MS is trying really hard to out do themselves.

ikidd ,
@ikidd@lemmy.world avatar

I’d use ME before the adware that is the current version. It wasn’t that bad, it was just Win98 with some visual slop on top that crashed slightly more often.

Aceticon ,

Also a lot of people were “on call” to handle any problems when the year changed, so the few problem that had passed unnoticed when doing the fixed and did pop up when the year changed, got solved a lot faster than they normally would.

caseyweederman ,

And it was okay because a lot of people worked really really hard to make it be okay.

caseyweederman ,

Y2K was going to be the end of civilisation. This was basically done by the time I woke up today.

HumanPenguin , (edited )
@HumanPenguin@feddit.uk avatar

Yep pretty much but on a larger scale.

1st please do not believe the bull that there was no problem. Many folks like me were paid to fix it before it was an issue. So other than a few companies, few saw the result, not because it did not exist. But because we were warned. People make jokes about the over panic. But if that had not happened, it would hav been years to fix, not days. Because without the panic, most corporations would have ignored it. Honestly, the panic scared shareholders. So boards of directors had to get experts to confirm the systems were compliant. And so much dependent crap was found running it was insane.

But the exaggerations of planes falling out of the sky etc. Was also bull. Most systems would have failed but BSOD would be rare, but code would crash and some works with errors shutting it down cleanly, some undiscovered until a short while later. As accounting or other errors showed up.

As other have said. The issue was that since the 1960s, computers were set up to treat years as 2 digits. So had no expectation to handle 2000 other than assume it was 1900. While from the early 90s most systems were built with ways to adapt to it. Not all were, as many were only developing top layer stuff. And many libraries etc had not been checked for this issue. Huge amounts of the infra of the world’s IT ran on legacy systems. Especially in the financial sector where I worked at the time.

The internet was a fairly new thing. So often stuff had been running for decades with no one needing to change it. Or having any real knowledge of how it was coded. So folks like me were forced to hunt through code or often replace systems that were badly documented or more often not at all.

A lot of modern software development practices grew out of discovering what a fucking mess can grow if people accept an “if it ain’t broke, don’t touch it” mentality.

sep ,

Was there patching systems and testing they survived the rollover months before it happened.
One software managed the rollover. But failed the year after. They had quickly coded in an explicit exception for 00. But then promptly forgot to fix it properly!.

nickiam2 , (edited )

I work in hospitality and our systems are completely down. No POS, no card processing, no reservations, we’re completely f’ked.

Our only saving grace is the fact that we are in a remote location and we have power outages frequently. So operating without a POS is semi-normal for us.

abbiistabbii ,
@abbiistabbii@lemmy.blahaj.zone avatar

We’re all going to be so smug.

digdilem ,

Am on holiday this week - called in to help deal with this shit show :(

Botzo ,

Don’t worry, George Kurtz (crowdstrike CEO) is unavailable today. He’s got racing to do gt-world-challenge-america.com/…/virginia-interna…

Suoko ,
@Suoko@feddit.it avatar

A couple of days ago a Windows 2016 server started a license strike in my farm … Coincidence?

laughterlaughter ,

Yes.

fin ,

That’s hell of a strike to the crowd

HulkSmashBurgers ,

Me too. Additionally, I use guix so if a system update ever broke my machine I can just rollback to a prior system version (either via the command line or grub menu).

Cysioland ,
@Cysioland@lemmygrad.ml avatar

Immutable systems sound like something desperately needed, tbh. It’s just such an obvious solution and I’m surprised that it’s been invented so late

totally_notAcat ,

That’s assuming grub doesn’t get broken in the update…

shirro ,

I isn’t even a Linux vs Windows thing but a competent at your job vs don’t know what the fuck you are doing thing. Critical systems are immutable and isolated or as close as reasonably possible. They don’t do live updates of third party software and certainly not software that is running privileged and can crash the operating system.

I couldn’t face working in corporate IT with this sort of bullshit going on.

msage ,

So it’s Linux vs Windows

Cyber ,

No it’s Crowdstrike… we’re just seeing an issue with their Windows software, not their Linux software.

Sethayy ,

That being said Microsoft still did hire crowd strike and give them the keys to release an update like this.

End result still is windows having more issues than linux

umbrella ,
@umbrella@lemmy.ml avatar

I couldn’t face working in corporate IT with this sort of bullshit going on.

im taking you don’t work in IT anymore then?

KillerTofu ,

There are state and government IT departments.

rozodru ,

This is just like “what not to do in IT/dev/tech 101” right here. Every since I’ve been in the industry for literally decades at this point I was always told, even when in school, “Never test in production, never roll anything out to production on a Friday, if you’re unsure have someone senior code review” of which, Crowdstrike, failed to do all of the above. Even the most junior of junior devs should know better. So the fact that this update was allowed go through…I mean blame the juniors, the seniors, the PM’s, the CTO’s, everyone. If your shit is so critical that a couple bad lines of poorly written code (which apparently is what it was) can cripple the majority of the world…yeah crowdstrike is done.

magic_lobster_party ,

It’s incredible how an issue of this magnitude didn’t get discovered before they shipped it. It’s not exactly an issue that happens in some niche cases. It’s happening on all Windows computers!

This can only happen if they didn’t test their product at all before releasing to production. Or worse: maybe they did test, got the error, and they just “eh, it’s probably just something wrong with test systems”, and then shipped anyway.

This is just stupid.

CalcProgrammer1 ,
@CalcProgrammer1@lemmy.ml avatar

It’s also a “don’t allow third party proprietary shit into your kernel” issue. If the driver was open source it would actually go through a public code review and the issue would be more likely to get caught. Even if it did slip through people would publically have a fix by now with all the eyes on the code. It also wouldn’t get pushed to everyone simultaneously under the control of a single company, it would get tested and packaged by distributions before making it to end users.

Aphelion ,

It’s actually a “test things first and have a proper change control process” thing. Doesn’t matter if it’s open source, closed source scummy bullshit or even coded by God: you always test it first before hitting deploy.

Aceticon ,

More generally: delegate anything critical to a 3rd party and you’ve just put your business at the mercy of the quality (or lack thereof) of their own business processes which you do not control, which is especially dangerous in the current era of “cheapest as possible” hiring practices.

Having been in IT for almost 3 decades, a lesson I have learned long ago and which I’ve also been applying to my own things (such as having my own domain for my own e-mail address rather than using something like Google) was that you should avoid as much as possible to have your mission critical or hard to replace stuff dependent on a 3rd Party, especially if the dependency is Live (i.e. activelly connected rather than just buying and installing their software).

I’ve managed to avoid quite a lot of the recent enshittification exactly because I’ve been playing it safe in this domain for 2 decades.

catculation ,
@catculation@lemmy.zip avatar

Even 911 is impacted

Bitrot ,

In the US 911 is decentralized, so widespread things will always affect it in some places. Solarwinds hack was another one.

Assuming the entire phone system isn’t down, there are typically very shitty to deal with workarounds for CAD outages.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •