There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Slow recovery from IT outage begins as experts warn of future risks

Fault in CrowdStrike caused airports, businesses and healthcare services to languish in ‘largest outage in history’

Services began to come back online on Friday evening after an IT failure that wreaked havoc worldwide. But full recovery could take weeks, experts have said, after airports, healthcare services and businesses were hit by the “largest outage in history”.

Flights and hospital appointments were cancelled, payroll systems seized up and TV channels went off air after a botched software upgrade hit Microsoft’s Windows operating system.

It came from the US cybersecurity company CrowdStrike, and left workers facing a “blue screen of death” as their computers failed to start. Experts said every affected PC may have to be fixed manually, but as of Friday night some services started to recover.

As recovery continues, experts say the outage underscored concerns that many organizations are not well prepared to implement contingency plans when a single point of failure such as an IT system, or a piece of software within it, goes down. But these outages will happen again, experts say, until more contingencies are built into networks and organizations introduce better back-ups.

TheDemonBuer ,
@TheDemonBuer@lemmy.world avatar

Here’s an idea: don’t give one company kernel level access to the OS of millions of PCs that are necessary to keep whole industries functioning.

NuXCOM_90Percent ,

So we should have five different cyber security solutions at any given site? That wheezing is the sound of every it person on the planet queuing to swing a sock full of nickles at you.

Crowdstrike was near ubiquitous because it was the best tool out there. And plenty of threats were prevented because of it.

The answer isn’t to force every single site to manage everything themselves. It is to increase oversight on ci/CD models

HubertManne ,

I read his comment more about the kernel level access more than the one company.

NuXCOM_90Percent ,

Like it or not, that is the most effective way to collect the data these solutions need.

This isn’t riot anti cheat where it is of questionable effectiveness. Crowdstrike was demonstrably amazing at its job.

riskable ,
@riskable@programming.dev avatar

Crowdstrike has clients that run on MacOS and Linux. Only the Windows version requires kernel level access. I believe it has something to do with the absolute shitshow that is Windows security model but it might also be because it runs a 31-year-old filesystem that still doesn’t allow one process to read another process’s files while they’re open.

TheDemonBuer ,
@TheDemonBuer@lemmy.world avatar

Crowdstrike was near ubiquitous because it was the best tool out there.

I understand the reason for it, but that ubiquity comes with potential dangers, as we saw on Friday. But, no, I don’t think the solution is “five different cyber security solutions” at every site. However, different cyber security solutions for different industries might not be such a bad idea. Or, I suppose the root of the problem might be the ubiquity of the OS. Should every PC be running the same jack of all trades but master of none OS?

NuXCOM_90Percent ,

Again, all you are doing is increasing complexity and punting it to a support staff who are likely unqualified to even know what crowdstrike did.

This was one of those rare cases of capitalism working. There are many options. There was one that was miles ahead of all the others and that dominated.

HK65 ,

How did capitalism as in private ownership structures help here?

ansiz ,

I mean, Microsoft themselves regularly shits the bed with updates, even with Defender updates. It’s the nature of security, they have to have that kind of access to stop legit malware. That’s why these kind of outages happen every few years. This one just got to much coverage from the banking and airline issues. And I’m sure future outages will continue to get similar coverage.

But the Crowdstrike CEO was also at McAfee in 2010 when they shit the bed and shut down millions of XP machines so it seems like he needs a different career…

SkyNTP ,

The problem is the monoculture. We are fucking addicted to convenience and efficiency at all costs.

A diverse ecosystem, if a bit more work to manage, is much more resilient, and wouldn’t have been this catastrophe.

Our technology is great, but our processes suck. Standardization. Just in time. These ideas create incredibly fragile organizations. Humanity is so short sighted. We are screwed.

AtHeartEngineer ,
@AtHeartEngineer@lemmy.world avatar

Also the obligatory: “don’t run infrastructure on Microsoft products, run Linux”

ID411 ,

I love me some risk identification right after an incident.

credo ,

After. Action. Review.

AtHeartEngineer ,
@AtHeartEngineer@lemmy.world avatar

Should be done even when things go right.

ByteOnBikes ,

I’m actually pretty excited to go to work on Monday.

We have spent the past few years hardening our security and simplifying our critical systems. One way to doing that was to move a much off Microsoft as possible.

And since I’ve been on vacation for the past week, I’m either going to walk into a nightmare shit show or everyone is going to be cheering that we are fully operational since we don’t depend on Microsoft.

FlyingSquid ,
@FlyingSquid@lemmy.world avatar

C-suite to experts: Are the future risks short term or long term? Specifically longer term than my golden parachute?

hedgehogging_the_bed ,

This is why “they are the biggest” isn’t a good reason to pick a vendor. If all these companies had been using different providers or even different OS, it wouldn’t have hit so many systems simultaneously. This is a result of too much consolidation at all levels and one issue with the Microsoft OS monopoly.

ImADifferentBird ,
@ImADifferentBird@lemmy.blahaj.zone avatar

The issue, in this case, is more about Crowdstrike’s broad usage than Microsoft’s. The update that crippled everything was to the Crowdstrike Falcon Sensor software, not to the OS.

Funnily enough, they had a similar issue with an update to the Linux version of the software a few months ago, that didn’t have these broad-reaching consequences largely due to the smaller Linux user base. Which means this is starting to look like a pattern, and there are going to need to be some serious process changes at Crowdstrike to prevent things like this in the future.

Anybody’s guess if those changes happen or not.

Z3k3 ,

This would have been a fun MIR had my systems be impacted.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •