There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

NaibofTabr ,

Wow, I didn’t realize CrowdStrike was widespread enough to be a single point of failure for so much infrastructure. Lot of airports and hospitals offline.

The Federal Aviation Administration (FAA) imposed the global ground stop for airlines including United, Delta, American, and Frontier.

Flights grounded in the US.

The System is Down

r00ty Admin ,
r00ty avatar

Apparently at work "some servers are experiencing problems". Sadly, none of the ones I need to use :(

r00ty Admin ,
r00ty avatar

My favourite thing has been watching sky news (UK) operate without graphics, trailers, adverts or autocue. Back to basics.

uuhhhhmmmm ,

I think we’re getting a lot of pictures for !pbsod

Speculater ,
@Speculater@lemmy.world avatar

And subscribed!

solomon42069 ,

Why is no one blaming Microsoft? It’s their non resilient OS that crashed.

blackn1ght ,

Probably because it’s a Crowdstrike issue, they’ve pushed a bad update.

solomon42069 ,

OK, but people aren’t running Crowdstrike OS. They’re running Microsoft Windows.

I think that some responsibility should lie with Microsoft - to create an OS that

  1. Recovers gracefully from third party code that bugs out
  2. Doesn’t allow third party software updates to break boot

I get that there can be unforeseeable bugs, I’m a programmer of over two decades myself. But there are also steps you can take to strengthen your code, and as a Windows user it feels more like their resources are focused on random new shit no one wants instead of on the core stability and reliability of the system.

It seems to be like third party updates have a lot of control/influence over the OS and that’s all well and good, but the equivalent of a “Try and Catch” is what they needed here and yet nothing seems to be in place. The OS just boot loops.

EnderMB ,

It’s not just Windows, it’s affecting services that people that primarily use other OS’s rely on, like Outlook or Federated login.

In these situations, blame isn’t a thing, because everyone knows that a LSE can happen to anyone at any time. The second you start to throw stones, people will throw them back when something inevitably goes wrong.

While I do fundamentally agree with you, and believe that the correct outcome should be “how do we improve things so that this never happens again”, it’s hard to attach blame to Microsoft when they’re the ones that have to triage and ensure that communication is met.

solomon42069 ,

I reckon it’s hard to attach blame to Microsoft because of the culture of corporate governance and how decisions are made (without experts).

Tech has become a bunch of walled gardens with absolute secrecy over minor nothings. After 1-2 decades of that, we have a generation of professionals who have no idea how anything works and need to sign up for $5 a month phone app / cloud services just to do basic stuff they could normally do on their own on a PC - they just don’t know how or how to put the pieces together due to inexperience / lack of exposure.

Whether it’s corporate or government leadership, the lack of understanding of basics in tech is now a liability. It’s allowed corporations like Microsoft to set their own quality standards without any outside regulation while they are entrusted with vital infrastructure and to provide technical advisory, even though they have a clear vested interest there.

lanolinoil ,
@lanolinoil@lemmy.world avatar

banks wouldn’t use something that black box. just trust me bro wouldn’t be a good pitch

catloaf ,

If you trust banks that much, I have very bad news for you.

barsquid ,

AFAICT Microsoft is busy placing ads on everything and screen logging user activity instead of making a resilient foundation.

For contrast: I’ve been running Fedora Atomic. I’m sure it is possible to add some kernel mod that completely breaks the system. But if there was a crash on boot, in most situations, I’d be able to roll back to the last working version of everything.

Damage ,

The thought of a local computer being unable to boot because some remote server somewhere is unavailable makes me laugh and sad at the same time.

Munkisquisher ,

A remote server that you pay some serious money to that pushes a garbage driver that prevents yours from booting

lanolinoil ,
@lanolinoil@lemmy.world avatar

yeah so you can’t get Chinese government spyware installed.

Passerby6497 ,

Not only does it (possibly) prevent booting, but it will also bsod it first so you’ll have to see how lucky you get.

Goddamn I hate crowdstrike. Between this and them fucking up and letting malware back into a system, I have nothing nice to say about them.

Cryophilia , (edited )

It’s bsod on boot

And anything encrypted with bitlocker can’t even go into safe mode to fix it

rxxrc OP ,

I don’t think that’s what’s happening here. As far as I know it’s an issue with a driver installed on the computers, not with anything trying to reach out to an external server. If that were the case you’d expect it to fail to boot any time you don’t have an Internet connection.

Windows is bad but it’s not that bad yet.

__init__ ,

It’s just a fun coincidence that the azure outage was around the same time.

sasquash ,

never do updates on a Friday.

rozodru ,
@rozodru@lemmy.ca avatar

yeah someone fucked up here. I mean I know you’re joking but I’ve been in tech for like 20+ years at this point and it was always, always, ALWAYS, drilled into me to never do updates on Friday, never roll anything out to production on Friday. Fridays were generally meant for code reviews, refactoring in test, work on personal projects, raid the company fridge for beer, play CS at the office, whatever just don’t push anything live or update anything.

And especially now the work week has slimmed down where no one works on Friday anymore so you 100% don’t roll anything out, hell it’s getting to the point now where you just don’t roll anything out on a Thursday afternoon.

0x0 ,

And especially now the work week has slimmed down where no one works on Friday anymore

Excuse me, what now? I didn’t get that memo.

meanmon13 ,

Yeah it’s great :-) 4 10hr shifts and every weekend is a 3 day weekend

rozodru ,
@rozodru@lemmy.ca avatar

sorry :( yeah I, at most, do 3 days in the office now. Fridays are a day off and Mondays mostly everyone just works from home if at all. downtown Toronto on Mondays and Fridays is pretty much dead.

Blackmist ,

Yep, anything done on Friday can enter the world on a Monday.

I don’t really have any plans most weekends, but I sure as shit don’t plan on spending it fixing Friday’s fuckups.

spyd3r ,
@spyd3r@sh.itjust.works avatar

Never update unless something is broken.

Toribor ,
@Toribor@corndog.social avatar

This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.

Ookami38 ,

I think vulnerabilities found count as “something broken” and chap you replied to simply did not think that far ahead hahah

huginn ,

For real - A cyber security company should basically always be pushing out updates.

iknowitwheniseeit ,

BTW, I use Arch.

Nachorella ,

If it was Arch you’d update once every 15 minutes whether anything’s broken or not.

Passerby6497 ,

That’s advice so smart you’re guaranteed to have massive security holes.

robocall ,
@robocall@lemmy.world avatar

Buy the dip!

jecht360 ,
@jecht360@lemmy.world avatar

More like short them. This is going to be devastating for their business. I could see them losing tons of customers.

bdonvr ,

Huh, so that’s why the office couldn’t order pizza last night lmfao

umami_wasbi ,

No one bother to test before deploying to all machines? Nice move.

pufferfisherpowder ,

YOLO 🚀🙈

huginn ,

This outage is probably costing a significant portion of Crowd strike’s market cap. They’re an 80 billion dollar company but this is a multibillion outage.

Someone’s getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.

TheBat ,
@TheBat@lemmy.world avatar

Puts on Crowdstrike?

bdonvr ,

The amount of servers running Windows out there is depressing to me

franklin ,
@franklin@lemmy.world avatar

The four multinational corporations I worked at were almost entirely Windows servers with the exception of vendor specific stuff running Linux. Companies REALLY want that support clause in their infrastructure agreement.

Blackmist ,

I’ve had my PC shut down for updates three times now, while using it as a Jellyfin server from another room. And I’ve only been using it for this purpose for six months or so.

I can’t imagine running anything critical on it.

ccdfa ,

Windows server, the OS, runs differently from desktop windows. So if you’re using desktop windows and expecting it to run like a server, well, that’s on you. However, I ran windows server 2016 and then 2019 for quite a few years just doing general homelab stuff and it is really a pain compared to Linux which I switched to on my server about a year ago. Server stuff is just way easier on Linux in my experience.

conciselyverbose ,

It doesn’t have to, though. Linux manages to do both just fine, with relatively minor compromises.

Expecting an OS to handle keeping software running is not a big ask.

0xD ,

Well with your level of expertise you should probably not be running anything, to be honest :)

Rinox ,

I dunno, but doesn’t like a quarter of the internet kinda run on Azure?

atocci ,

And 60% of Azure is running Linux lol

Rinox ,

I guess Spotify was running on the other 40%, as many other services

pewgar_seemsimandroid ,

so 40% of azure crashes a quarter of the internet…

index ,

play stupid games win stupid prizes

ari_verse ,

A few years ago when my org got the ask to deploy the CS agent in linux production servers and I also saw it getting deployed in thousands of windows and mac desktops all across, the first thought that came to mind was “massive single point of failure and security threat”, as we were putting all the trust in a single relatively small company that will (has?) become the favorite target of all the bad actors across the planet. How long before it gets into trouble, either because if it’s own doing or due to others?

I guess that we now know

SupraMario ,

No bad actors did this, and security goes in fads. Crowdstrike is king right now, just as McAfee/Trellix was in the past. If you want to run around without edr/xdr software be my guest.

ansiz ,

All of the security vendors do it over enough time. McAfee used to be the king of them.

zdnet.com/…/defective-mcafee-update-causes-worldw…

bleepingcomputer.com/…/trend-micro-antivirus-modi…

techradar.com/…/microsoft-releases-fix-for-botche…

veam ,

oh joy. can’t wait to have to fix this for all of our clients today…

iturnedintoanewt ,
@iturnedintoanewt@lemm.ee avatar

You have no idea how much fun its being.

Passerby6497 ,

I’m so tired of all the fun…

stochastic_parrot ,

“Today”, right. I wish you a good weekend stranger.

ramble81 ,

We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •