There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Major IT outage affecting banks, airlines, media outlets across the world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

Mikina ,

I see a lot of hate ITT on kernel-level EDRs, which I wouldn’t say they deserve. Sure, for your own use, an AV is sufficient and you don’t need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.

The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.

Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.

I’m not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn’t warranted here.

Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.

Einridi ,

The problem here isn’t if you should run EDR or not it’s that people need to take the risk and responsibilities seriously and the cure needs to to be better than the disease.

If you need to hand remote kernel level access over to a company it’s in you to make sure they have the security, QA and basic competency to shoulder that responsibility.

And when it comes out that they don’t even run or verify their code before deploying it to millions of machines all at once, it’s on you for buying not vetting them.

Just like users should check files and where they come from before running them IT professionals need to do the same.

Mwa ,
@Mwa@thelemmy.club avatar

Windows moment 🤣

1luv8008135 ,

Everyone is assuming it’s some intern pushing a release out accidentally or a lack of QA but Microsoft also pushed out July security updates that have been causing bsods on the 9th(?). These aren’t optional either.

What’s the likelihood that the CS file was tested on devices that hadn’t got the latest windows security update and it was an unholy union of both those things that’s caused this meltdown. The timelines do potentially line up when you consider your average agile delivery cadence.

EncryptKeeper ,

I don’t think so. I do updates every two months so I haven’t updated Windows at all in July and it still crashed my servers

1luv8008135 ,

Microsoft installs security updates automatically.

EncryptKeeper ,

Not on any of my servers. All windows updates have to be manually approved installed from the local WSUS server.

skymtf ,

Some intern is getting their ass beat right now, never release into prod without extensive test.

SapphironZA ,

It’s likely not an intern’s fault. Likely a C suite not authorizing the testing infrastructures requested by the developers and sysops people.

elrik ,

If an intern can release to prod without extensive testing there are bigger issues.

Given the scope of the potential impact, if anyone can release to prod and have that deploy to all customers without some form of a canary release strategy, then there are still issues.

fosho ,

https://lemmy.ca/pictrs/image/94996946-b202-49b0-b307-ce38639ad900.jpeg

lustyargonian ,

Linux and Mac just got free advertisment.

LordWiggle ,
@LordWiggle@lemmy.world avatar

The words ‘Mac’ and ‘free’ aren’t allowed in the same sentence.

1luv8008135 ,

Also, enterprise infrastructure running on a Mac? It’s something I’ve never heard of in over a decade and a half of working in tech. And now I’m curious. Is it a thing?

ndru ,

I’ve never heard of Macs running embedded systems - I think that would be a pretty crazy waste of money - but Mac OS Server was a thing for years. My college campus was all Mac in the G4 iMac days, running MacOS Server to administer the network. As far as I understand it was really solid and capable, but I guess it didn’t really fit Apples focus as their market moved from industry professionals to consumers, and they killed it.

andxz , (edited )

Depending on what your definition of “enterprise” is, I’ve attended what was at the time a fairly large and prestigious art school that ran everything on Macs.

They even preferred that we didn’t bring windows laptops, although after some… rather intense protests by pretty much anyone under 25 we did get to bring our own peripherals.

Edit: I’ll also add that outside the shitty keyboards and mice, the server system they had set up with our accounts on etc was completely fine.

Never had a single issue with it and it was my first ever touching a Mac.

1luv8008135 ,

When you say ran everything, what kinda stuff?

andxz , (edited )

This is almost 20 years ago today, so my memory is a bit hazy, but basically each student had an account with a certain amount of server space. I can’t remember the size, but given the amount of digital files we produced it would’ve been at minimum 500GB+/student. We could also “see” the account folder for everyone else in our class for file sharing and stuff.

There were also accounts/folders for each teacher which were used to turn in the primary copy of whatever assignment we had done if it was in digital form. Physical art were scanned or photographed also, as a sort of backup. We were also required to back every project up via USB sticks, ofc.

There was also a rack with individual docks for each digital camera that they had which allowed us to get our photographs transferred to our own folders. Since we could access those files from our accounts it also was a part of that server system.

There were also several networked and customised Macs used for single tasks, like larger printing projects and also for an airgapped paintgun for a lack of a better description. We avoided having to wear masks when we printed large sheets in single colours with it, for example. I have no idea what software that thing used, I think I used it like once or twice.

Now, I’ll freely admit that I haven’t touched a Mac since I left that school, and I’ve never had any interest in them whatsoever, so I don’t know what they used or if it even exists anymore. Someone with more knowhow maybe does?

I do remember them specifically (proudly) telling us it all ran on Macs, otherwise I probably wouldn’t have any reason to believe so. The “server room” was basically what looked like a glorified closet with a rack and a couple of Macs that didn’t look like the ones we students used. This was just before the all-in-one models were introduced, iirc.

arin ,

Heard linux systems had a similar issue a few months ago

lustyargonian ,

Yeah the more I read it seems some crucial security software deployed bad patch.

I’m sure whatever OS would dominate would face widespread issues just by the numbers.

But to be fair it’s fun to make fun of Windows/MSFT :p

witx ,

This is proof you shouldn’t invest everything in one technology. I won’t say everyone should change to Linux because it isn’t immune to this, but we need to push companies to support several OS

thirteene ,

The issue here is kernel level applications that can brick a box. Anti viruses compete for resources, no one should run 2 at once

Entropywins ,

That’s why I run 3 at once…

Doomsider ,

4 for the win!

spacesatan ,

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas.

Never trust a texan

ace_garp ,
@ace_garp@lemmy.world avatar

Totally Texas!

arin ,

To be fair Austin is the least Texan of Texas

merc ,

Is it still at all Austiney though? With all the companies moving to Austin, I wonder how much of the original Austin is left.

arin ,

That’s my point, also same with Salt Lake City Utah

uis ,

Meanwhile Kaspersky: thinks if so incompetent people can even make antivirus at all

misterkiem ,

lol

https://lemmy.world/pictrs/image/b9db4edd-aa39-4133-a4df-6c1d340f9323.png

too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404

greywolf0x1 ,

I can only see 368 comments rn, there must be some weird-ass puritan server blocking .ml users. It’s not beehaw as I can see comments from there.

I can only conclude that it is probably some liberals trying to block “Tankies” and no comment of value was lost.

Gemini24601 ,
@Gemini24601@lemmy.world avatar

Why do people run windows servers when Linux exists, it’s literally a no brainer.

Swarfega ,

Because all software runs from Linux right…

secret300 , (edited )

It could if more people just used Linux

theluckyone ,

Servers weren’t much of a problem, they’re mostly virtual and could be just restored from a backup. The several hundred workstations were a problem. They needed a physical touch. All are encrypted with BitLocker, requiring passkeys stored in AD. Over half are laptops. Most of those don’t have wired ethernet ports, and an account with local admin rights hasn’t logged in since the day they were imaged. Throw in a proper LAPS config, where randomly generated passwords of three dozen characters in length are also stored in AD…

… Yeah, today was a bad day.

Art3sian ,
@Art3sian@lemmy.world avatar

Are you a Linux user?

Hey everyone, shut up! This guy is a Linux user and he’s here to tell us about using Linux.

Doomsider ,

Look! We found the anti anti-microsoft user in their natural habitat of talking shit on social media. They are from the same family bootlickers and related to the Tossaladers. Don’t mention you are an alternative OS person around them or they may go into a blind rage.

Art3sian ,
@Art3sian@lemmy.world avatar

Linux users and vegetarians. Neither can shut the fuck up. Both have made it their identity.

AA5B ,

My last companyoffered apps as either hosted or on-prem. Once they decided the on-prem version had to be Windows for our customer base, it made sense to have one build, one installer. I’m very happy not to be there today.

My current employer is all Linux servers and Engineers use Mac laptops. The only ones affected were Management and HR, LoL. Back of the line for you, we have customers to help!

WordBox ,

No brainer… Blames windows… For a 3rd party issue… The same 3rd party that’s done the same thing here to Linux recently… No brainer achieved.

shirro ,

They run Windows and all this third party software because they would rather pay subscriptions and give up control of their business than retain skilled staff. It has nothing todo with Linux vs Windows. Windows isn’t the reason doors fall of Boeing planes. It is the myopia of modern business culture.

nevetsg ,

I built a home virtual server host on Proxmox. It destroyed itself 3 times in 6 months. I gave up and installed server 2019 with HyperV. Has been rock solid for years.

BurnSquirrel ,

I’m so exhausted… This is madness. As a Linux user I’ve busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.

specterspectre ,

You’re comment I came looking for. You get a standing ovation or something.

DaneGerous ,

A month or so ago a crowdstrike update was breaking some of our Linux vms with newer kernels. So it’s not just the os.

kureta ,

How? I’m really curious to learn.

DaneGerous ,

I don’t know how on either one. I just know it happened.

nevemsenki ,

Crowdstrike bricked networking on our linuxes for quite a few versions.

Robert7301201 ,

This isn’t really a Windows vs Linux issue as far as I’m aware. It was a bad driver update made by a third party. I don’t see why Linux couldn’t suffer from the same kind of issue.

We should dunk on Windows for Windows specific flaws. Like how Windows won’t let me reinstall a corrupted Windows Store library file because admins can’t be trusted to manage Microsoft components on their own machine.

menemen ,
@menemen@lemmy.world avatar

I am no expert. But afaik drivers normally are integrated into the kernel and intensively tested by several parties before getting onto your computer. Only for proprietary drivers this would be problematic under Linux.

nevemsenki ,

Crowdstrike by default loads its own kernel modules on linux as well, not much different from how it works under Windows.

Flatfire ,

What are you, an apostle? Lol. This issue affects Windows, but it’s not a Windows issue. It’s wholly on CrowdStrike for a malformed driver update. This could happen to Linux just as easily given how CS operates. I like Linux too, but this isn’t the battle.

PythagreousTitties ,

🙄 and then everyone clapped

5redie8 ,

Yeah it’s all fun and games until you actually convince someone and then you gotta explain how a bootloader works to someone who still calls their browser “Google”

Sam_Bass ,

Good ol microsloth

qjkxbmwvz ,

As much as it pains me to say it, it’s not really Microsoft at fault here, it’s CrowdStrike.

kaffiene ,

MS should have done testing as well. They can’t dodge this

qjkxbmwvz ,

Oh for their cloud services absolutely, you’re right.

candybrie ,

I’m used to IT doing a lot of their work on the weekends as to not impact operations.

badbytes ,

Stop running production services on M$. There is a better backend OS.

ILikeBoobies ,

There’s a better frontend OS

Doesn’t mean people want to go away from what they know

wizardbeard ,

There’s a shit ton more reasons than that, but in short: I highly doubt anyone suggesting a company just up and leave the MS ecosystem has spent any considerable amount of time in a sysadmin position.

ILikeBoobies ,

You’ll find xp in use because they don’t want to pay for a new system

And Linux/BSD is way more expensive because not as many people are familiar with it

dan ,
@dan@upvote.au avatar

The issue was caused by a third-party vendor, though. A similar issue could have happened on other OSes too. There’s relatively intrusive endpoint security systems for MacOS and Linux too.

Swarfega ,

That’s the annoying thing here. Everyone, particularly Lemmy where everyone runs Linux and FOSS, thinks this is a Microsoft/Windows issue. It’s not, it’s a Crowdstrike issue.

uis ,

Everyone, particularly Lemmy where everyone runs Linux and FOSS, knows it is a Crowdstrike issue.

stringere ,

More than that: it’s an IT security and infrastructure admin issue. How was this 3rd party software update allowed to go out to so many systems to break them all at once with no one testing it?

Malfeasant ,

Bingo. I work for a small software company, so I expect shit like this to go out to production every so often and cause trouble for our couple tens of thousands of clients… But I can’t fathom how any company with worldwide reach can let it happen…

SapphironZA ,

That’s because cloudstrike likely has significantly worse leadership compared to your company.

They have a massive business development budget though.

wizardbeard ,

From what I understand, Crowdstrike doesn’t have built in functionality for that.

One admin was saying that they had to figure out which IPs were the update server vs the rest of the functionality servers, block the update server at the company firewall, and then set up special rules to let the traffic through to batches of their machines.

So… yeah. Lot of work, especially if you’re somewhere where the sysadmin and firewall duties are split across teams. Or if you’re somewhere that is understaffed and overworked. Spend time putting out fires, or jerry-rigging a custom way to do staggered updates on a piece of software that runs largely as a black box?

Edit: re-read your comment. My bad, I think you meant it was a failure of that on CrowdStrike’s end. Yeah, absolutely.

kaffiene ,

It’s a MS process issue. This is a testing failure and a rollout failure

HelloHotel ,

Windows is imfamous for a do-it-yourself install process, they are likely using their own deployment tools. If anything, criticize them for not helping the update process at all.

wizardbeard ,

This had nothing to do with MS, other than their OS being impacted. Not their software that broke, not an update pushed out by their update system. This is an entirely third party piece of software that installs at the kernel level, deeper than MS could reasonably police, even it somehow was their responsibility.

Thid same piece of software was crashing certain Linux distros last month, but it didn’t make headlines due to the limited scope.

kaffiene ,

My bad i thought this went out with a MS update

dan ,
@dan@upvote.au avatar

Microsoft would never push an update on a Friday. They usually push their major patches on Tuesdays, unless there’s something that’s extremely important and can’t wait.

polle ,

Its an snakeoil issue.

Scrollone ,

Many news sources said it’s a “Microsoft update”, so it’s understandable that people are confused.

Also, there was an Azure outage yesterday.

stringere ,

Crowdstrike did the same to Linux servers previously.

AuntieFreeze ,

Webroot had something similar ish earlier this year. Such a pain.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •