This, in a nutshell, was the problem Friday, and it still carries on to next week in some places:

Not only hundreds of computers stopped working because of CrowdStrike, to fix those, you had to manually get to the computer.

Not only do you have to get to these hundred computers with an IT department that has 10 or so people on call, but many of these computers are locked away in offices you don’t have access to or belong to people who happen to be in a different state or a different country.

Not only many of these computers were impossible to reach, but there were computers IT didn’t even know they had in their inventory until they got BSOD, and users who work have been remote since 2020 or were forgotten in the basement next to their red staple called the helplines until those collapsed.

Not only IT have computers they didn’t know existed, but management was (and is) in full panic mode and wanted a full report of inventory (how many computers you have vs how many are fixed) ASAP. Because IT can’t give them such a report because they don’t even know how many computers they have or are affected, management demands that IT keeps reporting every few hours for a meeting that goes for an hour between all IT departments in an organization while they are running around fixing issues.

The technical issue is always just the tip of the iceberg.

At least at my organization, most managers are IT folks themselves and have known the people they work with for years, even decades. They trust each other to do their job, and it shows. And I have to say (I wish for the day I wouldn’t have to say this and it will just be normal) that the few women who lead IT groups through all this did an amazing job. They led by example and came up with ideas that helped everyone. Happy to be working in a place like this.