Lessons learned from the tech outage

Most of us have likely seen it at least once before. We’re in the middle of an important task, such as entering thousands of entries into an Excel spreadsheet or posting our latest update to Facebook, and all of a sudden the computer screen turns blue.

About a week ago, the screens of an estimated 8.5 million Windows computers came down with the blue screen of death. It was like watching Inside Out 2, when orange-haired Anxiety takes over our beloved Riley.

The multi-day tech flub, dubbed the “largest IT outage in history,” was reportedly prompted by a flawed software update for Microsoft devices pushed to clients by cybersecurity firm CrowdStrike late Thursday, July 18. The effects were felt worldwide, including disorder at airports, 911 dispatch outages and system failures at government agencies on Friday.

Although U.S.-based carriers American Airlines, United Airlines, Delta Air Lines, Allegiant Air, Sun Country and Frontier Airlines all reported issues on Friday, Delta was the slowest to recover.

While the outage affected computers used to check in passengers for several airlines, it also disrupted crucial software used to schedule Delta crew members.

“In particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown,” Delta CEO Ed Bastian said in the statement.

In addition, the timing was impeccable, Bastian said, taking place on what he described as “the busiest travel weekend of the summer,” with 90 percent of Delta’s flights fully booked.

Thousands of North Dakota State Fair goers were also affected. Rapper Lil Wayne was scheduled to perform in concert on Saturday, but flight disruptions pushed his show back to Wednesday.

My daughter was on her way to Minot when she heard the news. Disappointed, she turned around and drove back home. The only positive part of the trip was pizza at Rosa’s on the way home. She missed out on the concert completely, as she had other plans for Wednesday.

On Monday, a full three days after the first effects were felt, there were still hundreds of flights grounded. In fact, over 1,000 flights into, within or departing from the United States were canceled Monday, and over 9,000 were delayed, according to flight tracking site FlightAware.

While your local newspaper offices didn’t suffer any setbacks this time, an event of the magnitude we’ve just experienced is enough to rattle any tech user. So I took this event as an opportunity to study up on what went wrong and how we can learn.

After all, the soothsayers are already warning that these “outages” are likely to happen more often in the future.

So, what can we learn from the biggest tech outage in history?

Backup, backup, backup

It’s been said for decades that we should back up all our data, because hard drives and devices can and do fail with no warning. Our company invested in a server a few years ago, and I’ve been reminding staff ever since that they must keep anything that’s important or worth saving on the server. That means breaking the habit of storing files on individual desktops.

Redundancy is key here, readers. Hence why I purposely duplicated that all-important word above – “backup.” There are three copies of all the data stored on our server – and it’s available remotely, onsite and in the cloud. That way if devices fail, we can access all the server’s data from a different computer quickly and easily. If the server fails, we can recover the data from one of the backups and get back to work.

I also have an iCloud Drive account where I save projects I work on at home, which are also available on other devices in the office when needed.

Yes, we are talking about redundancy here, not the annoying kind like a coworker that repeats himself every five minutes. It may be blatantly obvious to some, but others don’t realize that redundant backups can make the best of unexpected surprises. Can you spot the redundancies in that last sentence?

In addition to redundant data storage, it’s helpful to have alternative software to complete tasks. I didn’t think about that prior to this outage, but it makes sense. For example, we have both Microsoft and Google accounts, so if one system goes down, we can use the other. If the outage is internet-related, you use desktop software such as Wordpad (PC), Pages (Mac) or OpenOffice to create and print documents.

Beware of cybercriminals

We must always stand ready for potential cyber attacks. In my business, I intercept daily attempts from cybercriminals in my inbox, online and on my phone.

As news of the outage broke across the world, malicious actors seized the opportunity to up the ante on their criminal activity.

The Department of Homeland Security issued a bulletin Friday, which was updated throughout the weekend, warning of “threat actors taking advantage of this incident for phishing and other malicious activity.”

CrowdStrike reported that cybercriminals pounced on their clients, creating fake websites, impersonating CrowdStrike employees and even selling bogus software to scam vulnerable individuals.

In other words, it was like “Night at the Roxbury’’ for the scammers. When will these greaseballs get out of their parent’s basement, get a haircut and get a real job?

The problem was exacerbated by the sheer volume of people “googling” for answers to the tech nightmare. The lesson here is to check your sources, folks. Take a few extra minutes to seek out official sources or an IT professional to help.

For more on how to prepare yourself and your business from future tech outages, check out this article by author, executive coach and mentor Dr. Aneish Kumar, “Lessons from the Recent Tech Outage,” I found on LinkedIn.