Some believe the Blaster worm helped trigger the 2003 blackout, and that the cyber angle was buried. The lights went out across eight states and Ontario three days after the worm hit.
Three days after the worm
Ars Technica recalled how the outage across the Northeast and part of Canada, lasting almost 16 hours, set off a wave of speculation: lightning, a power plant fire, and cyberterrorism by the Blaster worm, which had appeared only three days earlier. At the time, Ars wrote, the worm seemed a likely possibility.
There was even a claim of responsibility. Ars noted reports that al-Qaida had claimed the attack after the fact. In April 2004, the Security Working Group of the US-Canada task force reported no evidence that malicious actors caused or contributed to the outage, and no evidence that worms circulating at the time affected the companies' generation and delivery systems. Ars added, half-joking, that it still liked the theory that the blackout was Microsoft's way of dealing with Blaster.
https://arstechnica.com/uncategorized/2004/04/3611-2/
The control room went blind
The official investigation's chapter on how the blackout began describes a utility that lost sight of its own grid. At 13:31 EDT, FirstEnergy's Eastlake 5 generating unit tripped. Shortly after 14:14, the alarm and logging system in FirstEnergy's control room failed, and it was not restored until after the blackout. After 15:05, some of FirstEnergy's 345-kilovolt lines began tripping as they sagged into overgrown trees, and operators had no working alarms to tell them.
The report also sets out the machine at the center of it. The energy management system in FirstEnergy's Ohio control center was a GE Harris XA21, brought into service in 1995. Apart from minor patches, the last major update was in 1998, and on August 14 the system was not running the latest release. A grid that served tens of millions depended on alarm software that failed without making a sound.
https://ferc.gov/sites/default/files/2020-05/ch5.pdf
A worm in the same company
Weeks after the outage, Linux.com reported on the congressional hearings and why the worm question would not go away. Transcripts of phone calls showed at least one FirstEnergy technician could not answer neighboring grid operators because a computer problem kept him from seeing the status of his systems. Rep. Edward Markey reportedly said FirstEnergy had been hit by Blaster days before the shutdown and asked the CEO, H. Peter Burg, whether the company's computers were protected. Burg replied that its grid-control computers were not susceptible to Microsoft viruses.
The article then pointed to reporting by Kevin Poulsen showing that another Microsoft worm, Slammer, had already crashed two Unix-based control systems at FirstEnergy's Davis-Besse nuclear plant earlier that year. A contractor's unprotected connection inside the firewall let the worm in, and the traffic overwhelmed the plant's process computer and safety display system. If a worm could reach a nuclear plant's control network, believers ask, why not the utility's control room?
https://www.linux.com/news/did-blaster-worm-play-role-august-14-blackout/
The security expert who was not convinced
In December 2003, Bruce Schneier, one of the best known names in computer security, wrote for Salon that he was not satisfied. He noted that the interim report blamed an unlucky series of failures. The computer hosting FirstEnergy's alarm and logging software failed, then the backup computer, then several remote-control consoles. At the same time, a status computer at the Midwest grid operator, MISO, failed too, and the report said a technician who tried to repair it forgot to turn it back on when he went to lunch.
Schneier acknowledged the report did not blame Blaster. "I'm less convinced," he wrote. "The failure of computer after computer within the FirstEnergy network certainly could be a coincidence, but it looks to me like a malicious worm." He wasn't describing a planned strike. He called cyberterrorism fears hype, but argued that worms hitting the wrong systems at the wrong moment could turn a small failure into a huge one.
https://www.schneier.com/essays/archives/2003/12/blaster_and_the_grea.html
A bug only found by slowing time down
In February 2004, The Register reported that a code audit had finally found a flaw in GE's XA/21 system. "It had never evidenced itself until that day," FirstEnergy spokesman Ralph DiNicola said. "This fault was so deeply embedded, it took them weeks of pouring through millions of lines of code and data to find it." When the main system failed, the backup server kicked in and failed too, swamped by the queue of unprocessed events. Operators were unaware for over an hour that they were looking at outdated information.
The Register also noted why the worm theory gained traction. SecurityFocus had reported that two systems at a FirstEnergy nuclear plant had been hit by the Slammer worm earlier in the year. FirstEnergy's answer was that a unique combination of events and alarm conditions had triggered the bug, a fault that had never shown itself in years of service until the afternoon the grid fell.
https://www.theregister.com/on-prem/2004/02/12/software-bug-contributed-to-blackout/1521839
Thirteen cyber recommendations for a blackout with no cyber cause
A longer Register investigation in April 2004 explained how the bug was found. GE Energy's team went through roughly a million lines of C and C++ in the alarm routine and could only reproduce the crash in its Florida lab after deliberately slowing the system down. About eight weeks after the blackout, they identified a race condition: two processes got write access to the same data at once, and the alarm application spun into an infinite loop. Its window of opportunity was measured in milliseconds.
The same article notes that the task force included a computer security team, given the timing three days after Blaster. GE said that in the early stages nobody knew the root cause. And although the final report found no cyberattack, it included 13 recommendations aimed at protecting grid systems from intruders, after investigators found practices that created "potential opportunities for cyber system compromise." To believers, that is an odd conclusion for a blackout supposedly unrelated to hacking. Vote on what you think happened.
https://www.theregister.com/security/2004/04/08/tracking-the-blackout-bug/1431138