Several crises, malfunctions at Duke Energy led to rolling blackouts on Christmas Eve, utility officials tell state regulators

Three power plants malfunctioned when instrumentation lines froze. Frigid temperatures hampered the operation of several nuclear units. Energy demand forecasts failed. Nearby utilities in other states, struggling to keep their own customers warm, had no power to sell.

On Christmas Eve, a multiverse of mishaps prompted Duke Energy for the first time in state history to inflict rolling blackouts on thousands of North Carolinians. More than 1,300 megawatts, equivalent to 3.6% of Duke Energy’s generating power in North Carolina was subject to “load shedding,” as it’s known in utility speak. 

Those outages snuffed out holiday lights, collapsed inflatable yard reindeer, and transformed hot, aromatic ovens into dark tombs. What’s worse, Duke Energy notified its customers far too late. In most cases residents didn’t know the blackouts were imminent until their TVs cut out in the middle of the Hallmark movie.

On Tuesday, 10 days after the energy and public relations disaster, the North Carolina Utilities Commission summoned Duke Energy to explain itself.

The inclement weather was not a surprise, Duke Energy officials acknowledged. Throughout the week prior, the utility’s own meteorologists tracked Winter Storm Elliott, while the Weather Channel documented its devastation in wall-to-wall coverage of blizzard porn: cities buried under feet of snow whipped by high winds, dozens of people dying, temperatures plummeting as much as 30 degrees in as many minutes.

The utility also knew of the disaster’s inauspicious timing: Christmas weekend, when every bulb, TV, stove, holiday decoration and electrical appliance is maxed out. When every department store and grocery is lit, vying for shoppers’ last dollars.

“We believed we had sufficient power,” Kendal Bowman, president of Duke Energy’s North Carolina operations, told the Utilities Commission – not only to meet peak demand but with more than 2,500 megawatts of reserve – wiggle room.

But as the storm bore down on North Carolina on the evening of Dec. 23, data showed a “divergence between actual load and the modeling forecast,” Bowman said. Nonetheless, as late as 10 p.m. utility officials believed there was still 900 to 1,100 MW of reserve in each of its two North Carolina territories.

They were wrong.

Energy demand, shortage threatened stability of the Eastern Interconnection power grid

T he events of early morning Dec. 24 showed how widespread disaster had threatened the grid, not only for North Carolina, but also throughout the region.

A few hours after midnight on Christmas Eve, as temperatures dropped into the single digits and demand “grew faster than models showed,” Bowman said. “The power we purchased from out of state didn’t show up, and the hard truth was that the demand would eclipse capacity.”

The first domino fell in Duke Energy Carolinas territory, which serves 2.5 million residential, commercial and industrial customers. Starting at midnight on Christmas Eve, utility officials cut back power at the Dan River combined cycle plant, which runs largely on natural gas, to 360 MW, roughly half of its capacity, said Sam Holeman, vice president of transmission. (This is also known as derating.) 

Some of the plant’s instrumentation had frozen, and to prevent the facility from failing altogether, operators had to reduce the strain. The Buck plant in Salisbury encountered low pressure issues and had to be derated after peak energy usage had passed.

Solar energy “performed as expected,” Duke officials said, although it was not available overnight during the peak hours of 2 to 6 a.m.

Another 750 MW of power that Duke Energy had bought was cut, and an additional 250 MW in “non-firm” but planned purchases, never materialized.

Meanwhile, in Duke Energy Progress territory, the crisis unfolded at 2 a.m. Energy output was reduced at two coal-fired plants: Roxboro No. 3 unit, followed by Mayo No. 1. Although insulated for frigid temperatures, instrumentation had nonetheless frozen at both units. And Duke couldn’t buy power anywhere, from anyone.

By 6 in the morning, as early risers began to brew their coffee, energy demand continued to ramp up. The ACE (Area Control Error), essential to a stable regional energy grid because it gauges the balance of electricity generation was “not in a desired state,” said Scott Batson, senior vice president and chief distribution officer.

In fact, the situation was dire. Duke deployed an emergency alert system, as required by the North American Energy Reliability Corporation, which oversees the integrity of much of the grid.

“If you don’t take steps to reduce load, then there could be an “uncontrolled loss of the system,” Holeman said – a massive, multi-state blackout across the Eastern Interconnection grid. “Every system operator has an obligation to the broader interconnection.”

At 6:14 a.m. rolling blackouts in Duke Energy Carolinas territory began. Customers in Duke Energy Progress territory had a bit more time; their blackouts began at 8 o’clock.

The snafu was exacerbated by a faulty automated “load shed” tool that sets outage durations and prioritizes circuits to temporarily be taken offline. Duke has been using the tool since 2017, Batson said, and tested it as recently as Dec. 12, 2022.

At 6:45 a.m., 234 circuits had been taken offline. Twenty-five minutes later the automated tool stopped responding, requiring utility workers to manually reduce the loads.

At 7:35, the Energy Control Center tripped two transmission lines to maintain the integrity of the grid; by 8 a.m., 269 circuits had been taken out of service – manually.

Residential and small industrial/commercial customers are cut first. Power is shut off to critical infrastructure, such as hospitals and nursing homes, only as a last resort, and those facilities often have stand-by generators.

But the process of restoring power was also fraught, according to Duke’s presentation before the commission. For example, operators had to identify circuits that had gone down because of wind, then ensure they had been repaired before re-energizing them. The 269 circuits had to be restored in sections and gradually to avoid stressing the grid.

Power to customers subject to the blackout was restored by 4 p.m. Christmas Eve, and those who had lost power because of storm damage had also been restored. Still, customers were asked to conserve energy on Dec. 25 and 26 to prevent another crisis.

“Your description of the events are concerning and scary,” Utilities Commission chairperson Charlotte Mitchell said, especially the risk posted to the larger Eastern Interconnection system. “We need to understand what happened here to protect against this happening again.”

Communication breakdown

Tens of thousands of customers were already out of power because of storm damage, not blackouts. Yet as Duke workers scrambled to prevent a wholesale collapse of the North Carolina energy grid, the general public was caught off-guard by the blackouts.

Bowman told the utilities commission that the Duke Energy is “re-evaluating its communication strategy.”

Because of the rapidly evolving situation, Duke didn’t dispatch its communication plan until after the blackouts started, Batson said. This included social media, traditional news media, the utility’s mobile app and a large message on the outage map – provided a customer could access any of those resources without power.

Customers received no advance warning of the blackouts, Batson explained, “because it takes several hours to get the messages to the carriers before texts are sent out.”

However, in many cases the estimated restoration times were overly optimistic – initially just 15 minutes. As the updates arrived by text, phone or email – to those who knew to sign up for the messaging service – the restoration times were extended to 30 to 60 minutes.

Duke finally changed the message to “by the end of the day,” which, depending on your definition of “day” could mean 6 p.m. or 11:59 p.m. Either way, on Dec. 24, the sun has set and it’s dark.

Energy planning models perplexed by confluence of events

The low temperature on Christmas Eve – 10 degrees in Raleigh and 9 in Charlotte — were far from the coldest ever recorded in North Carolina. Those readings occurred in 1985, when the mercury dropped to minus 9 and minus 5, respectively.

Even as recently as January 2018, when the state was sucked into a polar vortex, it was colder, between 5 and 8 degrees in major cities. Nonetheless, there were no rolling blackouts.

Sam Holeman said that previously Duke was able to buy power from other utilities. This time, “all of our neighbors were tight,” he said.

Preston Gillespie, Duke Energy senior vice president and chief generation officer, said the utility had prepared for the recent cold snap. “We recognized the need for preparation,” he said. “With the weather approaching, all the facilities had completed their preparations. We monitored the weather forecast to ensure there was enough generation for demand.”

That included 93% of the utility’s nuclear capacity, 11 of 15 coal-fired power plants, and eight of nine combined-cycle, which use natural gas. (Twenty-three of the 55 natural gas combustion turbine units switched from burning natural gas to oil during the peak of the Christmas Eve crisis, a Duke official said.)

Modeling from three third-party services showed that the utility could meet demand. The models, said Nelson Peeler, senior vice president and chief transmission officer, base their forecasts on “known history and how the load behaves” on holidays, on weekdays, and certain months of the year.

The model was baffled by the convergence of a weekend, a holiday, high winds and low temperatures. “It didn’t predict customer behavior” – electricity demand – Peeler said.

This scenario “was not seen by this model as clearly as in the past. And there are more errors in extreme conditions,” Peeler said, adding that the model now has data to better predict demand in similar future situations.

Julie Janson, Duke Energy executive vice president and chief executive officer, apologized to the commission and customers for the crisis. The utility will cooperate with the federal and state investigators, she said. “We appreciate how significant this is. … We own what happened.”