Breaking point
After a second review of my previous analysis and the official reports from REE and the Spanish government, I believe the following information I am now sharing marks the point where this issue clearly goes far beyond personal curiosity. 1.2 GW of DER generation was lost within a 10-minute window, 90 minutes before the blackout, for no known reason.
1.2 GW
Excerpt from the Spanish Government report (page 101):
"The 11:00 deviation, based on the available information, cannot be explained by generation behavior, but rather correlates with an unforeseen and sudden deviation in demand, which increases by about 1,200 MW with no known cause and without resembling nearby days. This “anomalous” increase in demand, marked in the graph below, is compatible with a temporary reduction in distributed and small-scale generation (including self-consumption) on lower voltage networks."
In the following image, you can see two elements A and B:
A) The official graph from the Spanish Government report shows an anomalous 1.2 GW increase in demand, equivalent to the output of a nuclear power plant. This indicates that the generation which had been matching that demand was suddenly lost. Crucially, this lost generation did not originate from large plants connected to the transmission grid, but rather from self-consumption and small-scale industrial solar inverters. The blue circle highlights the moment demand suddenly increases. As the inverters momentarily ceased to operate due to an unknown cause, the loads began drawing power directly from the grid, so demand increased.
B) This is a segment of the graph, starting at 11:00, that I created to identify Momentary Cessation and Trip patterns in the telemetry from thousands of residential (rooftop self-consumption) and small-scale industrial inverters on the day of the blackout. This telemetry data was shared with me by an anonymous, but reliable, source.
As the reader can verify, these patterns correlate exactly, including the two-peak pattern, with the anomalous 1.2 GW increase in demand. At 11:00, there were no reports of relevant oscillations.
But this is not the only precise coincidence.
Excerpt from Spanish Government report (p. 85)
"In the inverters connected to low voltage (Graph 36), alarm spikes associated with inadequate voltage levels are detected around 12:10 and 12:27, which coincide with a reduction in the aggregate generation represented by these inverters as a whole."
Same scenario, same significant peaks at 12:10 and 12:27 in my graph. This means that, once again, hundreds of megawatts of distributed generation were lost due to inverters entering momentary cessation or tripping.
Although this dataset of inverters may represent only a small portion of the total, it constitutes a statistically significant sample. As I previously mentioned, I chose to share these findings because I was confident that, when scaled to the full deployment across Spain, the percentages and other figures I’ve presented would remain consistent. I believe the figures for distributed generation losses included in the Spanish government’s report confirm this.
It's important to note that I created this graph and shared it with the relevant Spanish authorities many hours before the official government report was published, as I believed it could be relevant to the investigation.
For additional context, I previously explained the approach and background of my analysis in the following posts:
https://www.linkedin.com/pulse/report-season-here-ruben-santamarta-big6f
https://www.linkedin.com/pulse/quest-initiating-event-ruben-santamarta-kthlf
Conclusions
I believe there is sufficient evidence to justify an assessment by the relevant authorities, as no power grid incident of this scale has been reported in any other country.
I strongly believe that the relevant authorities should urgently require immediate access (if this hasn’t already been done) to the logging data from all inverters deployed across Spain.
This is essential to investigate (under any conceivable scenario) how it was possible for 1.2 GW of residential (rooftop self-consumption) and small-scale industrial photovoltaic generation to 'disappear' within a 10-minute window, just 90 minutes before the blackout, with a similar loss occurring just minutes before the total collapse of the system.
Pioneering OT Cybersecurity for the Sustainable World | SaaS Innovator | Global Speaker | MSc Cyber | Leadership Coach
2moRuben I'm with you
Head of Energy at Zühlke | Co-Founder wattz AG | Lecturer | Energy & CleanTech Leader | Strategy, Sales & Transformation
3moThanks for sharing, Ruben, based on Lab Simulations ZHAW School of Engineering Petr Korba and Artjoms Obusevs it looks like 'wrong' settings of inverters: PV inverters behave 'incorrectly' in Spanish mode (they all switch off at 50.2Hz for about 10min - better would be continuously reduction of active power). A learning to take jnto consideration for all ENTSO-E countries and Eurelectric for #EnergySecurity.
Hacker, manager & public speaker at BV cybersecurity
3moThx for the thorough analysis Ruben Santamarta. Looking forward to see what the post mortem analysis will say about the final root cause.
R&D-ENERGY-DISTRICT HEATING-BIOMASS-PARTICLEBOARD-MDF-PLYWOOD-FURNITURE-SAWMILL-PALLET BLOCKS-PELLET-PHOTOVOLTAIC-CHP-COGENERATION-ETS CO2-CARBON CAPTURE-HYDROGEN-E-FUELS-E-METHANOL-P2X-POWER MONITORING-IOT
3moRuben Santamarta we have to understand and investigate if these trippings are cause or consequence. I mean, these trippings were caused by voltage oscillacions or they caused the voltage oscillacions??? In my poor opinion, looks like they were caused by the previous voltage oscillations but they, as mentioned by REE, was the cause of reducing the flows to France and then increasing again the reactive power to be compensated. Good job and nice analysis.
Industrial cybersecurity Consultant, Performed Cyber Risk Study of the ICS used in the NATO CEPS.
3moThe fiction book to read is "Blackout" by Ellsberg (based on Budestag 2011 study). There the tech that drives the scenario are smart meters. BTW from someone with solar panels a sig power gen loss results from a cloud passing overhead. In that case the tech adjusts. An adversary I suspect could find a way to fool the tech. Just like the sensors on the ill fated Boeing 737 Max sensors fooled the flight control system. BTW is there a link to the cited report in your article? Thanks