Pieces of a puzzle
This week, I’m sharing some previously undisclosed technical details that are helping to piece the puzzle together. I have to admit, to my surprise, that some of these pieces fit remarkably well. I also address the legal and market perspectives, thanks to two experts in their respective fields who generously took the time to thoroughly answer my questions.
Long read ahead, so here's an index:
Introduction
There are quite a few people new over here, so just a brief summary: since day one after the blackout, I’ve been publishing unbiased, technical weekly analyses featuring non-public data and original research. My intention is not to speculate, but to contribute verifiable data and context; a premise I’m still firmly following.
From the beginning, I’ve tried to approach these analyses optimistically, assuming that all actors involved were initially committed to providing accurate data and sound statements. Don’t get me wrong, I knew that probably wasn’t going to be the case, but I’ve kept that approach and let any inconsistencies speak for themselves. In fact, in Week 3 I wrote the following: “The vibe I'm getting is that everyone is being very careful about what they say, as it could eventually be used against them in court. To be honest, unless there's a breakthrough in the investigation, this whole issue is likely to drag on into a long legal battle.”
The legal battle
The legal battle, which is quite different from the technical one, is now starting to unfold in a big way, as Iberdrola is asking the Spanish Supreme Court to invalidate the legality of the working groups created by the Spanish government to investigate the blackout.
I can’t overlook the fact that the legal perspective will likely become more relevant in this issue, to the detriment of the technical one. Although I’ll continue to approach my analysis from a cyber-physical perspective, I thought it would be interesting for readers to also include the opinion of a legal expert with a background in cybersecurity. I would like to thank C. María Sierra for kindly responding to my questions.
First off, although the case is currently in the Spanish courts, I was curious about its potential trajectory.
"It’s likely that the case will begin and initially progress within the Spanish jurisdiction, given that the events took place on national territory and involve Spanish infrastructure and actors. However, if there are allegations of fundamental rights violations, lack of independence in the investigation, or conflicts with EU regulations (such as the Directive on Security of Network and Information Systems – NIS), the case could escalate to the Court of Justice of the European Union (CJEU).
Furthermore, if any of the parties involved (for example, an energy company or a group of affected citizens) believes that national regulations are being misapplied or contradict EU law, they could submit a preliminary ruling request to the CJEU, which would open the door to a European-level review. The European Court of Human Rights (ECHR) could also intervene if violations of rights such as access to public information, effective judicial protection, or the right to a fair trial are alleged."
I couldn't miss the opportunity to ask whether there was any legal room for the Spanish government to publish a more transparent report, instead of the 'non-confidential' version, which was published in a heavily redacted form (including censured graphs and technical data...) at the request of multiple companies involved. Well, it seems there was, as she explains:
"Yes, there was legal room to publish a more transparent report. The Spanish Transparency Law (Law 19/2013) establishes the right of access to public information, including technical reports and administrative documents. Although the law allows limiting publication for reasons of security, commercial confidentiality, or data protection, these restrictions must be applied in a proportional and justified manner.
Removing graphs and technical data, if it does not directly affect industrial secrets or critical infrastructure, seems excessive and legally questionable, especially if it prevents public or parliamentary oversight. In cases like this, the Transparency and Good Governance Council could be called upon to assess whether the censorship applied was legitimate.
From a legal standpoint, transparency in how an energy crisis is managed is part of the collective right to a safe environment and accountable administration. Therefore, a more complete publication could have been possible with appropriate safeguards."
Finally, something that really concerns me is whether this 'data war' we have witnessed between all the actors involved might have been a limiting factor in the investigation. I asked whether, at the European level, this situation will, or should, lead to legislation aimed at streamlining and ensuring thorough investigations of this kind in the future.
"Absolutely, the lack of access to key data in critical investigations like this represents a systemic threat to democratic oversight and energy security. At the European level, directives already exist regulating data exchange between operators and institutions, such as the NIS2 Directive (2022/2555), but cooperation mechanisms remain weak or slow in emergency contexts.
This case could drive reforms on two fronts:
Therefore, yes: this conflict should motivate clearer European legislation on obligations for cooperation, exchange of technical data, and independent external audits."
CNMC is coming
To keep everyone focused, I think it’s worth reviewing the number of official reports that are currently available:
The next major development will be the report from the Spanish market regulator, the CNMC, which holds the legal authority to demand data from all actors involved. This contrasts with the government’s investigation, which relied primarily on data that was shared voluntarily. The CNMC entity seems to be approaching its investigation from five key perspectives:
It’s worth highlighting point 2. Since week 7, when I published an analysis based on telemetry data from thousands of residential and small-scale inverters provided by an anonymous but reliable source, I have been considering that something potentially significant occurred in the distribution network, specifically involving Distributed Energy Resources (DER). Let's keep in mind that an unexpected demand increase of 1.2 GW, plausibly linked to DER entering momentary cessation, occurring 90 minutes before the blackout, officially remains unexplained.
With that in mind, let’s dig a little deeper into point 2 and explore how it might relate to the other areas of investigation.
Follow the Map.
Let’s get our bearings with a map first.
The Spanish power grid is not homogeneous, neither its distribution nor its transmission network. There are profound differences intrinsically related to other factors such as geography, demography, services, level of industrialization, etc. Oversimplifying, we can say that there is a significant asymmetry between the north and the south.
The map above shows Spain’s transmission network, highlighting the high-voltage circuits (red for 400 kV, green for 220 kV) and their corresponding substations (though not all belong to REE). As you might expect, substations are more concentrated around highly industrialized (historically linked to synchronous generation) and urban regions.
Let’s put two additional maps on the table: the one on the left comes from the AELEC report and depicts the percentage of synchronous generation at 11:00 AM on the day of the blackout. On the right, you can find the map I presented in Week 7 (days before AELEC’s was published), based on the anomalous patterns detected in the inverters’ telemetry dataset mentioned earlier.
Unsurprisingly, both maps complement each other. As you can see, regions with lower synchronous generation tend to have a higher percentage of anomalous patterns detected in DER (mainly due to inverters entering momentary cessation). The most relevant example of this asymmetry is the comparison between Catalonia and Andalusia, both of which have the highest penetration of rooftop (self-consumption) solar inverters. However, despite voltage oscillations being detected throughout the territory, in Catalonia less than 10% of DER behaved anomalously, while in Andalusia this figure rose to almost 30%. Keep this in mind, as I’ll return to it later.
A Glimpse into Distribution-Side Voltages
As I mentioned in Week 7, I was initially unsure how the AC voltage values from the inverter’s telemetry dataset were calculated. Therefore, in my initial analysis, I chose to focus on identifying anomalous patterns instead. Now that these patterns have been shown to correlate with voltage oscillations and align with the findings in the official reports, I decided to explore the AC voltage values sensed by the inverters more closely to see if I could extract additional insights.
Specifically, I calculated the distribution of voltages (binned in 1V increments) within the 210–270 V range, focusing on the timeframe from 10:00 to 12:00. This period was deliberately chosen to exclude the ‘last stage’ (or, depending on your perspective, the ‘initial stage’) of the blackout. To highlight the asymmetry mentioned earlier, I present the results for Andalusia and Catalonia. Let’s delve into the details.
The Gaussian fit just represents a practical example of the Central Limit Theorem. I marked two reference lines: the 246 V line, representing the +7% limit of the low voltage regulation (230 V) for DSOs in Spain, and the 253 V line, corresponding to the +10% limit established by the European norm.
Unsurprisingly, there is a notable difference between Andalusia and Catalonia. The AC voltage sensed by thousands of rooftop solar inverters in Andalusia was outside the regulatory limits approximately 20% of the time, while in Catalonia this occurred only about 6% of the time. It’s important to note that for a comprehensive analysis, a control dataset (AC voltage values from other days ) would be necessary to determine whether these results are systemic or specifically linked to grid conditions prior to the blackout. That said, it is reasonable to interpret the data assuming the latter scenario.
The distribution network in Catalonia is characterized by a more urban topology and higher population density, supported by a strong grid infrastructure. Conversely, Andalusia’s distribution network is more rural, with significantly lower population density and generally weaker grid conditions. Therefore, when voltage disturbances began, allegedly in the south, Catalonia’s electrical zone (solid Q absorption) and inner grid structure provided greater resilience to these fluctuations.
This naturally raises another important question: when and why did the oscillations appear?
The ‘mystery’ of the oscillations
None of the available reports address what initially triggered the oscillation modes detected.
Let’s recall that two main frequency oscillations were observed that day: the well-known inter-area oscillation (east-center-west) at 0.2 Hz, and the unknown, anomalous, local, and forced oscillation at 0.6 Hz.
The curious thing is that, from 12:00 onward, both oscillations appear to be closely connected. This connection is acknowledged in official reports from REE, the Spanish Government, and ENTSO-E (though the ENTSO-E report is not yet official, there is a dedicated webpage).
“The first one took place from 12:03 to 12:07 CEST. Preliminary analysis of the available information indicates that this was a local, forced oscillation (i.e., induced by an external source), with a dominant frequency of 0.64 Hz, primarily affecting the Spanish and Portuguese power systems. As shown in Figure 5, the forced oscillation also excites the inter-area East-Centre-West mode (0.21 Hz), albeit with a smaller amplitude.” ENTSO-E
The second oscillation follows a similar pattern: first the 0.6 Hz mode appears (at 12:16), followed by the 0.2 Hz mode (at 12:19).
But what happened before 12:00? Pretty much the same. From 10:00 onward, whenever the 0.2 Hz oscillation (red) appears, the 0.6 Hz oscillation (green) is present as well. The following graph from the Spanish government report illustrates this.
But what event originally excited these oscillations? That remains unknown. However, according to the official reports, two relevant events occurred at 10:00 and 11:00. Let’s analyze them.
What happened at 10:00?
Everyone seems to agree that the voltage began to fluctuate atypically at 10:00. However, the root cause remains unclear. The AELEC report mentions that the reduction of the number of combined-cycle power plants connected to the grid at 10:00 coincides with the start of the fluctuations. However, since their report is essentially just slides, it’s not entirely clear to me whether they are claiming a causal relationship between these two events (as a native Spanish speaker, from my point of view, the way the headline is written seems to suggest this option)
If that were the case, I don’t find this reasoning convincing, as the following graph from a previous slide in the same report seems to contradict the claim.
It shows that a significant number of these combined-cycle power plants had been progressively decoupled before 10:00 (since sunrise), in favor of PV generation. This is also supported by publicly available generation data from REE.
Meanwhile, the observed fluctuations only began abruptly at 10:00. This is something AELEC itself revealed in a graph published on May 20.
On the other hand, there was a much clearer and more sudden event that occurred exactly at 10:00: a massive deviation of photovoltaic (PV) generation from the expected production. According to the Spanish government report, this event can be plausibly explained by market dynamics:
"According to this analysis, the deviation at 10:00 could be explained by a deviation in photovoltaic production between 9:55 and 10:05 of around 900 MW (photovoltaic production was scheduled to increase by nearly 2,000 MW at that time), the most likely cause being the reduction in the marginal price.”
Since photovoltaic generation was operating with a constant power factor for voltage control, a drop in PV output also meant a reduction in reactive power absorption, further affecting voltage stability. So, nearly 1 GW of PV generation in just a few minutes is something potentially relevant. As noted in the REE report (p. 8), while such behavior may be consistent with market signals, it presents a real challenge for maintaining grid stability
"From an electrical standpoint, these abrupt changes in inverter-based generation create significant imbalances in the system[...]"
In this regard, I sought the perspective of an energy market expert and reached out to Andrew W. Thompson for his insights, who kindly agreed to respond to my questions.
He remains cautious, as many details are still unknown, but when narrowing the analysis to what is known, the observed events appear consistent with regular market dynamics. For instance, regarding the marginal price explanation for the deviation of PV generation at 10:00, he explains: "makes sense because that is when the Spanish spot price went to zero (it was 3,52 EUR/MWh previously at 9:00 AM) and became negative in the hours following. This situation is quite common in the middle of the day Spain due to all the excess solar, where negative prices mean that generators would be paying someone to take their generation. Rather than pay to generate, solar/wind plants will often curtail their output instead (but not always, depending on how they are hedged)".
Coincidentally, shortly after these voltage fluctuations begin at 10:00, the first wave of DER disconnections (momentary cessations) in Andalusia starts to unfold (according to my analysis).
For reasons that remain unclear, this pattern of disconnections repeats roughly every 30 minutes, with each occurrence growing in magnitude. Also coincidentally, REE traces the beginning of the 0.6 Hz oscillation to 10:30 (REE report, p. 6).
While the 10:00 deviation appears consistent with market signals, the anomalous deviation at 11:00 remains unexplained, as the government report explicitly notes
What happened at 11:00?
I previously elaborated on this. In the image below, you can see four distinct elements:
On the left:
The graphs I generated show the number of DERs entering momentary cessation in Andalusia (top) and Catalonia (bottom), illustrating the asymmetry I’ve discussed throughout the publication. Although the absolute numbers remain significantly higher in Andalusia, it’s evident that the event at 11:00 also had a substantial proportional impact on DERs in Catalonia, even exceeding the disruption caused by the 'chaos' after 12:00. Something to take note of.
On the right:
The top graph, taken from the AELEC report, displays voltage fluctuations at a specific transmission-distribution interface node in Zaragoza. The same pattern seen in the previous graphs is clearly identifiable here. The bottom graph, from the government report, shows an anomalous increase in demand that coincides with this drop of DER generation.
Given the granularity (minutes) of the telemetry dataset that was shared with me, I cannot determine whether this was a cause or a consequence. However, it is clear that something in the transmission network significantly affected the distribution network, or perhaps the other way around. Hopefully, the CNMC’s investigation into the distribution network will provide a definitive explanation.
In this context, it's worth highlighting a detail from the government report (pg. 85–86, although redacted I think that the telemetry used in the analysis originates from Ingeteam): the report notes that in low-voltage inverters, voltage-related alarm spikes were observed around 12:10 and 12:27 (which also matches my analysis), coinciding with a drop in total generation from these inverters. In high-voltage inverters, similar voltage alarms occurred at the same times, however additional alarms were detected around 12:16, coinciding with the reappearance of a 0.6 Hz oscillation.
As a reminder, according to REE, the source of this 0.6 Hz oscillation is one of Europe’s largest photovoltaic plants: Iberdrola's Núñez de Balboa
What happened at Núñez de Balboa?
This is likely one of the key elements of the blackout that remains completely unexplained.
REE’s report states: “The other plant that evacuates power to the transmission grid through the same interconnection facility, as well as others connected at nearby substations, have been reviewed, and the only one exhibiting oscillations was the one indicated.”
‘Calzadilla B’ is the other photovoltaic plant that injects power into the grid via REE’s “Bievenida” substation (400 kV). No oscillations were detected at this plant.
The 'Bienvenida' 400 kV node has a total available capacity of 541 MW, shared between Calzadilla B (150 MW) and Núñez de Balboa (391 MW).
At 10:11, REE ordered a curtailment of photovoltaic generation in Extremadura to prevent overloads (Government report, p. 22). There is a chance that Núñez de Balboa complied with this order. According to REE, the plant began to oscillate at 10:30, though initially with low amplitude.
At 12:03, within seconds, Núñez de Balboa’s power output began oscillating with an estimated peak-to-peak amplitude of 70% (Government report, p. 76). Based on REE’s data, the plant was producing 250 MW at 12:15, which implies oscillations of approximately 175 MW peak-to-peak. It doesn't look good.
After 12:15, the plant increased its output from 250 MW to 350 MW (89.7% of its nominal capacity). While active power (P) stabilized, reactive power (Q) continued to oscillate, potentially indicating that the voltage controller (operating under a constant power factor) was struggling to keep up with the grid.
Núñez de Balboa uses Power Electronics' Freesun HEC V1500 inverters, integrated with DNV’s GreenPowerMonitor PPC/SCADA system.
While I will likely dedicate a specific analysis to these systems later, it seems clear that a combination of the analysis of these products, REE’s data and Iberdrola’s internal data, should be sufficient to fully understand, and replicate, the behavior observed during those critical hours.
Conclusions
I’ve included another map for this final section. It’s an official map from REE showing the Short-Circuit Ratio (SCR, one of the key parameters, including its different variants, used to define grid strength) across the 400 kV transmission network (as of 2023). However, I’ve added a detail I haven’t seen mentioned in any official reports or publications: the substation where the cascading effect began (S1 Huéneja, see The Granada Incident – II) is located in one of the areas with the lowest SCR in Spain, very close to the absolute minimum (Baza).
Perhaps it’s a coincidence, but in light of everything we’ve seen, I’d argue it’s more likely a consequence. However, I’m not an expert in power systems, so I won’t draw any conclusions on my own, as they would carry little to no value.
That said, when piecing the information together, a plausible scenario emerges: the dynamic behavior of IBR (including specific plants) and DER during grid disturbances, combined with systemic and specific issues in voltage control (power factor for renewables, improper reactive absorption by certain synchronous generation units, and a synchronous must-run operating close to its lower limit) left the transmission and distribution networks in an oscillating state that ultimately caused the entire power grid to collapse, coincidentally it began at the weakest point of the grid. For those interested, I recommend “Diagnosis and Mitigation of Observed Oscillations in IBR-Dominant Power Systems” (especially pages 80–84).
There remain many details and unknowns to clarify, along with data I believe should be made available to both the relevant authorities and entities investigating this complex event, and eventually to the public. I hope everyone stays committed to transparency with facts, rather than words.
The only way to truly prevent similar scenarios and address the engineering challenges ahead, including the ability to assess whether such events could be replicated through cyber means, is to fully understand what happened.
Technical Leader at Mandiant, Google Cloud Security
2moI believe you are homing in on the root cause even though you are not a forensics electrical engineer... The models and types of inverters along with their control inputs and outputs, the configuration settings, and replicating their behavior will need to be verified by the forensics engineer. This normal process takes months to generate a forensics report, so this is why it's not out yet.
Yesterday (July 16) ENTSO-E updated its dedicated webpage with important remarks and new information: "[...] Such cascading voltage increases have never before been linked to a blackout in any part of the European power system. If confirmed, this high-voltage blackout mode will require a thorough analysis and investigation by all power system experts of the ENTSO-E community." ENTSO-E is considering the "increased reactive power injection from distribution systems" as a potential key factor. https://www.entsoe.eu/news/2025/07/16/28-april-blackout-in-spain-and-portugal-expert-panel-releases-new-information/
Senior Energy Markets Consultant at N-SIDE | Electricity, mathematics & markets
2moThis is great work, Ruben! One thing that shocks me is that, despite the challenges caused by large increases in PV generation during the morning ramp (partly driven by a wholesale market with hourly granularity), REE has recommended postponing the move to 15-minute intervals in the day-ahead market, which is planned for October 2025 (see https://montelnews.com/news/8edd8073-ea72-4e20-a843-4101e707fedd/15-minute-trading-not-convenient-or-necessary-spain-tso). The switch to 15 minutes was precisely designed to better cope with the morning ramps of solar and prevent abrupt changes between settlement periods.
Thank you very much for the analysis
Energy Economist at The Brattle Group
2moAnother excellent addition Ruben!