No land in sight.

It's been three weeks, and there are still more questions than answers, and it'll probably stay that way in the weeks ahead. So, I think it’s a good moment to go over what we’ve got so far, add some context, bring in a few new concepts, and put some key data on the table.

See you in court

The vibe I'm getting is that everyone is being very careful about what they say, as it could eventually be used against them in court. To be honest, unless there's a breakthrough in the investigation, this whole issue is likely to drag on into a long legal battle.

I've put together a set of relevant official statements to help readers get up to speed with the current situation. To sum up: everything was working fine until everything failed.

Can we make sense of this? I'm not sure, but at least I'll try. As I've been doing so far, my intention is not to speculate, but to contribute verifiable data and context, so that we can be better prepared when the official explanations are released.

The appearance of Sara Aagersen, Minister for Ecological Transition and the Demographic Challenge, in the Spanish parliament provided some details but these only raised more questions , which in turn prompted reactions from additional entities, generating even more questions, and so on. I've seen some non-native (spanish) speakers struggling to get the right significance of these statements, so if that's your case I comprehensively explained it in 'The Granada Incident'.

If everything was working properly, then what went wrong?

The distribution network in Badajoz and Andalusia (the regions where the generation losses were initially detected) is managed by e-Distribution, a company owned by one of the Spanish energy giants: Endesa. As you can see in the pool of statements, Endesa declared that

"Our distribution network was functioning well up until the blackout, both in Granada and the rest of Spain, and we have not opened any internal investigation regarding it."

Ok, so transmission and distribution were allegedly working smoothly. What's left then? The non-transmission network and the generation plants. To quickly grasp what this really means, I created the following diagram (based on one by REE).

Essentially, according to the official statements, we should be able to narrow down the failure, attack, or whatever (FAW) to the area inside the red highlighted box. Everything outside of it was allegedly functioning perfectly. I must also assume that the FAW propagated from inside the red box to the outside, as REE explicitly stated that all power system parameters were within normal operating ranges. It's a mind-blowing scenario.

If everything was normal, I assume that includes the infamous oscillations as well. Let's briefly go over them.

Everyone agrees that there were two long-lasting frequency oscillations before the blackout. However, there’s an important distinction: the first one was 'unknown' to the operators, while the second had been known for a long time.

The latter can be found described in REE's official document "Plan de Desarrollo de la Red De Transporte de Energía Eléctrica (Periodo 2021-2026)" (it's very interesting document), which also aligns with the Minister's remarks on the oscillations.

"East-Central-West Mode: The generators at the western and eastern edges of the system (the Iberian Peninsula and the southern Balkans and Turkey) oscillate almost in counterphase with the generators in the center of the European continent (Denmark, Germany, Poland, Switzerland, the Czech Republic, and Italy). This mode typically appears at oscillation frequencies around 0.22–0.28 Hz and is generally less well damped than the East-West mode."

So, at 12:07, the unknown oscillation was damped. 12 minutes later, the known oscillation appeared, only to be damped two minutes after that. Then, once again, 12 minutes later, everything went bananas. The 12-minute interval is probably just a coincidence, but I tend to pay attention to these kinds of things because, in cyber-physical systems, they might reveal underlying issues with operations, measurements, and cyclical calculations. For instance, see this ENTSO-E report where a frozen measurement on a transmission line led to a flawed convergence in the state estimator. Also, grid disturbances have been found to cause deterministic failures in PV plants due to specific configurations in inverters, essentially a common-cause failure. NERC describes this situation in a very interesting report: "Odessa Disturbance".

In the Week 2 analysis, I mentioned that PV generation started behaving oddly well before the oscillations began, starting around 10:30 am, with graphs showing what looked like either curtailment or a loss of PV generation during peak hours, we even had to import electricity from France! There were also unexpected imbalances. For instance, at 12:00, just before the first oscillation, we can observe a drop in PV generation. Today Endesa, EDP and Iberdrola asked ENTSO-E to extend the investigation to the entire morning of the blackout and the days leading up to it. Time will tell what's really behind this request.

Now let’s go back and see what’s inside the previously mentioned red highlighted box.

1. Non-Transmission Network.

REE does not own all components of the Spanish power grid. Basically, some substations, interconnection facilities and lines, although connected to the REE's transmission network, are not operated by REE. This is especially relevant in the context of the blackout, because it defines a physical, logical, and legal frontier between REE and non-REE assets.

Downstream of REE’s substations, there is a multitude of substations and other components and facilities owned by a constellation of companies. Moreover, the asset owners are not necessarily the same entities that operate and manage them.

However, REE maintains visibility over power flows not only for operational safety but also for market regulation purposes. For example, whenever energy is transferred between agents, at so-called boundary points, a measurement point is established to capture flow parameters and total energy exchanged. This data is managed by REE’s dedicated measurement system: SIMEL. This process also follows a decentralized model: certified third-party companies can operate secondary concentrators that collect and forward these measurements to REE. Data authenticity and integrity are, in theory, protected by cryptographically secure methods.

2. Generation Plants

It’s evident that photovoltaic (PV) plants played a significant role in the blackout, at least from a quantitative standpoint. As I elaborated a few days ago, the reported figures on both partial and total generation losses don’t align unless PV output is taken into account. Official statements also confirm this interpretation. However, this doesn’t imply that PV plants were to blame for the outage, nor does it suggest that renewables are inherently unreliable. The reality is usually far more complex. I'll then focus on this kind of plant.

One of the questions is: why did so many different photovoltaic generation plants trip almost simultaneously? Is there anything that could link geographically distant plants together beyond transmission lines? Yes, both from a logical and operational perspective, as the complex architecture of the Spanish power grid is shaped by the way its market operates.

I previously introduced programming units in an earlier analysis, but let’s dive a little deeper. There are thousands of photovoltaic plants (Physical Units) in Spain, many with modest nominal power. However, they can choose to be part of a logical entity called a programming unit, thereby collectively acting as a larger generation plant

I used REE's official data to map all the Photovoltaic programming units. The following graph shows all those that represent ≥ 100 MW. Let's elaborate on it.

The single largest cluster is owned by ENEL, which includes dozens of PV plants. However, in terms of total power, the various programming units represented by NEXUS Energia account for over 1.6 GW, making it the top player. They do not own the assets themselves but rather represent, or aggregate, dozens of small photovoltaic power plants.

They also have their own REE-authorized Control Center (CCG).

Please note that, obviously, I did not attempt to gain unauthorized access. However, the user interface is accessible without any form of authentication and was even indexed by Google, as anyone can verify. Through it, the actual owners of the assets managed by NEXUS can view everything related to their plants, including dispatch and operating instructions received from REE (if any). I strongly hope this is merely a read-only front-end, and that there are no privileged accounts capable of issuing commands to the plants via this publicly accessible interface.

We have seen how the photovoltaic industry representatives have explicitly stated that their plants were disconnected. There are two main options:

1. Protections: All generation plants (including PV) must comply with the boundary protection requirements established by REE. However, REE stated that everything in the transmission network was operating correctly, which makes it notable that many different plants tripped almost simultaneously. When any of these protections are triggered, REE (ideally) receive information about it. If that was the case and communications were reliable, REE ought to have a clear way to identify why so many plants' protections were activated.

2. REE's Generation control (including SRAP). For example, the "Motivo Consigna REE" column in the previously mentioned UI refers to the generation control instructions transmitted by REE to physical units via their respective CCGs (see Week1 analysis for reference). REE could have instructed specific plants to curtail their production using one of the predefined reasons (1–6), or directly via a fast command from the SRAP (Sistema de Reducción Automático de Potencia), which was already introduced in Week 1

On top of that, secondary regulation is also decentralized, originally through the Regulación Compartida Peninsular (RCP), which has since been replaced by the Europe-compliant Sistema de Regulacion Secundaria (SRS). It is organized through a series of regulation areas made up of Programming Units from different market participants.

And yes, PV plants do participate in the secondary regulation market through these regulation areas. As far as I know, the information about the programming units that won the bids for upward/downward reserve on April 28th is not public. Although historically Combined Cycle plants have been the main contributors, it would be interesting to know whether the PV plants that tripped were originally expected to be part of the pool for this secondary reserve, especially since it is designed to be activated roughly 20 seconds after the initial imbalance.

Conclusions

It’s becoming increasingly difficult to reconcile the official statements from companies, government, and industry actors with the technical facts. The longer we go without a clear technical explanation (which I'm sure the working groups are working hard to figure out) for the initiating event and the subsequent failures, the more complex scenarios must be considered, without entirely ruling out a cyber component (though not necessarily implying a total blackout).

If there is a serious possibility that a cyber component was involved, time works against the investigation. I assume that all logs, configurations, and every piece of data from the plants feeding into the substation where the first generation loss was detected have been collected. Otherwise, the data might be lost forever.

Finally, I’d like to thank everyone for the great feedback I’ve been receiving on this series of posts about the blackout. I’m glad people find them helpful for keeping up with this complex situation.