Reference Architecture for Electric Energy OT.pdf
- 1. Reference Architecture for Electric Energy OT and Accompanying Profiles
- 2. 2 The SEI ETF is a working group composed of senior government, industry, and nonprofit representatives, stood up by the Secretary of Energy as mandated by Sec. 5726 of the 2020 NDAA. Through review of existing reference models and architectures, the Technical Project Team tasked with evaluating technology and standards identified a gap that there is no existing commonly accepted reference architecture for ICS. To produce a new model for ICS, the TPT reviewed 16 existing models—including the Purdue Enterprise Reference Architecture and Methodology and DHS’s recommended practices for improving ICS cybersecurity through defense-in-depth strategies—and developed a cumulative list of core elements that would fill gaps to make a more broadly applicable reference architecture structure to build unique ICS profiles onto. Specifically, the TPT prioritized developing this reference architecture specifically for the electrical sector, not limited to localized industrial processes, focused on properties of information passing between devices, and flexible enough to reflect advances in technology and design practices. To this end, the TPT developed the SEI ETF Reference Architecture for Electric Energy OT (RA) from which other domain- specific profiles could be derived. Upon reaching consensus, the team further developed specific profiles for substation, generation, distributed energy resources, and operation/network control center. The RA and profiles are presented as a stack of six levels grouped into four zones. Each level contains a set of devices and systems, with the physical processes and field devices on the lowest level and a hierarchy of processes and technical controls in each level above. The profiles also include new conceptual elements: security features and participating parties assigned to each of the six levels of the model. The SEI ETF Reference Architecture for Electric Energy OT serves as a baseline for the other profiles and introduces elements common to all the profiles, such as the five columns across all levels (security level/name, typical device examples, function, security features, actors). The Generic Profile includes six security levels spread across four zones: physical assets, operations, enterprise, and public. Zones are separated by demilitarized zones (DMZs), network segments typically located between two firewalls. Moving towards security implementation, the RA and profiles can be used as a starting point for the Engineered Cybersecurity Process flow, as detailed in slide 11. 3/8/2022 Introduction
- 3. 3 3/8/2022 Reference Architecture for Electric Energy OT Level 4 – Business/ Enterprise Level Level 3 – Control Center Level Level 2 – Facility Level Level 1 – Subsystem Level Level 0 – Process level Level 5 – Internet /Cloud Level NCITs Merging Units Protection Breaker I/O CT/PT Merging Units Indicators Sensors RTU / Gateways Local HMIs Domain Name System Server Public Cloud Servers Web Servers Domain Controllers Business Servers Operator Workstations SCADA/ Application Servers Database Servers I/O Servers Domain Controller Engineering Workstations Data Acquisition, Telemetry, Process Control Physical Assets Zone Operations Zone Enterprise Zone External Communication Internal Business Communication Subsystem Controllers IEDs Monitoring Internal Operational Communication Process Data Conversion, Local Control, Asset Monitoring Physical Process Interface Private Cloud Servers DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server DMZ – Web Servers, Email Servers, Remote Access Server DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Public Zone Security Features Participating Parties Typical Device Examples Security Level/ Name Function Note: RA does not include all possible security implementations (e.g. data diode vs firewall for ingress protection
- 4. 4 3/8/2022 Substation Profile • Risk Assessment • Security Awareness • Security Training • Access Control Policies • Management and Review • IDS/IPS • Network Monitoring devices • Encryption Control • SIEM • IT Manager • Business strategy • Planning • OT Manager • SCADA • Operations & Maintenance • EMS Support • Remote Employees • OT and IT Services • Vendors • 3rd Party Service providers • OEM/vendors • Remote monitoring • Device software updates • OT Manager • Eng/Designer • Relay Tech • Field Service Tech Level 4 – Business/ Enterprise Level Level 3 – Control Center Level Level 2 – Facility Level Level 1 – Subsystem Level Level 0 – Process level Level 5 – Internet /Cloud Level NCITs Merging Units Protection Breaker I/O CT/PT Merging Units Indicators Sensors RTU / Gateways Local HMIs Web Servers Email Servers Web Servers Domain Controllers Business Servers Operator Workstations SCADA/Application Servers Database Servers I/O Servers Domain Controller Engineering Workstations Data Acquisition, Telemetry, Process Control Physical Assets Zone Operations Zone Enterprise Zone External Communication Internal Business Communication Bay Controllers IEDs Monitoring Internal Operational Communication Process Data Conversion, Local Control, Asset Monitoring Physical Process Interface Enterprise Desktops DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server DMZ – Web Servers, Email Servers, Remote Access Server DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Public Zone Private/ Utility Cloud Cloud servers Security Level/ Name Typical Devices Function Security Features Participating Parties • Access Control Policies • Device Hardening • Security Logging • Patch Management • Malware Protection • Data Integrity Protection • IDS/IPS
- 5. 5 3/8/2022 Generation Profile • Risk Assessment • Security Awareness • Security Training • IT Manager • Business strategy • Planning • 3rd Party Service providers • OEM/vendors • Remote monitoring • Device software updates Level 4 – Business/ Enterprise Level Level 3 – Control Center Level Level 2 – Facility Level (Plant/Site) Level 1 – Subsystem Level (Generating Unit) Level 0 – Process level Level 5 – Internet /Cloud Level Actuators Protection Breaker I/O CT/PT Merging Units Indicators Sensors RTU /Gateways Historian Web Servers Email Servers Web Servers Domain Controllers Business Servers Operator Workstations SCADA/ Application Servers Database Servers I/O Servers Domain Controller Engineering Workstations Data Acquisition, Telemetry, Process Control, Local Control Physical Assets Zone Operations Zone Enterprise Zone External Communication Internal Business Communication Subsystem Controllers IEDs PLCs Monitoring Internal Operational Communication Process Data Conversion, Asset Monitoring Physical Process Interface Private Cloud Servers DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server DMZ – Web Servers, Email Servers, Remote Access Server DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Public Zone Public Cloud servers Local HMIs DCS/TCS Security Level/ Name Typical Devices Function Security Features Participating Parties • OT Manager • Operators • Eng/Designer • Relay Tech • Field Service Tech • Access Control Policies • Device Hardening • Security Logging • Patch Management • Malware Protection • Data Integrity Protection • IDS/IPS • Access Control Policies • Management and Review • IDS/IPS • Network Monitoring devices • Encryption Control • SIEM • OT Manager • SCADA • Operations & Maintenance • EMS Support • Remote Employees • OT and IT Services • Vendors
- 6. 6 3/8/2022 Generation Physical Asset Zone (Expanded Profile) Level 2 – Facility Level (Plant/Site) Level 1 – Subsystem Level (Generating Unit) Protection RTU /Gateways Historian Engineering Workstations Data Acquisition, Telemetry, Process Control, Local Control Subsystem Controllers IEDs PLCs Monitoring Process Data Conversion, Asset Monitoring DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Local HMIs DCS/TCS Subsystem 1 Level 1 – Subsystem Level (Generating Unit) Level 0 – Process level Actuators Protection Breaker I/O CT/PT Merging Units Indicators Sensors Engineering Workstations Subsystem Controllers IEDs PLCs Monitoring Local HMIs DCS/TCS Subsystem 2 Level 1 – Subsystem Level (Generating Unit) Level 0 – Process level Actuators Protection Breaker I/O CT/PT Merging Units Indicators Sensors Engineering Workstations Subsystem Controllers IEDs PLCs Monitoring Local HMIs DCS/TCS
- 7. 7 3/8/2022 Distributed Energy Resources (DER) Profile Level 4 – Business/ Enterprise Level Level 3 – Control Center Level Level 2 – Facility Level Level 1 – Subsystem Level Level 0 – Process level Level 5 – Internet /Cloud Level Domain Name System Server Public Cloud Servers Web Servers Domain Controllers Business Servers Operator Workstations SCADA/Application Servers Database Servers I/O Servers Domain Controller Data Acquisition, Telemetry, Process Control, Local Control Physical Assets Zone Operations Zone Enterprise Zone External Communication Internal Business Communication Internal Operational Communication Process Data Conversion, Local Control, Asset Monitoring Physical Process Interface Private Cloud Servers DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server DMZ – Web Servers, Email Servers, Remote Access Server DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Public Zone • Risk Assessment • Security Awareness • Security Training • IT Manager • Business strategy • Planning • 3rd Party Service providers • OEM/vendors • Remote monitoring • Device software updates Private/ Utility Cloud Merging Units PMU Breaker I/O CT/PT Merging Units Indicators Protective Relays Engineering Workstations Subsystem Controllers IEDs PLCs Monitoring Local HMIs RTU /Gateways Local HMIs Automation Controllers Not captured: Parallel control architectures Engineering Workstations Net Metering Security Level/ Name Typical Devices Function Security Features Participating Parties • OT Manager • Solar farm/wind farm/hydro site operator • Relay Tech • Field Service Tech • Local Station SCADA Network Monitoring • Local Device Monitoring • Ground State Truth Side Channel Monitoring • IDS/IPS • Access Control Policies • Management and Review • IDS/IPS • Network Monitoring devices • Encryption Control • SIEM • OT Manager • SCADA • Operations & Maintenance • EMS Support • Remote Employees • OT and IT Services • Vendors
- 8. 8 DER DMZ 3/8/2022 Regional Utility Scale DER Profile Level 4 – Business/ Enterprise Level Level 5 – Internet /Cloud Level Domain Name System Server, Webserver, Email Public Cloud Servers, VPN/Remote Access Server Enterprise Users Domain Controllers Business Servers Enterprise Zone Regional Utility External Business Communication Regional Utility Business Operations/ Communication Private Cloud Servers Public Zone Historian Human Machine Interfaces Jump/Diagnostic Host Patch/Backup Server Regional Utility Operation Aggregation Security Level/ Name Typical Devices Function PV Site N DER DMZ Historian Human Machine Interfaces Jump/Diagnostic Host Patch/Backup Server Business and Operations Communication Level 3 – Control Center Level Level 2 – Local Facility Level Level 1 – Subsytem Controller Level Level 0 – Process/ Field Level Internal DER Site Operational Communication Process Data Conversion, Local Control, Asset Monitoring Data Acquisition, Telemetry, Process Control, Local Control Physical Process Interface Operator HMI Workstations SCADA/Application Servers Database Servers I/O Servers Domain Controller RTU/ Gateways Protective Relays Net Metering Engineering Workstations Automation Controllers IEDs PLCs Monitoring Subsystem Controllers Local HMIs Local HMIs Engineering Workstations Indicators CT/PT Merging Units Breaker I/O Merging Units PMU Wind Site N+1 DER DMZ Historian Human Machine Interfaces Jump/Diagnostic Host Patch/Backup Server Business and Operations Communication Level 3 – Control Center Level Level 2 – Local Facility Level Level 1 – Subsytem Controller Level Level 0 – Process/ Field Level Internal DER Site Operational Communication Process Data Conversion, Local Control, Asset Monitoring Data Acquisition, Telemetry, Process Control, Local Control Physical Process Interface Operator HMI Workstations SCADA/Application Servers Database Servers I/O Servers Domain Controller RTU/ Gateways Protective Relays Net Metering Engineering Workstations Automation Controllers IEDs PLCs Monitoring Subsystem Controllers Local HMIs Local HMIs Engineering Workstations Indicators CT/PT Merging Units Breaker I/O Merging Units PMU Hybrid Site N DER DMZ Historian Human Machine Interfaces Jump/Diagnostic Host Patch/Backup Server Business and Operations Communication Level 3 – Control Center Level Level 2 – Local Facility Level Level 1 – Subsytem Controller Level Level 0 – Process/ Field Level Internal DER Site Operational Communication Process Data Conversion, Local Control, Asset Monitoring Data Acquisition, Telemetry, Process Control, Local Control Physical Process Interface Operator HMI Workstations SCADA/Application Servers Database Servers I/O Servers Domain Controller RTU/ Gateways Protective Relays Net Metering Engineering Workstations Automation Controllers IEDs PLCs Monitoring Subsystem Controllers Local HMIs Local HMIs Engineering Workstations Indicators CT/PT Merging Units Breaker I/O Merging Units PMU
- 9. 9 Level 3.2 – Network Control and Database Level Level 3.1 – Communication Level Level 4 – Business/ Enterprise Level • OT Manager • Maintenance • OT Services • IT Services • Vendors • EMS Support • Remote Employees • Network monitoring devices • IDS/IPS • Encryption Control • Access Control • SIEM Level 5 – Internet DMZ/Cloud Level • Risk Assessment • Security Awareness • Security Training • Access Control Policies • Management and Review • IDS/IPS • IT Manager • Business strategy • Planning • OT Manager • SCADA • Operations • EMS Support • Remote Employees • 3rd Party Service providers • OEM/vendors • Remote monitoring • Device software updates Web Servers Email Servers Cloud Servers Web Servers Domain Controllers Business Servers Enterprise Desktops Operations Zone Enterprise Zone Level 3.3 – Operator Room Operator Workstations Domain Controller Engineering Workstations SCADA/ Application Servers Database Servers I/O Servers Situational Awareness Network Monitoring Historian Servers HMI/Map board Front End Processor collecting Data from the Stations and Substations and sending control signals to the Stations and Substations Comms to RC/BA Comms to Neighboring Entities Comms between Main and Backup Control Center RAS Public Zone External Communication Internal Business Communication Patch Server, Electronic Access Control or Monitoring Systems, Physical Access Control System, Vulnerability Scanner, Baseline Server, Anti-virus Server, Log Server Private/ Utility Cloud DMZ – Internal Operational Communication Control Center Profile Security Level/ Name Typical Devices Function Security Features Participating Parties 3/8/2022
- 10. 10 Distribution Substation Transmission Substation Generation Plant Level 3.2 – Network Control and Database Level Level 3.1 – Communication Level Level 4 – Business/ Enterprise Level Level 5 – Internet DMZ/Cloud Level Web Servers Email Servers Cloud Servers Web Servers Domain Controllers Business Servers Enterprise Desktops Operations Zone Enterprise Zone Level 3.3 – Operator Room Operator Workstations Domain Controller Engineering Workstations SCADA/ Application Servers Database Servers I/O Servers Situational Awareness Network Monitoring Historian Servers HMI/Map board Front End Processor collecting Data from the Stations and Substations and sending control signals to the Stations and Substations Coms to RC/BA Coms to Neighboring Entities Coms between Main and Backup Control Center RAS Public Zone Patch Server, Electronic Access Control or Monitoring Systems, Physical Access Control System, Vulnerability Scanner, Baseline Server, Anti-virus Server, Log Server Private/ Utility Cloud DMZ – Security Level/ Name Typical Devices Control Center (Expanded Profile) Level 2 – Facility Level Level 1 – Subsystem Level Level 0 – Process level Protection Breaker I/O CT/PT Merging Units Indicators Sensors RTU /Gateways NCITs Merging Units Engineering Workstations Data Acquisition, Telemetry, Process Control, Local Control Subsystem Controllers IEDs Monitoring Process Data Conversion, Local Control, Asset Monitoring Physical Process Interface DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Local HMIs Physical Assets Zone Level 2 – Facility Level Level 1 – Subsystem Level Level 0 – Process level Protection Breaker I/O CT/PT Merging Units Indicators Sensors RTU /Gateways NCITs Merging Units Engineering Workstations Data Acquisition, Telemetry, Process Control, Local Control Subsystem Controllers IEDs Monitoring Process Data Conversion, Local Control, Asset Monitoring Physical Process Interface DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Local HMIs Physical Assets Zone Level 2 – Facility Level (Plant/Site) Level 1 – Subsystem Level (Generating Unit) Level 0 – Process level Actuators Protection Breaker I/O CT/PT Merging Units Indicators Sensors RTU /Gateways Historian Engineering Workstations Data Acquisition, Telemetry, Process Control, Local Control Physical Assets Zone Subsystem Controllers IEDs PLCs Monitoring Process Data Conversion, Asset Monitoring Physical Process Interface DMZ – Historian, Backup Director, Patch Server, Remote Access/Jump Server Local HMIs DCS/TCS 3/8/2022
- 11. 11 Engineered Cybersecurity Process Flow Reference Architecture to Security Implementation Reference Architecture • Starting point used for communication and organizational awareness Reference Architecture Profiles • Templates for OT control system including adaptation for OT applications Security Concept • High level overview of the main cybersecurity objectives including definitions of security features , necessary security functions Security Design • Detailed security solution and specification of cyber assets, security features and methodology to support the security concept Security Risk Assessment • Cyber Asset registry with detailed functions • Hazard analysis with consequences assessment • Risk profile developed Security Implementation • Execution of the security solution per the security design System Architecture • Security overview defining the system components, interfaces, data flow and the preliminary zoning of the design YOU ARE HERE 3/8/2022
- 12. 12 AOO: Asset Owner-Operator ICS: Industrial Control System CESER: Office of Cybersecurity, Energy Security, and Emergency Response CT/PT: Current Transformer/Potential Transformer DCS/TCS: Distributed Control System/Transmission Control System DER: Distributed Energy Resource DMZ: Demilitarized Zone EMS: Energy Management System ENG: Engineering HMI: Human Machine Interface I&C: Instrumentation and Control IDS/IPS: Intrusion Detection System/Intrusion Prevention System IED: Intelligent Electronic Device IEEE: Institute of Electrical and Electronics Engineers INL: Idaho National Lab I/O: Input/Output IT: Information Technology NCIT: Non-Conventional Instrument Transformers NDAA: National Defense Authorization Act NIST: National Institute of Standards and Technology NREL: National Renewable Energy Laboratory O&M: Operations and Maintenance OEM: Original Equipment Manufacturer OT: Operational Technology PLC: Programmable Logic Controller PMU: Phasor Measurement Unit RAS: Remote Access Server RC/BA: Reliability Coordinator/Balancing Authority RTU: Remote Terminal Unit SCADA: Supervisory Control and Data Acquisition SEI ETF: Securing Energy Infrastructure Executive Task Force SIEM: Security Information and Event Management VPN: Virtual Private Network 3/8/2022 Appendix A: Acronyms and Abbreviations