![© Siemens AG 2015
2015-11-04Page 15 Maurizio Bigoloni / RC IT EM DG SWS
IOT based Energy Management System
MongoDB aggregation framework
The Aggregation Framework allows easy and efficient aggregation on raw data. Here an example on how we
store raw data in the datapoints collection.
{
feedId: “ABCD”
date: 16/01/2016
values: [
{v: 10,
ts: 1452902400000},
{v: 20,
ts: 1452903000000},
{v: 30,
ts: 1452903600000}
]
}
{
feedId: “ABCD”,
v: 10,
ts: 1452902400000
}
{
feedId: “ABCD”,
v: 20,
ts: 1452903000000
}
{
feedId: “ABCD”,
v: 30,
ts: 1452903600000}
}
VS
Data pre-aggregation,1 document per day, has
several advantages:
Many fewer documents: 1 per day vs 1 per
datapoint (hundreds of them!)
Index space largely reduced, thus occupying less
disk and RAM
Less I/O operations working on just a single
document
All of this leads to overall better performance](https://image.slidesharecdn.com/monetaniotenergymanagementplatformbasedonmongodb-160118110741/85/Monet-an-IoT-Energy-Management-Platform-based-on-MongoDB-15-320.jpg)
![© Siemens AG 2015
2015-11-04Page 16 Maurizio Bigoloni / RC IT EM DG SWS
IOT based Energy Management System
MongoDB aggregation framework
Here the simple instruction to aggregate raw data into
db.datapoints.aggregate([
{ $match: { “feedId”: “ABCD”, “date”: { “$gte”: yesterday }}},
{ $unwind: “$values” },
{ $group:
{_id : {'$subtract': [{'$divide': ["$values.ts", 3600000]}, {'$mod': ["$values.ts", 3600000]}]}},
date: date,
max: {$max: "$values.v"},
min: {$min: "$values.v"},
avg: {$avg: "$values.v"},
sum: {$sum: "$values.v"}
}
]);](https://image.slidesharecdn.com/monetaniotenergymanagementplatformbasedonmongodb-160118110741/85/Monet-an-IoT-Energy-Management-Platform-based-on-MongoDB-16-320.jpg)
![© Siemens AG 2015
2015-11-04Page 17 Maurizio Bigoloni / RC IT EM DG SWS
IOT based Energy Management System
MongoDB aggregation framework
{
feedId: “ABCD”
date: 12/01/2016
values: [{
v: 10,
ts: 1452556800000
}]
}
{
feedId: “ABCD”
date: 16/01/2016
values: [
{v: 10,
ts: 1452902400000},
{v: 20,
ts: 1452903000000},
{v: 30,
ts: 1452903600000}
]
}
{
feedId: “ABCD”
date: 16/01/2016
values: [
{v: 10,
ts: 1452902400000}
]
}
{
feedId: “ABCD”
date: 16/01/2016
values: [
{v: 20,
ts: 1452903000000}
]
}
{
feedId: “ABCD”
date: 16/01/2016
values: [
{v: 30,
ts: 1452903000000}
]
}
{
_id: 403584,
date: 16/01/2016
max: 30,
min: 10,
avg: 20,
sum: 60
}
{
feedId: “ABCD”
date: 16/01/2016
values: [
{v: 10,
ts: 1452902400000},
{v: 20,
ts: 1452903000000},
{v: 30,
ts: 1452903600000}
]
}
match unwind group
Input Result](https://image.slidesharecdn.com/monetaniotenergymanagementplatformbasedonmongodb-160118110741/85/Monet-an-IoT-Energy-Management-Platform-based-on-MongoDB-17-320.jpg)
Monet, an IoT Energy Management Platform based on MongoDB
- 1. © Siemens AG 2015 siemens.com/EUW Monet An IoT Energy Management Platform based on MongoDB MongoDB Event | Milano, 14 January 2016
- 2. © Siemens AG 2015 2015-11-04Page 2 Maurizio Bigoloni / RC IT EM DG SWS Energy Management Division Key numbers ~ €11 bn Revenue ~ 53,000 People ~ 100 Sites ~ €350 m R&D Investments Locations Energy Management
- 3. © Siemens AG 2015 2015-11-04Page 3 Maurizio Bigoloni / RC IT EM DG SWS Energy Management Division Portfolio Services&Security Software/IT Grid control – big data analytics – grid application DigitalizationAutomation Communication, automation, protection, field devices Electrification Electrification Solutions High-voltage direct current (HVDC) transmission – grid access – FACTS – air-insulated/gas-insulated substations – power systems solutions – microgrids / nanogrids Products & Systems High-voltage switchgear and systems – power transformers – medium-voltage switchgears – distribution transformers – low-voltage switchboards and circuit breakers Large power generation TSOs1 Oil and gas Industria Infrastructures / construction DSOs2 and municipalities Distributed generation 1 Transmission system operators 2 Distribution system operators
- 4. © Siemens AG 2015 2015-11-04Page 4 Maurizio Bigoloni / RC IT EM DG SWS Digitalization Enables customers to turn threats into opportunities Digital services Vertical software Digitally enhanced electrification and automation Customers benefit • Increased productivity and flexibility • Shorter time to market • Improved uptime and lifetime Challenges Digitalization delivers answers ALERT! Balancing Peak avoidance Resilience Business models CO2 and cost avoidance Loss prevention Distributed optimization Customer focus
- 5. © Siemens AG 2015 2015-11-04Page 5 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System Keypoints • Micro-Services Architecture • Scalable/Reliable • Html5 web applications • Software as a Service • Standard protocols (MQTT, AMQP) • General-purpose data acquisition • Real-time data aggregation • Real-time data analysis CloudInternet of Things Analytics
- 6. © Siemens AG 2015 2015-11-04Page 6 Maurizio Bigoloni / RC IT EM DG SWS Internet of Things & Energy Management Innovative Services Energy Efficiency Real-time data acquisition and data aggregation are enabler for energy efficiency advanced services. Internet of Things paradigm allows integrating end-user devices to get preferences and behavior information. Energy Rules based on: • Real-time measures • Load & Generation Profile/ Forecast • User Preferences / Environmental data Energy Rules actions: • Load Control/Shifting • Storage Control • Comfort variables set points
- 7. © Siemens AG 2015 2015-11-04Page 7 Maurizio Bigoloni / RC IT EM DG SWS Internet of Things & Energy Management Innovative Services Demand Response Integrating energy stakeholder systems (TSO, DSO, Energy Vendor) with end-user systems and devices allows implementation of innovative services toward Demand Response: • Energy Vendor / Dynamic Price • DSO / Grid Emergency • DSO / Peak Shaving • DSO / Electric Mobility integration • Aggregation of Consumer/Producer
- 8. © Siemens AG 2015 2015-11-04Page 8 Maurizio Bigoloni / RC IT EM DG SWS Milan Expo 2015 A unique opportunity Expo 2015 a unique opportunity to build a Smart City from green field: • 1’100’000 m2 Area • 75MW Planned Power • 145 Countries • 53 Self-Built Pavilions • Fiber optics backbone • Wi-Fi infrastructure (2’700 AP) Siemens strategic partner of Enel for the Smart Grid technology at EXPO Milano 2015
- 9. © Siemens AG 2015 2015-11-04Page 9 Maurizio Bigoloni / RC IT EM DG SWS Milan Expo 2015 Project details SMART METERING GRID TECHNOLOGIES ELECTRIC MOBILITY SMART LIGHTING OPERATION CENTERS PV PLANTSSMART SUBSTATIONS ELECTRO-MOBILITY 50 8500 ARCHILEDE OUTDOOR SOLUTIONS SMART BUILDING 100 200 5 BUILDING MANAGEMENT ROOM AUTOMATION ENERGY STORAGE 1 30 300 2
- 10. © Siemens AG 2015 2015-11-04Page 10 Maurizio Bigoloni / RC IT EM DG SWS Milan Expo 2015 Energy Management System Online since May1st EMS backend REST API EMS web applications MQTT / AMQP Energy Monitoring / Reporting Energy Profiling / Forecast Energy Efficiency / Demand Response Enterprise Applications 132 5 6 1. Grid Substations: P measures every 5’ 2. GME Meter on each MV/LV transformer – via GSM every 15’ 3. Enel Meter on each LV line – via Wi-Fi every 5’ 4. Multi-meter on each controllable load; temperature & lighting sensors – via Wi-Fi every 5’ 5. Charging Units – via GSM every 15’ 6. Public Lighting Panles – every 60’ 4
- 11. © Siemens AG 2015 2015-11-04Page 11 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System IT architecture EMS backend Local Control REST API EMS web applications MQTT / AMQP Energy Monitoring / Reporting Energy Profiling / Forecast Energy Efficiency / Demand Response Enterprise Applications Smart Home Personal Devices Distribution Network SCADA Public Lighting Building Mngt System Electric Mobility Plant SCADA Smart Meter Meter Data Mngt FieldPremiseCloud MongoDB
- 12. © Siemens AG 2015 2015-11-04Page 12 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System Why a NO SQL database? During the design phase the database selection was a key step: SQL, NO-SQL, or both? At the end the choice was to go to NO-SQL only selecting MongoDB: • General-purpose data acquisition layer schema-less databases best option for modeling different kind of objects • Data acquisition layer requires a large number of WRITE operations NO-SQL more promising for keeping constants the WRITE performances • Design for Cloud MongoDB scalability fits well with Cloud • Given the full JavaScript application stack (node.js + html5) a JSON based document database as MongoDB resulted to be the natural choices for the entire system
- 13. © Siemens AG 2015 2015-11-04Page 13 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System Data acquisition Data acquisition layer in Monet is based on MQTT protocol (www.mqqt.org). MQTT is a standard lightweight protocol adopting the publish/subscribe paradigm. MQTT is based on the topic concept; a topic is a stream of data coming from or going to a particular I/O of a particular device. So it can contain data from the field but also commands. The Feed Broker is the Monet module that collects the data. It contains a MQTT broker called Mosca. When a message arrives to the broker, the payload is stored as raw data in MongoDB.
- 14. © Siemens AG 2015 2015-11-04Page 14 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System Real-Time data aggregation and data analysis Raw data coming from field devices needs to be analyzed; for any field variable there is the possibility to calculate Trends: • Curve at fixed precision from raw data (avg, min, max, sum) • Daily, Weekly, Monthly, Yearly trends calculated at different precisions Trends curves related to energy variables (energy, power) are aggregated by different hierarchies: • Geographical • Electrical • Technical • By usage, scenario, mode
- 15. © Siemens AG 2015 2015-11-04Page 15 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System MongoDB aggregation framework The Aggregation Framework allows easy and efficient aggregation on raw data. Here an example on how we store raw data in the datapoints collection. { feedId: “ABCD” date: 16/01/2016 values: [ {v: 10, ts: 1452902400000}, {v: 20, ts: 1452903000000}, {v: 30, ts: 1452903600000} ] } { feedId: “ABCD”, v: 10, ts: 1452902400000 } { feedId: “ABCD”, v: 20, ts: 1452903000000 } { feedId: “ABCD”, v: 30, ts: 1452903600000} } VS Data pre-aggregation,1 document per day, has several advantages: Many fewer documents: 1 per day vs 1 per datapoint (hundreds of them!) Index space largely reduced, thus occupying less disk and RAM Less I/O operations working on just a single document All of this leads to overall better performance
- 16. © Siemens AG 2015 2015-11-04Page 16 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System MongoDB aggregation framework Here the simple instruction to aggregate raw data into db.datapoints.aggregate([ { $match: { “feedId”: “ABCD”, “date”: { “$gte”: yesterday }}}, { $unwind: “$values” }, { $group: {_id : {'$subtract': [{'$divide': ["$values.ts", 3600000]}, {'$mod': ["$values.ts", 3600000]}]}}, date: date, max: {$max: "$values.v"}, min: {$min: "$values.v"}, avg: {$avg: "$values.v"}, sum: {$sum: "$values.v"} } ]);
- 17. © Siemens AG 2015 2015-11-04Page 17 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System MongoDB aggregation framework { feedId: “ABCD” date: 12/01/2016 values: [{ v: 10, ts: 1452556800000 }] } { feedId: “ABCD” date: 16/01/2016 values: [ {v: 10, ts: 1452902400000}, {v: 20, ts: 1452903000000}, {v: 30, ts: 1452903600000} ] } { feedId: “ABCD” date: 16/01/2016 values: [ {v: 10, ts: 1452902400000} ] } { feedId: “ABCD” date: 16/01/2016 values: [ {v: 20, ts: 1452903000000} ] } { feedId: “ABCD” date: 16/01/2016 values: [ {v: 30, ts: 1452903000000} ] } { _id: 403584, date: 16/01/2016 max: 30, min: 10, avg: 20, sum: 60 } { feedId: “ABCD” date: 16/01/2016 values: [ {v: 10, ts: 1452902400000}, {v: 20, ts: 1452903000000}, {v: 30, ts: 1452903600000} ] } match unwind group Input Result
- 18. © Siemens AG 2015 2015-11-04Page 18 Maurizio Bigoloni / RC IT EM DG SWS IOT based Energy Management System Some numbers 6 monthssystem running at EXPO Milano 2015 120.000.000 raw datapoints collected 45 GBraw datapoints collection 17 GB trend curves collection 5 GBenergy aggregation curves collection
- 19. © Siemens AG 2015 2015-11-04Page 19 Maurizio Bigoloni / RC IT EM DG SWS Internet of Things & Energy Management Conclusion • Connected Things • Home Devices • Personal Devices • Electric Grid • Communication Network • Distributed Generation • Innovative Services • Real-Time Data Analysis • Efficiency • Demand Response • Aggregation Internet of ThingsSmart Grid Digital Grid + =
- 20. © Siemens AG 2015 2015-11-04Page 20 Maurizio Bigoloni / RC IT EM DG SWS Maurizio Bigoloni Head of Operation RC IT EM DG SWS Via Vipiteno, 4 20128 Milano Phone: +39 02 243 23335 Mobile: +39 334 8888744 E-mail: maurizio.bigoloni@siemens.com Energy of Things Contact page siemens.com/EUW
Editor's Notes
- #2: Good Morning. I am MB responsible of the Operations team in Digital Grid Software Solutions Italy. In my team we develop different kind of software solutions starting from SCADA systems up to energy management applications.
- #5: Today I will talk about Digitalization and how it may enable our customers to convert threats into opportunities. Challenges like grid balancing, peak avoidance, grid resilience can be addressed by digitally enhanced electrification and automation systems and devices. I will focus on real-time data acquisition and data analysis for energy management – more precisely on how digitalization is an enabler for creating innovative energy services toward efficiency and demand-response.
- #6: Let’s now talk about how to design an IT system that can provide such innovative services. In my opinion there are three key pillars: Adopt an IOT paradigm, that means standard protocols and general-purpose data acquisition layer allowing integration of a wide range of systems and devices. Among the different raising standards I would like to highlight two of them: MQTT and AMQP. Just to make some concrete examples – the two main cloud PaaS providers Amazon and Microsoft they both have IOT layers based respectively on MQTT and AMQP. Go on Cloud. I believe that going in the Cloud for such applications is a must. As consequence Software as a Service business model is most likely to be the natural choice. Here a micro-services architecture with independent small services guarantees the modularity, reliability, and scalability of the system. Real Time Analytics. Data needs to be analyzed if we want value them. What we really need is real-time aggregation and data analysis if we need to manage energy flows. We are not talking about monitoring and reporting only but we are talking about services that requires real-time decisions for implementing efficiency or demand-response functionality.
- #7: Real-time data acquisition and data aggregation are enabler for energy efficiency advanced services; having: Real-time measures Load & Generation Profile/ Forecast User Preferences / Environmental data The EMS system can define rules to optimize energy consumption (minimize KWh) or to minimize energy costs (minimize €). Rule actions can be: Load Control/Shifting Storage Control Generation dispatching Comfort variables set points integrating Smart Buildings
- #8: While optimization logics in terms of consumption or cost are the objective of a single stakeholder the system can also put together the needs of different stakeholders by supporting demand-profile workflows. This will allow the DSO or the Energy Vendor to agree a specific demand profile with the user (or an aggregation of users) implementing active demand scenarios. Scenarios covering different objectives, for instance: Grid emergency Peak shaving Electric mobility grid integration
- #9: Most of the concepts I just introduced have been tested on a real scenario such as Milano Expo 2015. That was an unique opportunity to build a Smart City from green field: 1 million square meters area 75MW of planned power 145 countries present and 53 self-built pavilions In addition there is full fiber optics backbone connecting everything and a wifi for the public but also for the services. In this project Siemens is a strategic partner of Enel for the Smart Grid technology at EXPO Milano 2015.
- #10: In details @ Expo we had: -100 Smart distribution substations -200 Smart meters in the delivery substations and in the pavilions LV lines -photovoltaic plants and energy storage -30 Smart Buildings with more than 300 room automation devices for climate and lighting management and load control -A public lighting system with more than 8 thousands lamps -50 electric vehicles charging units
- #11: This slide represents the data flows implemented in the Expo project. First of all we have data coming from the network SCADA. Power measures from each substation every 5 minutes. Then we get data from each delivery – we have a GME meter on each MV/LV transformer – 15’ data via GSM. Then we get data from the main LV lines after the main power panel with the Smart Info devices – 5’ data via WiFi. Finally on specific loads we have multi-meter. We integrate also temperature & lighting sensors for which we get data every 5’.
- #12: Here it is a reference architecture for an IOT based Energy Management System. From the bottom we have the so –called technical systems such as: Distribution network SCADA Smart Meters or Meter Data Management Systems Other systems that are typically in the Grid: public lighting, building management, electric mobility All these systems still require a Local Control that means that they have automation functionalities that have to be executed locally. But all these systems will send real-time data to the EMS system – data that are currently available by these kind of systems but that are still kept in the control rooms. Additionally we have devices, typically on the end-user side that can be also integrated in the EMS system. On the cloud dedicated services take care of the data acquisition and data analysis exposing their functionality via a standard REST API interface. The API can then be accessed by specific EMS web applications or by any other enterprise application.
- #20: Here the conclusion. With Smart Grid we added an IT/communication network to the Electric Grid, now that we are moving in the IOT era we are going to integrate connected Things to the Smart Grid systems. In other words we are moving from Smart Grid to Digital Grid where innovative services will be reality soon.