S.2.e Specifications for Data Ingestion via Sunshine FTP
1 like717 views
This document provides specifications for ingesting pilot consumption and sensor data via FTP into Sunshine's data repository. It describes the required meter mapping file format and consumption/sensor data file formats, including naming conventions and metadata to include. Meter mappings relate devices to buildings and define reading types and frequencies. Data files group timestamped readings and costs into CSVs with specified naming and formatting.
![CIP-ICT-PSP-2012-6 – 325161
Page 5 of 6
"This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the
Competitiveness and Innovation Framework Programme by the European Community"
(http://ec.europa.eu/ict_psp).
Observable Property. The type of property observed.
o Combo box with selection limited to: Electrical energy, Gaseous fuel, Liquid fuel, Solid fuel, Steam fuel,
Thermal energy, Temperature.
Consumption Reading Type. Refers only to observed properties of energy consumption (so it is not
applicable to Temperature observations) and it specifies whether if readings are absolute (e.g. the
cumulative amount of cubic meters of gas measured by a gas meter from the time it was installed) or
relative (e.g. the amount of cubic meters consumed between the two last measurements).
Unit. The unit the observed property is measured into.
o Combo box with selection limited to: kWh, m3
, L, Kg, ton, °C
Measuring Frequency. Describes if consumption readings and sensor measurement are taken regularly
or not and at which frequency.
o Combo box with selection limited to: 15 = every 15 minutes, 1h = hourly, 1d = daily, 1w = weekly, 1m =
monthly, 1y = yearly, ir = irregular intervals.
o If the data collection frequency is not among the options, mark it as irregular.
Cost availability [€/unit]. States if data about energy cost are available in parallel with consumption
measures. This is obviously not applicable to temperature sensor measures. Energy cost data is always
relative to the period of time between the current observation and the previous.
o Combo box with selection limited to: yes, no, n.a.
Description and ID. A description of the sensor nature, location and ID.
o Example: Gas meter for the central heating system [ID: IT221E00001234]
Data File Name. This field is automatically filled as soon as the other field in the same row are. It
provides the string to be used as file name for the consumption/sensor data files described below.
o Example: FER-001_GASR_MCU_1h_nnnnnnn.csv
1.2 Consumption and sensor measurement data files
Data files contain the actual consumption readings (or sensor measurement) for a specific meter for a
specific period of time. Readings have to be grouped together, stored in CSV files and put on the project’s
FTP space where a specific folder has been set up for each pilot:
ftp://sunshineftp.sinergis.it/PilotName/Dynamic_data.
Readings have to be provided with the highest available frequency and can be collected at irregular
intervals.](https://image.slidesharecdn.com/s2e-specificationsfordataingestionvia-sunshineftpd24annex-160211075601/85/S-2-e-Specifications-for-Data-Ingestion-via-Sunshine-FTP-5-320.jpg)
![CIP-ICT-PSP-2012-6 – 325161
Page 6 of 6
"This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the
Competitiveness and Innovation Framework Programme by the European Community"
(http://ec.europa.eu/ict_psp).
During baseline data collection the grouping of single readings in a single file can be on a monthly basis,
while during Sunshine’s platform piloting period the grouping have to be done on a daily basis.
The format of the consumption file name is: MMM-MMM_OOOO_UUU_FF_nnnnnnnn.csv
o Where MMM-MMM stands for the meter/sensor ID, OOOO for the observable property, UUU for the
unit, FF for the data collection frequency and nnnnnnnn is a progressive numerical id.
o The basic string for the filename is automatically generated for each meter/sensor in the mapping file,
if the values in the table are properly filled. Example: FER-001_GASR_MCU_1h_nnnnnnn.csv
The format of the data contained in the CSV file is: timestamp;consumption;cost
o Timestamp has to be in UTC and has the format yyyy-mm-dd hh:mm:ss
o Consumption and cost are float values with a point as decimal separator
o Dataline example: 2013-11-01 01:15:00;0.9570;0.23
Cost data can be omitted if unavailable (as specified in the mapping file).
o Dataline example without cost: 2013-11-01 01:15:00;0.9570
No string header must be put in the CSV file
Each relative consumption reading (as well as cost) is an integrated value describing the energy
consumed during an interval of time. Relative consumption reading is assumed to refer to the interval
between the timestamp of the previous reading and the timestamp of the current reading.
Here follows an example of relative consumption data with cost:
[...]
2013-11-01 01:00:00;0.9570;0.23
2013-11-01 02:00:00;0.8340;0.22
2013-11-01 03:00:00;0.9933;0.24
2013-11-01 04:00:00;0.7750;0.19
[...]
And here is an example, for an indoor temperature sensor:
[...]
2013-11-01 15:00:00;22.5
2013-11-01 16:00:00;21.340
2013-11-01 17:00:00;20.9933
2013-11-01 18:00:00;20.7750
[...]](https://image.slidesharecdn.com/s2e-specificationsfordataingestionvia-sunshineftpd24annex-160211075601/85/S-2-e-Specifications-for-Data-Ingestion-via-Sunshine-FTP-6-320.jpg)
S.2.e Specifications for Data Ingestion via Sunshine FTP
- 1. CIP-ICT-PSP-2012-6 – 325161 Page 1 of 6 "This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme by the European Community" (http://ec.europa.eu/ict_psp). Annex to deliverable D2.4 Specifications for the ingestion of pilot's consumption and indoor sensor data via Sunshine's FTP Revision: v1.0
- 2. CIP-ICT-PSP-2012-6 – 325161 Page 2 of 6 "This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme by the European Community" (http://ec.europa.eu/ict_psp). REVISION HISTORY AND STATEMENT OF ORIGINALITY Revision Date Author Description v0.1 5th May 2014 Luca Giovannini Document created. V1.0 21st May 2014 Luca Giovannini Document reviewed and updated. Statement of originality: This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both. Moreover, this deliverable reflects only the author’s views. The European Community is not liable for any use that might be made of the information contained herein.
- 3. CIP-ICT-PSP-2012-6 – 325161 Page 3 of 6 "This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme by the European Community" (http://ec.europa.eu/ict_psp). Table of contents 1 General principles ...............................................................................................................4 1.1 Meter mapping file..................................................................................................................4 1.2 Consumption and sensor measurement data files ....................................................................5
- 4. CIP-ICT-PSP-2012-6 – 325161 Page 4 of 6 "This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme by the European Community" (http://ec.europa.eu/ict_psp). 1 General principles These guidelines have the purpose of defining how pilot should deliver consumption and indoor sensor data to Sunshine’s FTP server to be then stored into the central data repository. The guidelines are applicable to energy consumption readings and indoor sensors measurements from pilot buildings of Scenario 2 and to electrical energy consumption readings from light lines of Scenario 3. Data delivery via FTP will be available to pilots both in the baseline data gathering phase (up to Month 21) and in the subsequent piloting phase of Sunshine’s application. Data delivery via FTP is just one of the two possible data delivery means offered by the Sunshine’s infrastructure. Pilot that are able and willing to put in place a web service to expose this kind of data can instead refer to the guideline that defines how to interface the pilot’s web services with Sunshine’s platform via ESPI – Green Button data exchange protocol. The following sections describe in details the type of data to deliver and its format. First of all, pilots have to compile a mapping file that has the purpose of describing the basic properties of the meter/sensor and allow to identify which pilot building or light line does it belong to (section 1.1). Then, consumption readings and sensor measurements have to be grouped together, stored in CSV files and put on the project’s FTP space (section 1.2). 1.1 Meter mapping file The mapping file has the purpose of describing the basic properties of the meter/sensor and identifying which pilot building or light line does it belong to. A copy of this file has been put on the main pilot folder of the project’s FTP server, already filled with some example lines: ftp://sunshine.dedacenter.it/PilotName/meter_mapping.xlsx The part of the mapping file that has to be filled by pilots is in the tab “mapping” and has the following fields: Meter/Sensor ID. A unique ID that identifies the meter (or the sensor). o The format is PPP-nnn, where PPP is a label referring to the pilot (see table in tab “codelists”) and nnn is a progressive numerical id. o Example: FER-001. Feature of Reference. The string identifier of the pilot building the meter/sensor is attached to, as it appears in the file “Pilot building properties survey.xlsx" compiled by the pilot. For meters attached to light lines used in Scenario 3, this field has to be filled with a reference name for the light line. o Example: Scuola Materna Pacinotti
- 5. CIP-ICT-PSP-2012-6 – 325161 Page 5 of 6 "This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme by the European Community" (http://ec.europa.eu/ict_psp). Observable Property. The type of property observed. o Combo box with selection limited to: Electrical energy, Gaseous fuel, Liquid fuel, Solid fuel, Steam fuel, Thermal energy, Temperature. Consumption Reading Type. Refers only to observed properties of energy consumption (so it is not applicable to Temperature observations) and it specifies whether if readings are absolute (e.g. the cumulative amount of cubic meters of gas measured by a gas meter from the time it was installed) or relative (e.g. the amount of cubic meters consumed between the two last measurements). Unit. The unit the observed property is measured into. o Combo box with selection limited to: kWh, m3 , L, Kg, ton, °C Measuring Frequency. Describes if consumption readings and sensor measurement are taken regularly or not and at which frequency. o Combo box with selection limited to: 15 = every 15 minutes, 1h = hourly, 1d = daily, 1w = weekly, 1m = monthly, 1y = yearly, ir = irregular intervals. o If the data collection frequency is not among the options, mark it as irregular. Cost availability [€/unit]. States if data about energy cost are available in parallel with consumption measures. This is obviously not applicable to temperature sensor measures. Energy cost data is always relative to the period of time between the current observation and the previous. o Combo box with selection limited to: yes, no, n.a. Description and ID. A description of the sensor nature, location and ID. o Example: Gas meter for the central heating system [ID: IT221E00001234] Data File Name. This field is automatically filled as soon as the other field in the same row are. It provides the string to be used as file name for the consumption/sensor data files described below. o Example: FER-001_GASR_MCU_1h_nnnnnnn.csv 1.2 Consumption and sensor measurement data files Data files contain the actual consumption readings (or sensor measurement) for a specific meter for a specific period of time. Readings have to be grouped together, stored in CSV files and put on the project’s FTP space where a specific folder has been set up for each pilot: ftp://sunshineftp.sinergis.it/PilotName/Dynamic_data. Readings have to be provided with the highest available frequency and can be collected at irregular intervals.
- 6. CIP-ICT-PSP-2012-6 – 325161 Page 6 of 6 "This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme by the European Community" (http://ec.europa.eu/ict_psp). During baseline data collection the grouping of single readings in a single file can be on a monthly basis, while during Sunshine’s platform piloting period the grouping have to be done on a daily basis. The format of the consumption file name is: MMM-MMM_OOOO_UUU_FF_nnnnnnnn.csv o Where MMM-MMM stands for the meter/sensor ID, OOOO for the observable property, UUU for the unit, FF for the data collection frequency and nnnnnnnn is a progressive numerical id. o The basic string for the filename is automatically generated for each meter/sensor in the mapping file, if the values in the table are properly filled. Example: FER-001_GASR_MCU_1h_nnnnnnn.csv The format of the data contained in the CSV file is: timestamp;consumption;cost o Timestamp has to be in UTC and has the format yyyy-mm-dd hh:mm:ss o Consumption and cost are float values with a point as decimal separator o Dataline example: 2013-11-01 01:15:00;0.9570;0.23 Cost data can be omitted if unavailable (as specified in the mapping file). o Dataline example without cost: 2013-11-01 01:15:00;0.9570 No string header must be put in the CSV file Each relative consumption reading (as well as cost) is an integrated value describing the energy consumed during an interval of time. Relative consumption reading is assumed to refer to the interval between the timestamp of the previous reading and the timestamp of the current reading. Here follows an example of relative consumption data with cost: [...] 2013-11-01 01:00:00;0.9570;0.23 2013-11-01 02:00:00;0.8340;0.22 2013-11-01 03:00:00;0.9933;0.24 2013-11-01 04:00:00;0.7750;0.19 [...] And here is an example, for an indoor temperature sensor: [...] 2013-11-01 15:00:00;22.5 2013-11-01 16:00:00;21.340 2013-11-01 17:00:00;20.9933 2013-11-01 18:00:00;20.7750 [...]