![Rubric
www.ecb.europa.eu ©
SDW portal – supported by the SDMX model
• Navigation hierarchy - Category schema
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 23
Data flows are categorized always
to the leaf nodes in the navigation tree.
E.g: 12 series of the FM Data Flow
to the ‘Official interest rates’ category.
Constraints of FM dataset:
FREQ in [B, D]
PROVIDER_FM = 4F
INSTRUMENT_FM = KR
Link to portal](https://image.slidesharecdn.com/sdmxexperts2016capacitybuilding6aeng-g-161107160326/85/2016-SDMX-Experts-meeting-Using-SDMX-data-model-for-data-dissemination-and-data-retrieval-Gabor-Horvath-21-320.jpg)
2016 SDMX Experts meeting, Using SDMX data model for data dissemination and data retrieval, Gábor Horváth
- 1. Using SDMX data model for data dissemination and data retrieval. Introducing ECB’s Statistical Data Warehouse Gábor Horváth SDMX Expert Meeting, October 2016
- 2. Rubric www.ecb.europa.eu © Background • SDW key dissemination channel for ECB statistics since 2005! • Primary source of data & metadata for end users, websites (ECB website, Our statistics), visualization applications (ECBstatsApp); • Fully designed based on the SDMX information model • Access the public SDW via: http://sdw.ecb.europa.eu SDW = Data & metadata + SDW Portal repository 2 SDMX and ECB’s Statistical Data Warehouse SDW Databas e Data Metadata SDW Databas e Data Metadata SDMX and ECB’s Statistical Data Warehouse
- 3. Rubric www.ecb.europa.eu © 1. Data Structure Definitions (DSDs) 2. Concepts and Code lists 3. Data flows 4. Category schema 5. Reference Metadata 6. Notification SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 5
- 4. Rubric www.ecb.europa.eu © • Data Structure Definitions • Data flows: 280+, (18+ million time series) ECB SDW’s Content – Current status SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 6 Maintained by ECB Other Maintenance Agencies 110 50+
- 5. Rubric www.ecb.europa.eu © SDW Database – Structural Metadata Storage SMD = DSD, Statistical concepts, code lists are stored with their timestamps of validity. From 2005 – until the introduction of separate, dedicated SDMX registries, the SDW database fulfilled the role of SDMX registry. The SDW database stores for each artifact also its Maintenance agency…. SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 7 SDMX and ECB’s Statistical Data Warehouse
- 6. Rubric www.ecb.europa.eu © Accessing the Data Structure Definitions 1. 1. Studying the Complete DSD of a Data Flow (all dimensions, attributes, their codelists with complete code list and their usage on the portal is displayed). 2. Dimensions are used in the ‘MS Excel Pivot Export’s 3. ECB SDMX WS 2.1 : https://sdw-wsrest.ecb.europa.eu/service/datastructure/ECB/ECB_FMD2 SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 8 Link to portal 1. 2.
- 7. Rubric www.ecb.europa.eu © The SDW portal displays all attributes defined in the DSD. They can be accessed on different pages of the portal. Note: The list of pages is available on the ‘DSD’ of the Data Flow SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 9 Link to portal Level of attribute Page, where it is displayed Observation Quickview + Data Table Series Quickview + Metadata Sibling Quickview + Metadata Dataset Metadata
- 8. Rubric www.ecb.europa.eu © The SDW portal uses attributes: for the Data display (grid) and Data visualization (graphs). •COLLECTION (beginning of period, end of period, etc.) •DECIMAL attributes (default setting, can be changed under ‘My settings…’ SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 10
- 9. Rubric www.ecb.europa.eu © 1. Data Structure Definitions (DSDs) 2. Concepts and Code lists 3. Data flows 4. Category schema 5. Reference Metadata 6. Notification SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 11
- 10. Rubric www.ecb.europa.eu © Concepts and Codelists in SDW 1. Displayed in the DSDs of a Data Flow (all dimensions and attributes with their concepts with complete code list). 2. See Codelists of Dimension Filters in the ‘MS Excel Pivot Export’s 3. ECB SDMX WS 2.1 : https://sdw-wsrest.ecb.europa.eu/service/datastructure/ECB/ECB_FMD2 SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 12 Link to portal 1. 2.
- 11. Rubric www.ecb.europa.eu © Concepts and Codelists in SDW 2. 3. Dimension Filters can be used to “drill down” in the data cubes. 3.1 Base Filters (common in “all” data sets) 3.2. Data Set specific Dimension Filters. (Constraints preset the codes in the filters) Note: ‘Safe Drill Down’ After each step during the ‘drill down’ in the data cube, the content of the filters is always set to the set of available codes. Users will always find series during the drill down. 4. ECB SDMX WS 2.1 : https://sdw-wsrest.ecb.europa.eu/service/codelist/ECB/CL_CURRENCY SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 13 Link to portal
- 12. Rubric www.ecb.europa.eu © 1. Data Structure Definitions (DSDs) 2. Concepts and Code lists 3. Data flows 4. Category schema 5. Reference Metadata 6. Notification SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 14
- 13. Rubric www.ecb.europa.eu © SDW Database – Data Storage • Observations are stored in the traditional data warehousing design solutions: in start-schemas. • The structure of a Star-schema is defined in a DSD. • Data Set: Three Fact tables with the corresponding codelists. SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 15 DSDs can contain four levels of attributes: 1.Observation level 2.Time Series level 3.Sibling level (Sibling = groups where Frequency dim is wildcarded) 4.Data Set level Fact tables / data set Content 1 OBS table Dimensions, period, observations, Observation level attributes 1 SERIES table Dimensions, Series-level attributes 1 SIBLING table Dimensions, Sibling-level attributes Notes: • No observation is ever deleted! • Full life span of time series is available.
- 14. Rubric www.ecb.europa.eu © SDW Database – star schema of observationsobservations SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 16 Period Dimension 1 Dimension 2 Dimension n Observation value Observation attribute 1 Observation attribute m Valid from Valid to Code Description … … Code Description … … Code Description … … Code Description … … Code Description … … Fact tables with dimension tables
- 15. Rubric www.ecb.europa.eu © SDW Database – star schema of seriesseries attributes SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 17 Period Dimension 1 Dimension 2 Dimension n Series attribute 1 Series attribute l Valid from Valid to Code Description … … Code Description … … Code Description … … Code Description … … Code Description … … Fact tables with dimension tables
- 16. Rubric www.ecb.europa.eu © SDW Database – star schema of siblingsibling attributes SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 18 Period Dimension 1 Dimension 2 Dimension n-1 Sibling attribute 1 Sibling attribute k Valid from Valid to Code Description … … Code Description … … Code Description … … Code Description … … Code Description … … Fact tables with dimension tables
- 17. Rubric www.ecb.europa.eu © SDW Database – Management of data set tables 1. Creation of new data sets all data tables and views are automatically created before the first dissemination of the data. 2. Updates/changes in the DSD changes which does not affect the ‘backward compatibility’ of database tables are handled automatically. E.g: Introduction of new attributes in fact tables, introduction of new code values in dimension tables, increase of the length of codes in dimension tables. SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 19
- 18. Rubric www.ecb.europa.eu © SDW Database – Data & Metadata loads On average: •daily 200 data loads (on working days) •weekly 4 updates of SMD (including any change in codes, descriptions, etc) SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 20 File type File format Source/agency Data SDMX-ML Compact format (UTF-8) ECB + other external sources (converted) Structural Metadata SDMX-ML Structure Message (UTF-8) ECB, BIS + from other agencies.
- 19. Rubric www.ecb.europa.eu © SDW portal – supported by the SDMX model Data Download options The SDW portal offers SDMX-ML Data Download feature since 2006. The format has been slightly changed, and it is SDMX-ML 2.0 at the moment. In case of interest in data files in the SDMX-ML 2.1 format, please, use the SDMX RESTful Web-Services of the SDW. It can return statistical information also in JSON and different CSV formats! Link ‘API’ leads you to the Documentation of the Web-Services. SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 21 Link to portal
- 20. Rubric www.ecb.europa.eu © 1. Data Structure Definitions (DSDs) 2. Concepts and Code lists 3. Data flows 4. Category schema 5. Reference Metadata 6. Notification SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 22
- 21. Rubric www.ecb.europa.eu © SDW portal – supported by the SDMX model • Navigation hierarchy - Category schema SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 23 Data flows are categorized always to the leaf nodes in the navigation tree. E.g: 12 series of the FM Data Flow to the ‘Official interest rates’ category. Constraints of FM dataset: FREQ in [B, D] PROVIDER_FM = 4F INSTRUMENT_FM = KR Link to portal
- 22. Rubric www.ecb.europa.eu © SDW portal – supported by the SDMX model • Data Flows in the navigation hierarchy SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 24 All Data Sets without Constraints are Categorized under a dedicated Category ‘Full Content’
- 23. Rubric www.ecb.europa.eu © 1. Data Structure Definitions (DSDs) 2. Concepts and Code lists 3. Data flows 4. Category schema 5. Reference Metadata 6. Notification SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 25
- 24. Rubric www.ecb.europa.eu © SDW portal – supported by the SDMX model Reference Metadata is available on the ‘Metadata’ pages of SDW A list of Concepts from the SDMX Metadata Common Vocabulary (MCV) can be used e.g: Reference MD of the EXR Data Set SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 26 Link to portal
- 25. Rubric www.ecb.europa.eu ©27 SDW portal – Reference Metadata storage in SDW <<Contact>> „Alberto González Pandiella” <<Responsible business area>> “External Statistics Division” <<FAME shortcut >> „$ECB” Reference Metadata can be attached to data at different levels: - Data Sets -Groups/Siblings -Time Series SDMX and ECB’s Statistical Data Warehouse <<attach>> <<attach>> <<attach>>
- 26. Rubric www.ecb.europa.eu © SDW Database – Reference metadata storage SDMX and ECB’s Statistical Data Warehouse 28 SDMX and ECB’s Statistical Data Warehouse Metadata item Metadata id Metadata value Metadata type … …. Attach Attachment id Metadata id Description Dataflow … Constraint Attachment id Concept Code value … …
- 27. Rubric www.ecb.europa.eu © 1. Data Structure Definitions (DSDs) 2. Concepts and Code lists 3. Data flows 4. Category schema 5. Reference Metadata 6. Notification SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 29
- 28. Rubric www.ecb.europa.eu © SDW portal – supported by the SDMX model SDW offers a Notifications service •Users can subscribe to email notifications in cases of changes in the observations of one or multiple time series from one or more Data Flows. •Changes include: (i) new observation added, (ii) existing observation revised (iii) observation deleted. SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 30 The SDW database and portal do not server as Registries and they do not provide Notification services for changes in the Structural Metadata.
- 29. Rubric www.ecb.europa.eu © Thank you for your attention! GÁBOR HORVÁTH European Central Bank Statistical Information Services Division SDMX and ECB’s Statistical Data Warehouse SDMX and ECB’s Statistical Data Warehouse 31