BDE SC6-ws-05/12/2016 technology part - SWC
Download as PPTX, PDF0 likes606 views
The document discusses a pilot project within the Big Data Europe program to create an online dashboard of economic data from municipal budgets. The project aims to aggregate budget and spending data from multiple sources and formats, normalize it using RDF, and analyze and visualize the data to provide insights for citizens, researchers, and decision makers. Technical components used include Apache Flume, Kafka, Spark, HDFS, Virtuoso triplestore, and D3 for visualization. An initial version has been implemented and will be evaluated with municipalities and other stakeholders.
BDE SC6-ws-05/12/2016 technology part - SWC
- 1. BIG DATA EUROPE PILOT SC6: CITIZEN BUDGET ON MUNICIPAL LEVEL EUROPE IN A CHANGING WORLD - INCLUSIVE, INNOVATIVE AND REFLECTIVE SOCIETIES WORKSHOP: THE CHALLENGES OF BIG DATA FOR SOCIETIES IN A CHANGING WORLD, 05 DECEMBER 2016 MARTIN KALTENBÖCK (CFO, SEMANTIC WEB COMPANY) Integrating Big Data, Software & Communities for Addressing Europe’s Societal ChallengesBDE SC6 Workshop
- 2. Big Data Europe (CSA: 2015-17) Show societal value of Big Data: 7 Domains Lower barrier for using big data technologies o Required effort and resources o Limited data science skills Help establishing cross- lingual/organizational/domain Data Value Chains 16-déc.-16
- 3. Big Data Europe 16-déc.-16 COORDINATION Stakeholder Engagement (Requirements Elicitation) SUPPORT Design, Realise, Evaluate Big Data Aggregator Platform Create and Manage Societal Big Data Interest Groups Cloud-deployment ready Big Data Aggregator Platform CSA Measures Results
- 4. THE BDE PLATFORM ARCHITECTURE & COMPONENTS Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges
- 5. The three Big Data „V“ Variety is often neglected
- 6. Current State of Platform Architecture
- 7. Adding a Semantic Layer to Data Lakes Manufacturing Marketing Sales SupportAccounting Semantic Data Lake • central place for model, schema and data historization • Combination of Scale Out (cost reduction) and semantics (increased control & flexibility) • grows incrementally (pay-as-you-go) Inbound Data Sources Outbound and Consumption Inbound Raw Data Store Data Lake (order of magnitude cheaper scalable data store) Knowledge Graph for Relationship Definition and Meta Data Frontend to Access Relationship and KPI Definition / Documentation Frontend to Access (ad hoc) Reports Outbound Data Delivery to Target Systems JSON-LD CSVW R2RMLXML2RDF
- 8. Why to use BDE Technology? Hortonworks Cloudera MapR Bigtop BDE File System HDFS HDFS NFS HDFS HDFS Installation Native Native Native Native lightweight virtualization Plug & play components (no rigid schema) no no no no yes High Availability Single failure recovery (yarn) Single failure recovery (yarn) Self healing, mult. failure rec. Single failure recovery (yarn) Multiple Failure recovery Cost Commercial Commercial Commercial Free Free Scaling Freemium Freemium Freemium Free Free Addition of custom components Not easy No No No Yes Integration testing yes yes yes yes -- Operating systems Linux Linux Linux Linux All Management tool Ambari Cloudera manager MapR Control system - Docker swarm UI+ Custom
- 9. SC6 PILOT CITIZENS BUDGET ON MUNICIPAL LEVEL ARCHITECTURE & COMPONENTS Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges
- 10. SC6 in Big Data Europe – what is included Europe in a changing world - inclusive, innovative and reflective societies Social Sciences Smart Statistics (Digital) Humanities Digital (Research) Archives 16-déc.-16www.big-data-europe.eu
- 11. SC6: Social Sciences 16-déc.-16www.big-data-europe.eu Pilot focus area: Citizens budget spending on municipal level Big Data Focus area: Statistical and research data linking & integration Selected Key Data assets: Detailed budget execution data in city level, statistical data from public data portals and statistical offices, federated social sciences data
- 12. SC6 Pilot: Idea & Objectives State of the Art: o Budget: the most important document of public policy o Budget execution affects everyday lives o Citizens are more involved in city level activities Objective: Can we make budgets more useful for citizens, researchers and decision makers? 16-déc.-16
- 13. SC6 Pilot: Idea & Objectives Create an online Dashboard on Economic Data o Harvest data from several sources in diff. formats o Normalise the data (RDF) o Link & map the data (attributes, structure, languages) o Analyse the data – financial ratios (comparisons, predictions etc.) o Visualise the analysis on an online dashboard including help & infos to understand data & analysis o Procide raw data (for further use as open data)16-déc.-16www.big-data-europe.eu
- 14. 2 H2020 projects working together on the SC6 Pilot • Big Data Europe • Your Data Stories SC6 Pilot core team: Ivana Versic (Cessda), Michalis Vafopoulos (NCSR-D), Martin Kaltenböck (SWC), Jürgen Jakobitsch (SWC), Hossein Abroshan (Cessda) SC6 Pilot Partners
- 15. Data used / produced in Pilot Budget Data and Budget Execution Data Municipality of Athens, Greece o Description: budget execution data in detail o Frequency: daily o Ownership: open o Format: API Municipality of Thessaloniki, Greece o Description: budget execution data in detail o Frequency: daily o Ownership: open o Format: csv, xls (files for download provided) 16-déc.-16www.big-data-europe.eu Municipality of Kalamaria, Greece o Description: budget execution data in detail o Frequency: weekly o Ownership: open o Format: csv, xls (files for download provided) Additional Open Data o Description: economic taxonomies etc. o Ownership: open o Format: RDF (skos, owl), other o E.g. COFOG (UN Classification) Size of Data o ~ 30 Mio triples (statements) for 1 year
- 16. 4 Vs of Big Data in SC6 Pilot Variety: requirement based on the harvesting of budget data and budget execution data from several sources, available in different structures and formats. Volume: requirement regarding the growing amount of open budget data available as well as of budget execution data Velocity: requirements regarding budget execution data that is provided on continuous basis by the publisher (daily, weekly, monthly). Veracity: Veracity refers to the biases, noise and abnormality in data. Even for within the same country there are differences on the published data because often are coming from different systems or public accounting standards are not enforced absolutely uniformly (e.g. different municipal departments) 16-déc.-16www.big-data-europe.eu
- 17. SC6: Social Sciences www.big-data-europe.eu Pilot Architecture & Components
- 18. SC6 Pilot - Architecture 16-déc.-16www.big-data-europe.eu
- 19. SC6 Pilot: Technical Components Apache Flume, https://flume.apache.org/ (data ingestion) Apache Kafka, http://kafka.apache.org (messaging service) Apache Spark, http://spark.apache.org (distributed analysis, transformation) Apache HDFS, http://hadoop.apache.org (raw data storage) SWCs’ PoolParty Semantic Suite, http://poolparty.biz (data consolidation, curation, mapping) OpenLink s’ Virtuoso, http://virtuoso.openlinksw.com (triple store – data storage) Apache HTTP, http://httpd.apache.org (linked data serving) Apache Avro, http://avro.apache.org/docs/current/ (intermediate data schema) D3 JS Library, https://d3js.org/ (visualisation of RDF data using SPARQL queries) SWCs’ PoolParty GraphSearch (SPARQL based interface component for filter & faceted search) 16-déc.-16www.big-data-europe.eu
- 20. SC6 Pilot: 1st version implemented 16-déc.-16www.big-data-europe.eu https://bde.poolparty.bizGraphSearchSC6
- 21. SC6 Pilot: Pilot Evaluation Evaluation Approach SC6 Pilot (starts 01/2017): Invite municipalities to evaluate and use the system Invite community (open data, data community, BDE community, W3C) Evaluate within the participating projects (BDE, DataStories, invite: OpenBudget) BDE SC6 workshop in Cologne, 5.12.2016 Additional evaluation – tests over time with a growing amount of data a growing number of different sources & formats docked onto the system additional analytics in place 16-déc.-16www.big-data-europe.eu
- 22. How to benefit best from BDE 16-déc.-16www.big-data-europe.eu • BDE Workshops& Webinars • Use & expand the BDE Platform (BDE github) • Visit Website: news, events, community, … • Big Data Europe W3C Community Group • 7+1x Mailing Lists – stay tuned! • BDE Platform website coming soon!! • Related EC Call on Big Data, open until 02 Feb2017: Policy-development in the age of big data: data-driven policy-making, policy-modelling
- 23. Contacts: CESSDA, http://cessda.net/ Ivana Ilijasic Versic, ivana.versic@cessda.net Hossein Abroshan, hossein.abroshan@cessda.net NCSR-D, http://www.demokritos.gr/?lang=en Michalis Vafopoulos, vafopoulos@gmail.com Semantic Web Company (SWC), http://www.semantic-web.at Martin Kaltenböck, m.kaltenboeck@semantic-web.at Jürgen Jakobitsch, j.jakobitsch@semantic-web.at 16-déc.-16www.big-data-europe.eu
- 24. Questions & Contacts www.big-data-europe.eu 16-déc.-16 #BigDataEurope Martin Kaltenböck CFO, Semantic Web Company m.kaltenboeck@semantic-web.at http://www.linkedin.com/in/martinkaltenboeck https://twitter.com/kalte2707 http://de.slideshare.net/MartinKaltenboeck http://blog.semantic-web.at
Editor's Notes
- #3: Project obecjtives: Addressing each of the Societal Challenge domains (7), we have a domain representative for each & a pilot instantiation of the BDE platform for each in progress One of the challenges to Big Data opportunities is the lack of skills (data science) – our aim is to provide out of the box technology with not a lot of training required to use and apply BDE technology can be applied in multiple domains and in different phases within Data Value Chains, working with different data providers and addressing multiple objectives (as opposed to current solutions, which tend to be very specific to one data source or domain, and address one objective.
- #4: Project obecjtives: Addressing each of the Societal Challenge domains (7), we have a domain representative for each & a pilot instantiation of the BDE platform for each in progress One of the challenges to Big Data opportunities is the lack of skills (data science) – our aim is to provide out of the box technology with not a lot of training required to use and apply BDE technology can be applied in multiple domains and in different phases within Data Value Chains, working with different data providers and addressing multiple objectives (as opposed to current solutions, which tend to be very specific to one data source or domain, and address one objective.
- #6: http://www.gi.de/nc/service/informatiklexikon/detailansicht/article/big-data.html
- #8: Data Lake is a storage repository for big data scale raw data in original data formats. late binding approach to schema: “Let us decide, when we need it.” scale out architecture on commodity infrastructure, mostly with HFS/Hadoop/Spark, which gives a huge cost advantage – about factor 10 compared to data warehouses. Semantic Data Lake = Data Lake + Knowledge Graph management of structure (vocabularies/schemas, KPIs trees, metadata, …) on top of the Data Lake is performed in a knowledge graph - a complex data fabric representing all kinds of things and how they relate to each other. A knowledge graph is unique regarding flexibility, multiple views and metadata capabilities. Based on the Resource Description Framework (RDF) standard and Linked Data principles.
- #9: Data Lake is a storage repository for big data scale raw data in original data formats. late binding approach to schema: “Let us decide, when we need it.” scale out architecture on commodity infrastructure, mostly with HFS/Hadoop/Spark, which gives a huge cost advantage – about factor 10 compared to data warehouses. Semantic Data Lake = Data Lake + Knowledge Graph management of structure (vocabularies/schemas, KPIs trees, metadata, …) on top of the Data Lake is performed in a knowledge graph - a complex data fabric representing all kinds of things and how they relate to each other. A knowledge graph is unique regarding flexibility, multiple views and metadata capabilities. Based on the Resource Description Framework (RDF) standard and Linked Data principles.