SC4 Workshop 2: Soren Auer BDE project Overview
Download as PPTX, PDF1 like274 views
The document discusses the Big Data Europe project, which aims to address societal challenges through integrating big data, software, and communities. It outlines 7 societal challenge pilots focusing on domains like health, agriculture, energy, transport, climate, social sciences, and security. Each pilot instantiates the Big Data Europe architecture to address a specific problem, such as replicating an open pharmaceutical data platform, monitoring energy production units, analyzing climate simulation data, and detecting man-made changes from remote sensing images. The document also provides details on the Big Data Europe platform components and compares it to other Hadoop distributions.
SC4 Workshop 2: Soren Auer BDE project Overview
- 1. BIG DATA EUROPE H2020 CSA (2015-17) 22.9.2016 Integrating Big Data, Software & Communities for Addressing Europe’s Societal ChallengesSC4 Workshop
- 2. Big Data in Marketing 10-oct.-16www.big-data-europe.eu
- 3. Big Data in Intelligence 10-oct.-16www.big-data-europe.eu
- 4. Big Data Europe (CSA: 2015-17) Show societal value of Big Data: 7 Domains Lower barrier for using big data technologies o Required effort and resources o Limited data science skills Help establishing cross- lingual/organizational/domain Data Value Chains 10-oct.-16
- 5. Consortium
- 6. Stakeholder Engagement & Activities M12M6 M18 M24 M30
- 7. ARCHITECTURE & COMPONENTS Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges
- 8. The three Big Data „V“ Variety is often neglected Quelle: Gesellschaft für Informatik
- 9. Adding a Semantic Layer to Data Lakes Manufacturing Marketing Sales SupportAccounting Semantic Data Lake • central place for model, schema and data historization • Combination of Scale Out (cost reduction) and semantics (increased control & flexibility) • grows incrementally (pay-as-you-go) Inbound Data Sources Outbound and Consumption Inbound Raw Data Store Data Lake (order of magnitude cheaper scalable data store) Knowledge Graph for Relationship Definition and Meta Data Frontend to Access Relationship and KPI Definition / Documentation Frontend to Access (ad hoc) Reports Outbound Data Delivery to Target Systems JSON-LD CSVW R2RMLXML2RDF © eccenca.com https://www.eccenca.com/en/products-corporate-memory.html
- 10. Current State of Platform Architecture
- 11. BDE SANSA Stack Distributed Machine Learning (ML) algorithms that work out of the box on RDF data and make use of its structure / semantics Examples: o Tensor Factorization for e.g. KB completion o Spatiotemporal analytics o Anomaly prediction o Clustering o Association rules o Decision trees 10-oct.-16www.big-data-europe.eu
- 12. Platform components Search/indexing Data processing Apache Solr Apache Spark Data acquisition Apache Flink Apache Flume Semantic Components Message passing Strabon Apache Kafka Sextant Data storage GeoTriples Hue Silk Apache Cassandra SEMAGROW ScyllaDB LIMES Apache Hive 4Store Postgis OpenLink Virtuoso 10-oct.-16www.big-data-europe.eu
- 13. BDE vs Hadoop distributions Hortonworks Cloudera MapR Bigtop BDE File System HDFS HDFS NFS HDFS HDFS Installation Native Native Native Native lightweight virtualization Plug & play components (no rigid schema) no no no no yes High Availability Single failure recovery (yarn) Single failure recovery (yarn) Self healing, mult. failure rec. Single failure recovery (yarn) Multiple Failure recovery Cost Commercial Commercial Commercial Free Free Scaling Freemium Freemium Freemium Free Free Addition of custom components Not easy No No No Yes Integration testing yes yes yes yes -- Operating systems Linux Linux Linux Linux All Management tool Ambari Cloudera manager MapR Control system - Docker swarm UI+ Custom 10-oct.-16www.big-data-europe.eu
- 14. 7 SC-PILOT INSTANTIATIONS Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges
- 15. Pilots: Overview SC1: Health & Pharm. SC2: Food & Agr. SC3: Energy SC4: Transport 10-oct.-16www.big-data-europe.eu SC5: Climate SC6: Social Sciences SC7: Security
- 16. SC1: Life Sciences & Health 10-oct.-16www.big-data-europe.eu SC1: Life Sciences & Health
- 17. SC1: Life Sciences & Health 10-oct.-16www.big-data-europe.eu Big Data Focus area: Large-scale heterogeneous pharma-research data linking & integration Selected Key Data assets: ACD Labs / ChemSpider, ChEBI, ChEMBL, ConceptWiki, DrugBank, ENZYME, Gene Ontology, GO Annotation, SwissProt, WikiPathways
- 18. SC1: Life Sciences & Health 10-oct.-16www.big-data-europe.eu Pilot 1: Replicate Open PHACTS functionality on the BDE infrastructure using Open Source solutions Reasons: • Deployment possible in-house • Apply to other domains (e.g. Agriculture) • Using extra BDE functionalities (e.g. logging, analysis)
- 19. SC2: Food & Agriculture 10-oct.-16www.big-data-europe.eu SC2: Food & Agriculture
- 20. SC2: Food & Agriculture 10-oct.-16www.big-data-europe.eu AGINFRA Big Data Focus area: Large-scale distributed agricultural data integration Selected Key Data assets: INFOODS, AQUASTAT Green Learning Network (GLN), Agricultural Bibliography Network (ABN), AgroVoc, AquaMaps, Fishbase
- 21. SC2: Food & Agriculture 10-oct.-16www.big-data-europe.eu Pilot focus area: Viticulture (from the Latin word for vine) is the science, production, and study of grapes. It deals with the series of events that occur in the vineyard.
- 23. SC3: Energy 10-oct.-16www.big-data-europe.eu Pilot focus area: System monitoring in energy production units. Big Data Focus area: Real-time turbine monitoring stream processing and analytics Selected Key Data assets: European Energy Exchange Data, smart meter sensor data, gas/fuels market/price data, consumption statistics, stratigraphic model data (geology, geophysics)
- 24. SC4: Transport 10-oct.-16www.big-data-europe.eu The Fraunhofer Society is a German research organization with 67 institutes spread throughout Germany, each focusing on different fields of applied science. The Centre for Research and Technology-Hellas (CERTH) founded in 2000 is one of the leading research centres in Greece. CERTH includes the Hellenic Institute of Transport (HIT): Land, Sea and Air Transportation as well as Sustainable Mobility services ERTICO - ITS Europe is a partnership of around 100 companies and institutions involved in the production of Intelligent Transport Systems (ITS). IAIS
- 25. SC4 Pilot Focus Area 10-oct.-16www.big-data-europe.eu Info mobility based on Mobility Pattern IdentificationPilot 4: Multisource data collection for the provision of accurate info-mobility and advanced transport planning service in Thessaloniki, Greece
- 26. SC4: Twitter data in Thessaloniki
- 27. SC4: Floating Car Data www.big-data-europe.eu Real time traffic conditions information based on a combination of traffic modeling and real time measurements (traffic flow and speed) >1.200 vehicles (one taxi fleet) • Circulating 16-24 hours/day • Pulse each 100m or 10s • 500-2.500 pulses /minute Speeds along a 2km stretch
- 29. SC5: Climate 10-oct.-16www.big-data-europe.eu Pilot focus area: Supporting data-intensive climate research Big Data Focus area: Enormous simulation time. Extremely complicated computing model. Selected Key Data assets: European Grid Infrastructure (EGI). Access to several data centres hosted at CNRS-Lyon, NCSR-D Athens, INFN-Milan, NIKhEF-Amsterdam.
- 30. SC6: Social Sciences 10-oct.-16www.big-data-europe.eu SC6: Social Sciences
- 31. SC6: Social Sciences 10-oct.-16www.big-data-europe.eu Pilot focus area: Citizens budget spending on municipal level Big Data Focus area: Statistical and research data linking & integration Selected Key Data assets: Federated social sciences data catalogs, statistical data from public data portals and statistical offices (e.g. EuroStats, UNESCO,
- 33. SC7: Security 10-oct.-16www.big-data-europe.eu Pilot focus area: Getting insight in man-made surface changes triggered by automatic detection, news, or social media information Big Data Focus area: Image data analysis Selected Key Data assets: Earth Observation data (e.g. Very High Resolution Satellite Imagery acquired from commercial providers and governmental systems) and collateral data for supporting CFSP/CSDP missions and operations
- 34. SC7: Security 10-oct.-16www.big-data-europe.eu Pilot 7: Ingestion of remote sensing images and social sensing data to detect and verify man-made changes on the Earth surface for security applications Evacuation route planning Monitoring of critical infrastructures Border security Satellite image data is HUGE and computational intensive to compare Smart ‘focus’ algorithms are needed to prioritize the analysis jobs Reasons:
- 35. WEB: www.big-data-europe.eu EMAIL: info@big-data-europe.eu PROJECT COORDINATION Prof. Sören Auer, auer © cs.uni-bonn · de (Fraunhofer IAIS) > Dr. Simon Scerri (Deputy), scerri © cs.uni-bonn · de (Fraunhofer IAIS) EIS Department/Group, Fraunhofer IAIS & CS Department Uni-Bonn, Bonn, Germany Fraunhofer IAIS: Leads Fraunhofer Big Data Alliance Questions & Contacts www.big-data-europe.eu 10-oct.-16 #BigDataEurope
Editor's Notes
- #5: Project obecjtives: Addressing each of the Societal Challenge domains (7), we have a domain representative for each & a pilot instantiation of the BDE platform for each in progress One of the challenges to Big Data opportunities is the lack of skills (data science) – our aim is to provide out of the box technology with not a lot of training required to use and apply BDE technology can be applied in multiple domains and in different phases within Data Value Chains, working with different data providers and addressing multiple objectives (as opposed to current solutions, which tend to be very specific to one data source or domain, and address one objective.
- #6: 9/16 partners: Sole or joint domain representatives of 7 SC domains (COORDINATION ROLE) Other 7/16 partners: technical support (SUPPORT ROLE) Fraunhofer coordinates the project
- #7: Centered around the 7 SC communitiess, we follow a number of iterations each having five steps: 1) Engaging with stakeholders for 2) collecting feedback and translating to requirements and identify 3) domain-specific data assets. Prototypes are deployed as 7 Pilots use-cases (4) and evaluated with the community (5) – currently we are in the first iteration at step 4 (Pilot conceptualisation and deployment).
- #9: http://www.gi.de/nc/service/informatiklexikon/detailansicht/article/big-data.html
- #10: Data Lake is a storage repository for big data scale raw data in original data formats. late binding approach to schema: “Let us decide, when we need it.” scale out architecture on commodity infrastructure, mostly with HFS/Hadoop/Spark, which gives a huge cost advantage – about factor 10 compared to data warehouses. Semantic Data Lake = Data Lake + Knowledge Graph management of structure (vocabularies/schemas, KPIs trees, metadata, …) on top of the Data Lake is performed in a knowledge graph - a complex data fabric representing all kinds of things and how they relate to each other. A knowledge graph is unique regarding flexibility, multiple views and metadata capabilities. Based on the Resource Description Framework (RDF) standard and Linked Data principles.