Big Data Technical Benchmarking, Arne Berre, BDVe Webinar series, 09/10/2018
Download as PPTX, PDF0 likes137 views
The document discusses big data benchmarking and outlines the goals of the DataBench project. It aims to develop a toolbox for both technical and business benchmarks following a holistic benchmarking approach. The toolbox will integrate existing benchmarking initiatives and identify gaps to contribute new benchmarks. It will provide a way to derive metrics and key performance indicators from benchmarks in a homogenized way. The toolbox will include a web interface for users to specify benchmarking requirements. Human: You did a great job summarizing the key points. Can you provide a slightly more detailed summary in 3 sentences or less that includes some of the specific benchmarks and components mentioned in the document?
Big Data Technical Benchmarking, Arne Berre, BDVe Webinar series, 09/10/2018
- 1. Big Data Technical Benchmarking Arne J. Berre, SINTEF, Todor Ivanov, Univ. Frankfurt, Tomas Pariente Lobo, Atos BDVe – Databench Webinar, October 9, 2018 11/10/2018 DataBench Project - GA Nr 780966 1
- 2. Technical Benchmarks in Databench Workflow 2© IDC 2© IDC Technical BenchmarksBusiness Benchmarks
- 3. Goals&Objectives • The DataBench Toolbox will be a component-based system of both vertical (holistic/business/data type driven) and horizontal (technical area based) big data benchmarks. following the layered architecture provide by the BDVA reference model. Holistic benchmarking approach for big data • It should be able to work or integrate with existing benchmarking initiatives and resources where possible. Not reinventing the wheel, but use wheels to build a new car • The Toolbox will investigate gaps of industrial significance in the big data benchmarking field and contribute to overcome them. Filling gaps • The Toolbox will implement ways to derive as much as possible the DataBench technical metrics and business KPIs from the metrics extracted from the integrated benchmarking. Homogenising metrics • It will include a web-based visualization layer to assist to the final users to specify their benchmarking requirements, such as selected benchmark, data generators, workloads, metrics and the preferred data, volume and velocity, as well as searching and monitoring capabilities. Web user interface
- 5. Identifying and Selecting Benchmarks
- 6. 7 23 Domain/Sector/Busi ness solutions KPIs (Manufact, Transport, Energy,.. Business Transport Manufacturing Energy .. Domain X … 22 Standards x x x x x x x x x x MetaData x Graph, Network x x x x x x x x x x x Text, NLP, Web x x x x x x x x x x x x x x x x x x x x x x Image, Audio x x x x Spatio Temp x x Time Series, IoT x x x x x x x Structured, BI x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 18 Visual Analytics x 17 Industrial Analytics (Descriptive, Diagnostic, Predictive, Prescriptive) x x x x 16 Machine Learning, AI, Data Science x x x x x x x x x x x x Streaming/ Realtime Processing x x x x x x x Interactive Processing x x x x x x x x x x x x x x x x x x x x x x x Batch Processing x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x Data Privacy/Security 15 Data Governance/Mgmt x 14 Data Storage x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 19 Communication & Connectivity x x 9 Cloud Services & HPC, Edge x x x x x x TPC-H TPC-DSv1 LinearRoad HadoopWorkloadExamples GridMix PigMix MRBench CALDA HiBench YCSB SWIM CloudRank-D PUMABenchmarkSuite CloudeSuite MRBS AMPLabBigDataBenchmark BigBench BigDataBench LinkBench BigFrame PRIMEBALL SemanticPublishingBenchmark(SPB) SocialNetworkBenchmark StreamBench TPCx-HS SparkBench TPCx-V BigFUN TPC-DSv2 TPCx-BB Graphalytics YahooStreamingBenchmark(YSB) DeepBench DeepMark TensorFlowBenchmarks Fathom AdBench RIoTBench HobbitBenchmark TPCx-HSv2 BigBenchV2 Sanzu Pennmachinelearningbenchmark(PMLB) OpenMLbenchmarksuites Senska DAWNBench/MLPerf IDEBench ABench 1999 2002 2004 2009 2011 2018 BDVA Reference Model 2016 2017 2014 Verticals,incl.DatatypesAnalytics,Processing,DataManagement,Infra Benchmarks 2015 2013 2012 2010 2008 2007 Updating with new Benchmarks
- 7. Identifying and Selecting Benchmarks
- 8. Dimensions of Technical Benchmarks 9
- 9. Summary Category Year Name Type Domain Data Type Micro- benchmarks 2010 HiBench Micro-benchmark Suite Micro-benchmarks, Machine Learning, SQL, Websearch, Graph, Streaming Benchmarks Structured, Text, Web Graph 2015 SparkBench Micro-benchmark Suite Machine Learning, Graph Computation, SQL, Streaming Application Structured, Text, Web Graph 2010 YCSB Micro-benchmark cloud OLTP operations Structured 2017 TPCx-IoT Micro-benchmark workloads on typical IoT Gateway systems Structured, IoT Application Benchmarks 2015 Yahoo Streaming Benchmark Application Streaming Benchmark advertisement analytics pipeline Structured, Time Series 2013 BigBench/TPCx-BB Application End-to-end Benchmark a fictional product retailer platform Structured, Text, JSON logs 2017 BigBench V2 Application End-to-end Benchmark a fictional product retailer platform Structured, Text, JSON logs 2018 ABench (Work-in- Progress) Big Data Architecture Stack Benchmark set of different workloads Structured, Text, JSON logs 10
- 10. Some of the benchmarks to integrate (I) Year Name Type 2010 HiBench Big data benchmark suite for evaluating different big data frameworks. 19 workloads including synthetic micro-benchmarks and real-world applications from 6 categories which are micro, machine learning, sql, graph, websearch and streaming. 2015 SparkBench System for benchmarking and simulating Spark jobs. Multiple workloads organized in 4 categories. 2010 Yahoo! Cloud System Benchmark (YSCB) Evaluates performance of different “key-value” and “cloud” serving systems, which do not support the ACID properties. The YCSB++ , an extension, includes many additions such as multi-tester coordination for increased load and eventual consistency measurement. 2017 TPCx-IoT Based on YCSB, but with significant changes. Workloads of data ingestion and concurrent queries simulating workloads on typical IoT Gateway systems. Dataset with data from sensors from electric power station(s) 11 Micro-benchmarks:
- 11. Some of the benchmarks to integrate (II) Year Name Type 2015 Yahoo Streaming Benchmark (YSB) The Yahoo Streaming Benchmark is a streaming application benchmark simulating an advertisement analytics pipeline. 2013 BigBench/TPCx-BB BigBench is an end-to-end, technology agnostic, application-level benchmark that tests the analytical capabilities of a Big Data platform. It is based on a fictional product retailer business model. 2017 BigBench V2 Similar to BigBench, BigBench V2 is an end-to-end, technology agnostic, application-level benchmark that tests the analytical capabilities of a Big Data platform 2018 ABench (Work-in- Progress) New type of multi-purpose Big Data benchmark covering many big data scenarios and implementations. Extends other benchmarks such as BigBench 12 Application-oriented benchmarks:
- 12. The BigBench specification comprises two key components: a data model specification a workload/query specification. The structured part of the BigBench data model is adopted from the TPC-DS data model The data model specification is implemented by a data generator, which is based on an extension of PDGF. BigBench 1.0 workload specification consists of 30 queries/workloads (10 structured from TPC-DS, and 20 adapted from a McKinsey report on Big Data use cases and opportunities). BigBench 2.0 … The BigBench data model The BigBench 2.0 overview Rabi T., et al. The Vision of BigBench 2.0, 2016. Proceedings of the Fourth Workshop on Data analytics in the Cloud. Article No. 3, http://blog.cloudera.com/blog/2014/11/bigbench-toward-an-industry-standard-benchmark-for-big-data-analytics/
- 14. Benchmark any step of the Linked Data lifecycle Ensure that benchmarking results can be found, accessed, integrated and reused easily (FAIR principles) Benchmark Big Data platforms by being the first distributed benchmarking platform for Linked data. The Hobbit platform comprises several components: Single components are implemented as independent containers. Communication between these components is done via a message bus. Everything is dockerized, from the benchmarked system to all the components Principles: • Users can test systems with the HOBBIT benchmarks without having to worry about finding standardized hardware • New benchmarks can be easily created and added to the platform by third parties. • The evaluation can be scaled out to large datasets and on distributed architectures. • The publishing and analysis of the results of different systems can be carried out in a uniform manner across the different benchmarks.
- 15. Summary • DataBench: • A framework for big data benchmarking for PPP projects and big data practitioners • We will provide methodology and tools • Added value: • An umbrella to access to multiple benchmarks • Homogenized technical metrics • Derived business KPIs, • A community around • PPP projects, industrial partners (BDVA and beyond) and benchmarking initiatives are welcomed to work with us, either to use our framework or to add new benchmarks
- 16. Big Data Benchmark session at EBDVF'2018 11/10/2018 DataBench Project - GA Nr 780966 17 Monday November 12th, 1700 – 1830,EBDVF'2018, Vienna
- 17. Arne.J.Berre@sintef.no todor@dbis.cs.uni-frankfurt.de tomas.parientelobo@atos.net Evidence Based Big Data Benchmarking to Improve Business Performance
Editor's Notes
- #2: Gabriella Cattaneo (IDC) will provide ideas on how big data benchmarking could help organizations to get better business insights and take informed decision
- #5: The approach is as follows: The DataBench Toolbox allows to reuse exisiting and new benchmarks in a uniform way, collects and calculate coherent metrics that make them comparable This Toolbox will be used by different stakeholders (BDV PPP projects, industries, etc.) to benchmark and compare with others The metrics will be them derived and transformed into a set of coherent business KPIs of industrial significance
- #15: I miss the Business KPÎs formulas from WP2 to WP3. WP3 will implement the transformation, but the design should come form WP2. I miss also a functional overview of the whole process independent from the WP view (more conceptual)