TUW-ASE Summer 2015 - Quality of Result-aware data analytics
0 likes886 views
The document discusses quality of results (QoR) in data analytics workflows, detailing various workflow structures, systems, and metrics involved in ensuring the effectiveness of analytics processes. It addresses challenges regarding data quality, performance evaluation, and the integration of QoR-aware systems, emphasizing the need for optimization in composition and execution of workflows. The document also highlights the importance of integrating human and software evaluations for accurate quality assessment in complex workflows.
TUW-ASE Summer 2015 - Quality of Result-aware data analytics
- 1. Quality of Result-aware data analytics Hong-Linh Truong Distributed Systems Group, Vienna University of Technology truong@dsg.tuwien.ac.at http://dsg.tuwien.ac.at/staff/truong 1ASE Summer 2015 Advanced Services Engineering, Summer 2015 Advanced Services Engineering, Summer 2015
- 2. Outline Data analytics: workflow structures and systems Enable quality of results for data analytics Quality of data in data analytics workflows ASE Summer 2015 2
- 3. Data Analytics Conceptual View ASE Summer 2015 3 Data Processing FrameworksData Processing Frameworks Streaming/Online Data Processing Batch Data Processing Hybrid Data Processing Static data(Near) realtime data Decision Data Analysis Analytics, Tools, Processes & Models
- 4. Data analytics workflows ASE Summer 2015 4 Things People DaaSDaaS Computation Service Computation Service We use the term „workflow“ in a generic meaning!!!
- 5. Different views of (data analytics) workflow systems 5 View Domain view Business Workflow Scientific/E- science Workflow Data/Computati on view Data intensive workflow Computatio n intensive workflow Human- intensive workflow System view Grid workflow Enterprise workflow Cloud- based workflow Execution model view Service- based workflow Batch job workflow Interactive workflow ASE Summer 2015
- 6. Pros and cons of (data analytics) workflow systems ASE Summer 2015 6 Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields. 2006. Workflows for E-Science: Scientific Workflows for Grids. Springer-Verlag New York, Inc., Secaucus, NJ, USA. Bertram Ludäscher, Mathias Weske, Timothy M. McPhillips, Shawn Bowers: Scientific Workflows: Business as Usual? BPM 2009: 31-47 Mirko Sonntag, Dimka Karastoyanova, Frank Leymann: The Missing Features of Workflow Systems for Scientific Computations. Software Engineering (Workshops) 2010: 209-216 Lavanya Ramakrishnan and Beth Plale. 2010. A multi-dimensional classification model for scientific workflow characteristics. In Proceedings of the 1st International Workshop on Workflow Approaches to New Data-centric Science (Wands '10). ACM, New York, NY, USA, , Article 4 , 12 pages. DOI=10.1145/1833398.1833402 http://doi.acm.org/10.1145/1833398.1833402 Jia Yu and Rajkumar Buyya. 2005. A taxonomy of scientific workflow systems for grid computing. SIGMOD Rec. 34, 3 (September 2005), 44-49. DOI=10.1145/1084805.1084814 http://doi.acm.org/10.1145/1084805.1084814 Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields. 2006. Workflows for E-Science: Scientific Workflows for Grids. Springer-Verlag New York, Inc., Secaucus, NJ, USA. Bertram Ludäscher, Mathias Weske, Timothy M. McPhillips, Shawn Bowers: Scientific Workflows: Business as Usual? BPM 2009: 31-47 Mirko Sonntag, Dimka Karastoyanova, Frank Leymann: The Missing Features of Workflow Systems for Scientific Computations. Software Engineering (Workshops) 2010: 209-216 Lavanya Ramakrishnan and Beth Plale. 2010. A multi-dimensional classification model for scientific workflow characteristics. In Proceedings of the 1st International Workshop on Workflow Approaches to New Data-centric Science (Wands '10). ACM, New York, NY, USA, , Article 4 , 12 pages. DOI=10.1145/1833398.1833402 http://doi.acm.org/10.1145/1833398.1833402 Jia Yu and Rajkumar Buyya. 2005. A taxonomy of scientific workflow systems for grid computing. SIGMOD Rec. 34, 3 (September 2005), 44-49. DOI=10.1145/1084805.1084814 http://doi.acm.org/10.1145/1084805.1084814
- 7. Hierarchical view of workflows (1) mProject1Service.java public void mProject1() { } mProject1Service.java public void mProject1() { } WorkflowWorkflow A();A(); <parallel> </parallel> <parallel> </parallel> Workflow Region nWorkflow Region n Activity mActivity m Invoked Application mInvoked Application m Code Region 1 Code Region 1 Code Region q Code Region q Code Region … Code Region … <activity name="mProject1"> <executable name="mProject1"/> </activity> <activity name="mProject1"> <executable name="mProject1"/> </activity> <activity name="mProject2"> <executable name="mProject2"/> </activity> <activity name="mProject2"> <executable name="mProject2"/> </activity> while () { ... } while () { ... } Hong Linh Truong, Schahram Dustdar, Thomas Fahringer: Performance metrics and ontologies for Grid workflows. Future Generation Comp. Syst. 23(6): 760-772 (2007) Hong Linh Truong, Schahram Dustdar, Thomas Fahringer: Performance metrics and ontologies for Grid workflows. Future Generation Comp. Syst. 23(6): 760-772 (2007) ASE Summer 2015 7
- 8. Representing and programming data analytics workflows Programming languages General- and specific-purpose programming languages, such as Java, Python, Swift Programming models, such as MapReduce, Hadoop, Complex event processing, Spark Descriptive languages BPEL and several languages designed for specific workflow engines They can also be combined 8ASE Summer 2015
- 9. Data analytics workflow execution models ASE Summer 2015 9 Data analytics workflows Data analytics workflows Execution EngineExecution Engine Local SchedulerLocal Scheduler jobjob jobjob jobjob jobjob Web serviceWeb serviceWeb service Web service People
- 10. Data analytics workflow execution models ASE Summer 2015 10 Data analytics workflows Data analytics workflows Execution EngineExecution Engine Service unit Local input data Analytics Results Web service MapReduce/Hadoop Sub-Workflow MPI Other solutions Servers/Cloud/Cluster
- 11. Examples of systems and frameworks for data analytics workflows ASE Summer 2015 11 ASKALONASKALON KEPLERKEPLER TAVERNATAVERNA TRIDENTTRIDENT Apache ODE + WS-BPEL Apache ODE + WS-BPEL PegasusPegasus JOperaJOperaADEPTADEPT MapReduce/HadoopMapReduce/Hadoop SwiftSwiftRR
- 12. Some examples (1) ASE Summer 2015 12 Source: Gideon Juve, Ewa Deelman, G. Bruce Berriman, Benjamin P. Berman, Philip Maechling: An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2. J. Grid Comput. 10(1): 5-21 (2012) Source: Gideon Juve, Ewa Deelman, G. Bruce Berriman, Benjamin P. Berman, Philip Maechling: An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2. J. Grid Comput. 10(1): 5-21 (2012)
- 13. Some examples (2) ASE Summer 2015 13 Source: http://www.dps.uibk.ac.at/projects/brokerage/Source: http://www.dps.uibk.ac.at/projects/brokerage/
- 14. Some examples (3) ASE Summer 2015 14 Source: Cesare Pautasso, Thomas Heinis, Gustavo Alonso: JOpera: Autonomic Service Orchestration. IEEE Data Eng. Bull. 29(3): 32-39 (2006) Source: Cesare Pautasso, Thomas Heinis, Gustavo Alonso: JOpera: Autonomic Service Orchestration. IEEE Data Eng. Bull. 29(3): 32-39 (2006)
- 15. Some examples (4) ASE Summer 2015 15 Source: Sudipto Das, Yannis Sismanis, Kevin S. Beyer, Rainer Gemulla, Peter J. Haas, and John McPherson. 2010. Ricardo: integrating R and Hadoop. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data (SIGMOD '10). ACM, New York, NY, USA, 987-998. DOI=10.1145/1807167.1807275 http://doi.acm.org/10.1145/1807167.1807275 Source: Sudipto Das, Yannis Sismanis, Kevin S. Beyer, Rainer Gemulla, Peter J. Haas, and John McPherson. 2010. Ricardo: integrating R and Hadoop. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data (SIGMOD '10). ACM, New York, NY, USA, 987-998. DOI=10.1145/1807167.1807275 http://doi.acm.org/10.1145/1807167.1807275
- 16. QUALITY OF RESULTS ASE Summer 2015 16
- 17. Recall - QoR Characterize the results of analytics processes Different elements of QoR Performance Data quality Cost Form/data format of output results Etc. Customer: expects QoR Provider: offers QoR and enforces QoR ASE Summer 2015 17
- 18. Performance and Data Quality Aspects 18 Data Analytics Data in Data out Executed on Analytics Processes uses Execution time? Performance Overhead? Memory Consumption? Is the data good enough? How bad data impacts on performance? Is the data good enough to be stored and shared? Data quality metrics and models are strongly domain-specific Data quality metrics and models are strongly domain-specific Which processes should be used? ASE Summer 2015 18
- 19. Model QoR and analytics processes 19ASE Summer 2015 Hong Linh Truong, Schahram Dustdar: Principles of Software-Defined Elastic Systems for Big Data Analytics. IC2E 2014: 562-567 Hong Linh Truong, Schahram Dustdar: Principles of Software-Defined Elastic Systems for Big Data Analytics. IC2E 2014: 562-567
- 20. SO HOW DO WE ENABLE QOR-AWARE ANALYTICS? ASE Summer 2015 20
- 21. Solutions Computational resources provisioning? Replication of analytics ? Performance and cost measurement and optimization? Improve quality of input data ? Improve the quality of output data? ASE Summer 2015 21
- 22. 22 Hong Linh Truong, Peter Brunner, Vlad Nae, Thomas Fahringer: DIPAS: A distributed performance analysis service for grid service-based workflows. Future Generation Comp. Syst. 25(4): 385-398 (2009) Hong Linh Truong, Peter Brunner, Vlad Nae, Thomas Fahringer: DIPAS: A distributed performance analysis service for grid service-based workflows. Future Generation Comp. Syst. 25(4): 385-398 (2009) Well-addressed concerns -- performance ASE Summer 2015 22
- 23. Well-addressed concerns – performance/cost ASE Summer 2015 23 Source: David Chiu, Sagar Deshpande, Gagan Agrawal, Rongxing Li: Cost and accuracy sensitive dynamic workflow composition over grid environments. GRID 2008: 9-16 Source: David Chiu, Sagar Deshpande, Gagan Agrawal, Rongxing Li: Cost and accuracy sensitive dynamic workflow composition over grid environments. GRID 2008: 9-16
- 24. QUALITY OF DATA IN DATA ANALYTICS WORKFLOWS ASE Summer 2015 24
- 25. Very little support Qurator workbench “Personal quality models” can be expressed and embedded into query processors or workflows. Assume that quality evidence is presented Kepler A data quality monitor allows user to specify quality thresholds. Expect that rules can be used to control the execution based on quality. ASE Summer 2015 25 P Missier, S M Embury, M Greenwood, A D Preece, & B Jin, Managing Information Quality in e-Science: the Qurator Workbench, Proc ACM International Conference on Management of Data (SIGMOD 2007), ACM Press, pages 1150- 1152, 2007. Aisa Na’im, Daniel Crawl,Maria Indrawan, Ilkay Altintas, and Shulei Sun. Monitoring data quality in kepler. In Salim Hariri and Kate Keahey, editors, HPDC, pages 560–564. ACM, 2010. P Missier, S M Embury, M Greenwood, A D Preece, & B Jin, Managing Information Quality in e-Science: the Qurator Workbench, Proc ACM International Conference on Management of Data (SIGMOD 2007), ACM Press, pages 1150- 1152, 2007. Aisa Na’im, Daniel Crawl,Maria Indrawan, Ilkay Altintas, and Shulei Sun. Monitoring data quality in kepler. In Salim Hariri and Kate Keahey, editors, HPDC, pages 560–564. ACM, 2010.
- 26. Research questions What are main QoD metrics, what are the relationship between QoD metrics and other service level objectives, and what are their roles and possible trade-offs? How to support different domain-specific QoD models and link them to workflow structures? How to model, evaluate and estimate QoD associated with data movement into, within, and out to workflows? When and where software or scientists can perform automatic or manual QoD measurement and analysis How to optimize the workflow composition and execution based on QoD specification? How does QoD impact on the provisioning of data services, computational services and supporting services? ASE Summer 2015 26
- 27. Approach ASE Summer 2015 27 Core models, techniques and algorithms to allow the modeling and evaluating QoD metrics Core models, techniques and algorithms to allow the modeling and evaluating QoD metrics QoD-aware composition and executionQoD-aware composition and execution QoD-aware service provisioning and infrastructure optimization QoD-aware service provisioning and infrastructure optimization
- 28. Modeling and evaluating QoD metrics for data analytics workflows ASE Summer 2015 28
- 29. QoD-aware optimization for data analytics workflow composition and execution ASE Summer 2015 29
- 30. HOW TO INTEGRATE QOD EVALUATORS? AND WHICH CONCERNS NEED TO BE CONSIDERED? ASE Summer 2015 30
- 31. QoD metrics evaluation Domain-specific metrics Need specific tools and expertise for determining metrics Evaluation Cannot done by software only: humans are required Complex integration model Where to put QoD evaluators and why? How evaluators obtain the data to be evaluated? Impact of QoD evaluation on performance of data analytics workflows ASE Summer 2015 31
- 32. WHAT KIND OF OPTIMIZATION CAN BE DONE? ASE Summer 2015 32
- 33. QoD-aware optimization for data analytics workflows Improving quality of results Reducing analytics costs and time Enabling early failure detection Enabling elasticitiy of services provisioning Enabling elastic data analytics support Etc. ASE Summer 2015 33
- 34. EXAMPLE: QOD-AWARE SIMULATION WORKFLOWS ASE Summer 2015 34
- 35. 35 QoD-aware simulation workflows Michael Reiter, Hong Linh Truong, Schahram Dustdar, Dimka Karastoyanova, Robert Krause, Frank Leymann, Dieter Pahr: On Analyzing Quality of Data Influences on Performance of Finite Elements Driven Computational Simulations. Euro-Par 2012: 793-804 Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112 Michael Reiter, Hong Linh Truong, Schahram Dustdar, Dimka Karastoyanova, Robert Krause, Frank Leymann, Dieter Pahr: On Analyzing Quality of Data Influences on Performance of Finite Elements Driven Computational Simulations. Euro-Par 2012: 793-804 Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112 ASE Summer 2015
- 36. Hybrid resources needed for quality evaluation Challenges: Subjective and objective evaluation Long running processes Our approach Different QoD measurements Human and software tasks 36ASE Summer 2015
- 37. 37 Evaluating quality of data in workflows Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112 Michael Reiter, Uwe Breitenbücher, Schahram Dustdar, Dimka Karastoyanova, Frank Leymann, Hong Linh Truong: A Novel Framework for Monitoring and Analyzing Quality of Data in Simulation Workflows. eScience 2011: 105-112 ASE Summer 2015
- 38. QoD Evaluator Software-based QoD evaluators Can be provided under libraries integrated into invoked applications Web services-based evaluators Human-based QoD evaluators Built based on the concept human-based services Can be interfaces via Human-Task Simple mapping at the moment Human resources from clouds/crowds ASE Summer 2015 38
- 39. Open issues: quality-of-result (QoR) driven workflows Tradeoffs are challenging How to support QoR driven analytics? Some basic steps Conceptualize expected QoR Associate the expected QoR with workflow activities Use the expected QoR to match/select underlying services (e.g., data sources, cloud IaaS, etc Utilize the expected QoR and the measured QoR and apply elasticity principles for Refine the workflow structure Provision computation, network and data ASE Summer 2015 39
- 40. Exercises Read mentioned papers Discuss pros and cons of descriptive languages - and programming languages – based data analytics workflows Examine how QoD evaluators can be integrated into different programming models for QoR- aware data analytics workflows Implement some QoD evaluators Develop techniques for determining places where QoD evaluators can be performed ASE Summer 2015 40
- 41. 41 Thanks for your attention Hong-Linh Truong Distributed Systems Group Vienna University of Technology truong@dsg.tuwien.ac.at http://dsg.tuwien.ac.at/staff/truong ASE Summer 2015