Second review presentation
0 likes442 views
This document describes the design and implementation of a service monitoring console within eBay's Turmeric SOA framework. Key components include the Service Query Metrics Service (SQMS) for logging metrics, a metrics storage provider interface, and the Monitoring Console (TMC) service for displaying metrics and building the dashboard UI using collected data. The system allows centralized collection of application logs and generation of custom reports. A partial offline data aggregator was also implemented to work with the Cassandra storage backend.
![References
[1]Jeffrey Dean, Sanjay Ghemawat, MapReduce: Simplified Data
Processing on Large Clusters, In Sixth Symposium on Operating
System Design and Implementation(OSDI’04), San Francisco, CA,
December, 2004
[2] Eben Hewitt, Cassandra: The Definitive Guide, O’Reilly
Publications, November 2010.
[3] eBay Open Source Project, Turmeric SOA platform,
http:
//www.ebayopensource.org/index.php/Turmeric/HomePage](https://image.slidesharecdn.com/secondreviewpresentation-120429005633-phpapp01/85/Second-review-presentation-41-320.jpg)
![References
[4] eBay Open Source Project, Documentation of Turmeric SOA
platform,
https://www.ebayopensource.org/wiki/display/
TURMERICDOC110GA/Turmeric+Documentation+Overview
[5] eBay Open Source Project, Turmeric Source Code,
http://www.github.com/ebayopensource
[6] Internal eBay documentation
[7] Google Web Toolkit,
http://code.google.com/webtoolkit
[8] Apache Cassandra,
http://cassandra.apache.org/](https://image.slidesharecdn.com/secondreviewpresentation-120429005633-phpapp01/85/Second-review-presentation-42-320.jpg)
Second review presentation
- 1. Design and Implementation of a Service Monitoring Console within a Service Oriented Architecture Framework Arvind Krishnaa .J 31508104017 Guided By Dr. Chitra Babu HOD/CSE SSN College of Engineering Second Review - 30th March, 2012
- 2. Turmeric Runtime Platform 1 Client Runtime - Client application code written by the consumers of the service. Service Invocation Framework(SIF)
- 3. Turmeric Runtime Platform 1 Client Runtime - Client application code written by the consumers of the service. Service Invocation Framework(SIF) 2 Server Runtime - Deployment platform for developers to publish web services. Service Provider Framework (SPF)
- 4. Turmeric Runtime Platform 1 Client Runtime - Client application code written by the consumers of the service. Service Invocation Framework(SIF) 2 Server Runtime - Deployment platform for developers to publish web services. Service Provider Framework (SPF) 3 CodeGen - Generate instrumentation code for deploying services onto Turmeric.
- 5. Central Application Logging Framework Centralized collection of application server logs
- 6. Central Application Logging Framework Centralized collection of application server logs Base for collecting data for the monitoring console
- 7. Central Application Logging Framework Centralized collection of application server logs Base for collecting data for the monitoring console Ability to generate custom reports from this data
- 8. Central Application Logging Framework Centralized collection of application server logs Base for collecting data for the monitoring console Ability to generate custom reports from this data TMC is a specific instance of a report created using CAL
- 9. Architecture of the Monitoring Console 1 Can be abstracted to two frameworks running in parallel - metrics logging framework and metrics handler framework
- 10. Architecture of the Monitoring Console 1 Can be abstracted to two frameworks running in parallel - metrics logging framework and metrics handler framework 2 Both frameworks are web services deployed on an application server
- 11. Architecture of the Monitoring Console 1 Can be abstracted to two frameworks running in parallel - metrics logging framework and metrics handler framework 2 Both frameworks are web services deployed on an application server 3 SOA Query Metrics Service (sqms) - Logging logging handler framework
- 12. Architecture of the Monitoring Console 1 Can be abstracted to two frameworks running in parallel - metrics logging framework and metrics handler framework 2 Both frameworks are web services deployed on an application server 3 SOA Query Metrics Service (sqms) - Logging logging handler framework 4 Monitoring Console (console) - Metrics handler framework
- 13. SOA Query Metrics Service (SQMS) Figure 1: Architecture of SQMS
- 14. Working of SQMS 1 SPFServlet is the main servlet which handles the requests to the Turmeric runtime. 2 Initializes the CAL logger for logging metrics published by Turmeric services. 3 Metric data logged in a format called heartbeats. 4 Log message casted into a JMX MBean like object. 5 The MBean object is dispatched to the JMX servlet and published to a controller interface (Agent Service)
- 15. Metrics Storage Provider Java interface for defining the underlying storage mechanism
- 16. Metrics Storage Provider Java interface for defining the underlying storage mechanism Provides the general contract for defining the queries to store metrics into the persistence mechanism
- 17. Metrics Storage Provider Java interface for defining the underlying storage mechanism Provides the general contract for defining the queries to store metrics into the persistence mechanism Can be extended to provide custom implementations, for various Hibernate- compliant databases.
- 18. Metrics Storage Provider Java interface for defining the underlying storage mechanism Provides the general contract for defining the queries to store metrics into the persistence mechanism Can be extended to provide custom implementations, for various Hibernate- compliant databases. Currently two implementations are provided - DAO(based on a MySql storage) and Cassandra (partially implemented)
- 19. Metrics Logging 1 saveMetricValue(): It receives the timestamp, metric name, and a key-value pair to store the metric value, and converts it to row oriented format for MySQL storage (or name-value pair for Cassandra) 2 saveMetricSnapshot(): Encapsulates the above received metrics objects into a MBean instance and sends it to the JMX server for publication.
- 20. Logging helper utilities 1 Snapshot File Logger: Logs the aggregated metrics into a .log file 2 Differential Snapshot View Logger: Calculating the percentage changes between the metrics collected on two different days. 3 Monitoring System: Schedules the metrics logging process to be performed at regular intervals. Presently it is set at 60 seconds.
- 21. Logging Handlers 1 Metrics Storage Provider Interface Defines a framework which runs two MetricsSnapshotSchedulers in separate background threads, one for the client-side metrics and the other for the server-side metrics Calls the saveMetricsSnapshot() method to store the metric data. 2 DAO Metrics Provider : The provider implementation for MySQL persistence mechanism 3 Cassandra Storage Provider: Implementation for Cassandra storage. Any custom persistence mechanism can also be integrated.
- 22. Monitoring Console Service (TMC) Figure 2: Architecture of Monitoring Dashboard
- 23. Implementation of TMC service Responsible for constructing user interface of monitoring dashboard
- 24. Implementation of TMC service Responsible for constructing user interface of monitoring dashboard Retrieves the encapsulated data in the form of a MetricsData instance
- 25. Implementation of TMC service Responsible for constructing user interface of monitoring dashboard Retrieves the encapsulated data in the form of a MetricsData instance This object is serialized into JSON format and sent to the Google Web Toolkit’s charting API
- 26. Implementation of TMC service Responsible for constructing user interface of monitoring dashboard Retrieves the encapsulated data in the form of a MetricsData instance This object is serialized into JSON format and sent to the Google Web Toolkit’s charting API GWT API constructs the trends chart based on the JSON data
- 27. Implementation of TMC service Responsible for constructing user interface of monitoring dashboard Retrieves the encapsulated data in the form of a MetricsData instance This object is serialized into JSON format and sent to the Google Web Toolkit’s charting API GWT API constructs the trends chart based on the JSON data A REST URI is created for every JSON object, and a link is created in the dashboard.
- 28. Implementation of TMC service Responsible for constructing user interface of monitoring dashboard Retrieves the encapsulated data in the form of a MetricsData instance This object is serialized into JSON format and sent to the Google Web Toolkit’s charting API GWT API constructs the trends chart based on the JSON data A REST URI is created for every JSON object, and a link is created in the dashboard. The metrics detail can be exported into a CSV file as well.
- 29. Offline Aggregator - A Partial Implementation Objective of collecting the distributed metrics across cluster of nodes, by utilizing Cassandra storage. Only the storage into Cassandra keyspace has been implemented Hooks are provided to read data across distributed nodes, to make this feature full-fledged. Three main classes for implementing offline-aggregation 1 Readers 2 Writers 3 Erasers
- 30. Overview of Readers, Writers and Erasers Readers: Read the data from the offline cluster, de-serialize the MetricsData objects and display it in the dashboard. Different reader classes are available to customize metrics reading, in the future. (Classes need to be extended with custom implementation)
- 31. Overview of Readers, Writers and Erasers Readers: Read the data from the offline cluster, de-serialize the MetricsData objects and display it in the dashboard. Different reader classes are available to customize metrics reading, in the future. (Classes need to be extended with custom implementation) Writers: Write data into the offline cluster, by serializing the metrics into MetricsData objects. Different writer base classes available for extending with custom implementations.
- 32. Overview of Readers, Writers and Erasers Readers: Read the data from the offline cluster, de-serialize the MetricsData objects and display it in the dashboard. Different reader classes are available to customize metrics reading, in the future. (Classes need to be extended with custom implementation) Writers: Write data into the offline cluster, by serializing the metrics into MetricsData objects. Different writer base classes available for extending with custom implementations. Erasers: Erase the data from the offline cluster.
- 33. Overview of Readers, Writers and Erasers Figure 3: Overview of Reader, Writers and Erasers
- 34. Partial Demo 1 Nuts and bolts of Turmeric SOA services 2 Sample REST URLs for metrics 3 Walkthrough of database schema and significant implementation classes 4 Overview of data aggregator tool 5 Tour of the user interface
- 35. Screenshots - SMC Dashboard for an eBay service Figure 4: ShortURI Service
- 36. Screenshots - Turmeric Top volume graph Figure 5: Example Echo Service
- 37. Screenshots - Turmeric Consumer trends Figure 6: Example Echo Service
- 38. Data Aggregator tool Figure 7: Ad hoc querying
- 39. Data Aggregator tool – Contd. Figure 8: Ad hoc querying
- 40. Implementation Schedule Figure 9: Project Roadmap
- 41. References [1]Jeffrey Dean, Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, In Sixth Symposium on Operating System Design and Implementation(OSDI’04), San Francisco, CA, December, 2004 [2] Eben Hewitt, Cassandra: The Definitive Guide, O’Reilly Publications, November 2010. [3] eBay Open Source Project, Turmeric SOA platform, http: //www.ebayopensource.org/index.php/Turmeric/HomePage
- 42. References [4] eBay Open Source Project, Documentation of Turmeric SOA platform, https://www.ebayopensource.org/wiki/display/ TURMERICDOC110GA/Turmeric+Documentation+Overview [5] eBay Open Source Project, Turmeric Source Code, http://www.github.com/ebayopensource [6] Internal eBay documentation [7] Google Web Toolkit, http://code.google.com/webtoolkit [8] Apache Cassandra, http://cassandra.apache.org/