Using and Extending the Eclipse Test and Performance Tools Platform (TPTP) for Data Collection in Self-Healing Systems.

Editor's Notes

• #3: Our Focus how to use and extend the logging and monitoring tooling in the Eclipse Test and Performance Tools Platform (TPTP) for data collection in self-healing systems begin by exploring the motivation and history of the TPTP project explanation of TPTP's architecture (with focus on the project's extensibility and data collection components) explanation of how these tools are utilized in the vision of Autonomic Computing Demo to illustrate the collection of log and statistical data from a sample distributed two-tier application in real-time
• #4: Today’s complexity: Computing systems now entail dozens of subsystems (e.g. servers and databases) and hundreds of components (e.g. clients and applications) all connected across internal and external networks – together creating distributed systems. Outages of business-critical systems cost up to $2.8M US per year. System-wide outages requires a set of skilled laborers to determine the root cause of the failure (this is where Problem Determination begins) Problem Determination is essentially that, investigating/determine the root cause of a system (or a set of systems) failure. Goal of Problem Determination =  Costs +  Stability Yesterday (before Autonomic Computing) Each subsystem produces it own Problem Determination Data (or PD Data), such as traces and logs, that is viewed and analyzed by different people, who have different tools. Think of all systems speaking different languages (German, English, Japanese etc) To facilitate PD, a centralized tool that understands all these systems is required. In order to achive this tool, there must be a common language format. Therefore, PD data is transformed into CBEs to create a common format across all disparate systems that speak different languages. Today (with Autonomic Computing) Centralized tooling to collect and consolidate PD Data from disparate and distributed systems into a single management tool. This common event format enables (1) viewing, (2) navigating, (3) sorting, (4) filtering, (5) correlating, (6) and searching large amounts of log and trace data across distributed systems. CBE. Originated from IBM Research as a mechanism to cheaply enable Common Base Event compliance in legacy computing systems. Based on open standards to be useful and to promote widespread adoption in a heterogeneous environment.
• #5: Eclipse an open-source, universal tools platform began as a Java Development Environment and was extended to various IDEs in 2001 founded by IBM and other industry leaders TPTP – Test and Performance Tools Platforms (2002) project formerly known as Hyades extensible framework for automated testing, trace, profiling, monitoring, and asset management. composed of a unified data model and a united set of APIs and reference tools. TPTP is the manual tools created for self-managing systems.
• #6: Organized around a client-server model which may or may not be on the same physical machines. Client - local or remote application (eg. Eclipse Workbench) that is the terminal destination of host process data. Server: Host process – application under test. Agent – component that provides services to the host process for transferring application data to attached clients. Agent Controller – a standalone component used to launch new processes and attach to agents that coexist within processes. Provides extensible agent architecture for control capabilities and native data collection. Agent hosted within the Agent Controller may communicate data back to monitoring clients (e.g. TPTP client). Consider how a user uses TPTP: From the UI the user opens the profiling wizard and attaches to a remote system. Commands are sent using the Control Framework, via the Agent Control interface from the client to the target system The appropriate sub-system, say the trace collection, begins its data collection in real-time. As events are generated, they are sent back to the client using the Agent Controller’s Data Collection Framework. These events (xml messages) are persisted into the Eclipse Modelling Framework appropriately (ie. Trace events in the Trace Model) The UI layer will then render the data model for the user to analyze. EMF - is a modeling framework and code generation facility for building tools and other applications based on a structured data model. Data models may be designed using annotated Java, XML, or UML class diagrams and converted to an EMF model specification defined in XML Metadata Interchange (XMI). Data models are populated by model loaders consuming structured XML fragments originating from a variety of sources. TPTP contains four EMF data models: Trace, Log, Statistical, Test
• #8: Java Virtual Machine Profiling Interface (JVMPI) agent for profiling local or remote Java application behavior. Used for performance details (classes or methods) responsible for the poor execution performance and to analyze and locate memory leaks. Emits trace XML fragments based on a structured Event Specification for Java Profiling (trace.xsd). << Defined Schema A trace XML loader at the client side receives and parses the trace XML fragments returned from any agent Hierarchal data model capturing interaction within the execution stacks and heap (e.g. method X in class Y on thread Z). Statistical Agent - for real-time monitoring of operating systems, web servers, database servers and application servers Emits statistical XML fragments based on a structured schema (statistical.xsd). << Defined Schema Metrics: (1) CPU usage (2) Network activity (3) Disk activity (4) Memory usage
• #10: XML message-based agent for real-time monitoring of message generating facilities (e.g. loggers). Extensible architecture for crafting proprietary Logging Agents: Defined APIs and programming model. TPTP provides CBE and Logging Agent support for popular logging types.
• #11: Open-source specification (OASIS) to provide a common and standardized taxonomy for events occurring in hardware and software Unified format and terminology for the consistent interpretation of problem determination data to circumvent varying vendor, product and version representations. Hierarchal data model capturing textual message(s) and metadata (such as timestamp, source component, situation, and severity). Events consist of message(s) and metadata resulting from an occurrence or situation Situation data, the properties describing the situation including correlation information. Reporting Component - Component observing a situation Source Component – Component experiencing the problem GLA Tooling that transforms log and trace data in proprietary formats into the Common Base Event format for use as a problem determination resource. Assists developers in adapting log and trace data to the Common Base Event format without re-writing existing applications, but using mapping rules. Originated from IBM Research as a mechanism to cheaply enable Common Base Event compliance in legacy computing systems.
• #12: - Here is the 5 levels to achieve autonomic computing – we are currently between levels 1 and 2, such that, we have built tools to help in manual analysis/problem solving. - The goal is to reach level 5, where we have autonomic systems that learn based on historical events and take a proactive approach to healing itself in the event of a failure.
• #13: We have already seen how we transform problem determination data from disparate systems into the CBE format This PD data is then sent to a centralized system using the ESB (ie. A transport mechanism – such as the Agent Controller) Over time, a human diagnosis these events. The root cause and diagnosis are persisted. Eventually, an autonomic management system can be configured to use this knowledge and AUTONOMICALLY heal the system in failure using the feedback mechanism.
• #15: We have seen how: TPTP provides a centralized, open-source and extensible mechanism for problem determination To collects and consolidates log and trace data from disparate and distributed systems into a single management tool. Utilize a common event format for (1) viewing, (2) navigating, (3) sorting, (4) filtering and (5) searching large amounts of log and trace data. (4) Lead into future extension for Self-Healing capabilities using the analysis from known problems. IBM offers commercial products that EXTEND and EXPAND the Eclipse platform and TPTP framework.
• #18: - XML message-based agent for real-time monitoring of operating systems, web servers, database servers and application servers: Windows, Linux, JBoss and JOnAS Application Server, Apache/IHS, MySQL Emits statistical XML fragments based on a structured schema (statistical.xsd). CPU usage Network activity Disk activity Memory usage - Extensible architecture for crafting proprietary Statistical Agents: Defined APIs and programming model.
• #19: Used to persist arbitrary numerical data over time. Hierarchal data model capturing structured statistical data defined by descriptors (such as metric counters) Descriptors contain snapshot observations that contain actual typed statistical data: Discrete (integer/time) Contiguous (float/time) String (String/time)