Embedding Existing Heterogeneous Monitoring Techniques into a Lightweight, Distributed Integration Platform
- 1. Embedding Existing Heterogeneous Monitoring Techniques into a Lightweight, Distributed Integration Platform ADVCOMP 2009 (October 12th 2009, Sliema) Jan Niemann, GNS Systems Doreen Seider, German Aerospace Center (DLR) Folie 1 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into … > 12.10.2009
- 2. DLR Research Institution German Aerospace Center Space Agency Project Management Agency GNS Systems UNIX/Windows System Management High-Performance-Computing Engineering Data Management Software Development Folie 2 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 3. Motivation Monitoring in HPC/CAE Environments In CAE environments, a lot of software components do monitoring! Folie 3 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 4. Motivation Different Purposes for Monitoring Resource Management Monitoring Resources like Load, Licenses, … Examples: Grid Engine, PBS, Platform LSF Health Management Monitoring Resources like Load, Heat, SMART-Status Examples: Ganglia, Nagios Resource Planning Esp. License Monitoring Examples: We have created License Monitoring and Accounting Solutions for some of our customers. Folie 4 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 5. Motivation A typical setup has 3 Software Components that are monitoring All with different types of Sensors All with different administrative and user interfaces Some Resources are monitored more than once (Load, Licenses) Let‘s do it once! But allow reusability of existing solutions! Folie 5 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 6. Monitoring Techniques Ganglia (gmetric) Basically, a Sensor-Script for Ganglia just reports a simple value. May be written in sh, perl, python, … Whatever you/your Administrators like! #!/bin/sh echo 60.3 # gmetric --name temperature --value > $(cputemp) --type float --units Celsius There are a lot of these scripts available on the internet for free. Measure Infiniband performance Temperatures via IPMI … Folie 6 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 7. Monitoring Techniques Grid Engine (Load Sensors) We use a lot of scripts for the Grid Engine to make the scheduler aware of license usage. A bit more complicated, but again: may be written in any language. #!/bin/sh while read x ; do echo begin echo “global:lic_abq_free:5“ echo “global:lic_abq_used:8“ echo “global:lic_abq_total:13“ echo end done Platform LSF and PBS use similar mechanics. Administrators know how to write these scripts. We think that acceptability is a key factor for success! Folie 7 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 8. Realizing the Monitoring Environment Requirements Collect heterogeneous monitoring data from distributed hosts Provide them centralized via one unified interface Reuse existing sensor scripts Solution Unify accumulated data by wrapping the scripts with adapters Embed the scripts (i.e. the adaptors) in an integration platform Make use of the platform‘s infrastructure to collect the data Realize an embedded component providing the data Folie 8 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 9. Realizing the Monitoring Environment Unify the Heterogeneous Monitoring Data Wrapping each sensor script with an adapter which Executes the script Transforms the data Provides them via one unified interface Folie 9 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 10. Realizing the Monitoring Environment Integration Platform Unified environment with base software components Data management, distribution, privilege management, ... Integrating diverse and specific applications Benefit Integrated applications simply use already realized software components Enables basically independent applications to Interact Operate on same data Base software components are realized only once Folie 10 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 11. Realizing the Monitoring Environment Remote Component Environment (RCE) RCE is such an integration platform Characteristics Component-based Distributed OSGi/Eclipse RCP Developed by German Aerospace Center (DLR) – http://www.dlr.de Fraunhofer SCAI – http://scai.fraunhofer.de http://sourceforge.net/projects/rcenvironment Folie 11 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 12. Realizing the Monitoring Environment Remote Component Environment (RCE) Distributed deployment scenario of RCE Folie 12 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 13. Realizing the Monitoring Environment Adapter concept Distributed integration platform RCE Folie 13 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 14. Monitoring Environment On Top of RCE Folie 14 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 15. Outlook Implement clients using the monitoring environment GUI Alert system Resource management system Implement more adapters as needed Folie 15 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
- 16. Questions? Folie 16 Jan.Niemann@gns-systems.de, Doreen.Seider@dlr.de ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009