e-Infrastructure Integration-with gCube
Download as PPT, PDF0 likes498 views
The document summarizes key aspects of the gCube architecture and its approaches to enabling interoperability across heterogeneous resources and communities. gCube aims to hide heterogeneity by providing unified access to diverse resources, and embrace heterogeneity by supporting multiple protocols and models. It utilizes various approaches like blackboard communication, wrappers, and adapters to achieve interoperability for resource discovery, content access, data discovery, process execution, and security. Applications like AquaMaps and time series management are also discussed.
![Interoperability Approaches: Content Interoperability[1/2] gCube Open Content Management Architecture (OCMA) Assumption data stored in different storage back-ends diverse locations, models, access types few common primitives: documents, collections, repositories gCube allows to reach content that lies outside system expose content (reachable from) inside system perform coarse-grained as well as fine-grained retrieval, update, and addressing Runtime scalability autonomic read-only state replication, maximize throughput, minimize response time: discovery-time load balancing (through IS) reduce latencies Software plugin-based architecture to reduce development costs (plugins over Storage systems)](https://image.slidesharecdn.com/e-infrastructureintegrationwithgcube-110418091614-phpapp01/85/e-Infrastructure-Integration-with-gCube-13-320.jpg)
![Interoperability Approaches: Content Interoperability[2/2] T 1 T 2 adapts adapts factory gDoc … gDoc Read gDoc Write Content Manager Service ( OCMA Service) Adapts gCube doc model ( gDoc ) to an unbounded number of back-end types](https://image.slidesharecdn.com/e-infrastructureintegrationwithgcube-110418091614-phpapp01/85/e-Infrastructure-Integration-with-gCube-14-320.jpg)
![gCube offers solutions to: Decouple the business domain and infrastructure specific logic from the core “execution” functionality Invocate a wide range of logic components: SOAP and REST WebServices, Shell Scripts, Executable Binaries, POJOs, … Support most of the execution paradigms: batch, map-reduce, synchronous call Bridges key distributed computation technologies: grid (gLite and Globus), Condor, Hadoop Control and monitor the execution of a processing flow Staging of data among different storage providers Streaming data among computation elements Interoperability Approaches : Process Execution [1/2]](https://image.slidesharecdn.com/e-infrastructureintegrationwithgcube-110418091614-phpapp01/85/e-Infrastructure-Integration-with-gCube-16-320.jpg)
![Interoperability Approaches : Process Execution [2/2] By using adaptors that operate on a specific third party language and translate them into native constructs, allow for the creation of complex workflows that exploit several diverse technologies deployed on different infrastructures](https://image.slidesharecdn.com/e-infrastructureintegrationwithgcube-110418091614-phpapp01/85/e-Infrastructure-Integration-with-gCube-17-320.jpg)
![Interoperability Approaches : Security [1/2] gCube offers solutions : To secure access to gCube resources for interoperable external systems (incoming security) To ensure Interoperability of gCube security mechanisms with standards compliant security systems (reuse) To facilitate secure access to external resources for gCube services (outgoing)](https://image.slidesharecdn.com/e-infrastructureintegrationwithgcube-110418091614-phpapp01/85/e-Infrastructure-Integration-with-gCube-18-320.jpg)
![Authz: XACML for authz request/response protocol and policy definition SAML assertions to transport user/service authN information Argus-based approach (EMI Authz framework) having pluggable design to integrate additional PIPs SAML Profile for XACML 2.0 following the OASIS Authorization Interoperability Profile Specification AuthN: Production level SSL/HTTPS support Key- and Trust-Manager Interoperability Approaches : Security [2/2]](https://image.slidesharecdn.com/e-infrastructureintegrationwithgcube-110418091614-phpapp01/85/e-Infrastructure-Integration-with-gCube-19-320.jpg)
e-Infrastructure Integration-with gCube
- 1. e-Infrastructure Integration with gCube Andrea Manzi ( CERN ) Pasquale Pagano ( ISTI-CNR ) EGI User Forum 13 April 2011 Vilnius ( Lithuania ) www.d4science.eu
- 2. Outline D4Science II Ecosystem gCube architecture Interoperability approaches Resource Discovery Data Storage & Access Data Discovery Data Process Security Applications AquaMaps Time Series
- 3. D4Science II Ecosystem Heterogeneous resources Heterogeneous computational platforms Rich set of legacy applications Multiple administrative domains Evolving communities D4SCIENCE INFRASTRUCTURE Hadoop EGEE/EGI INSPIRE DRIVER GENESI-DR AquaMaps FAO Geonetwork FAO FIGIS Community A Community B
- 4. gCube architecture gCube run-time environment gCube Definition and Management Services gCube Application Services Presentation Services Portlets Application Support Layer Information Organization Services Storage Management Collection - Content - Metadata - Annotation -… Management Information Access Services Search Framework Ontology Management Personalization Service Index Management Framework DIR Support Framework Process Execution Management VRE Management Information System Security gCube Container gCore Framework User Services
- 5. Virtual Organization A Virtual Organization (VO) specifies how a set of users can access a set of resources what is shared who is allowed to share the conditions under which sharing can occur The concept of VO Is not adequate to cover some common scenarios Data needs to be assessed before to make it publically exploitable by the VO members. Restricted set of users have to collaborate to refine processes and implement show cases. Products generated through elaboration of data or simulation have to be validated by expert users.
- 6. Virtual Research Environment Virtual Research Environment (VRE) is a distributed and dynamically created environment where subset of resources can be assigned to a subset of users via interfaces for a limited timeframe at little or no cost for the providers of the infrastructure Integrated with cloud systems ( OpenNebula ) VRE 1 VO gCube is a first example of a VRE management system VRE 2
- 7. Interoperability: Assumptions Very rich applications and data collections are currently maintained by a multitude of authoritative providers Different problems require different execution paradigms: batch, map-reduce, synchronous call, message-queue, … Key distributed computation technologies exist: grid (gLite and Globus), distributed resource management (Condor), clusters (Hadoop), … Several standards are adopted in the same domain
- 8. Interoperability: Landscape Unstructured Data: blob (binary), and textual files Structured Data: tabular, statistical, geospatial, temporal, and textual data Compound Data: data composed by unstructured and structured data entities security
- 9. Interoperability: gCube Vision gCube objectives: hide heterogeneity , i.e. abstract over differences in location, protocol, and model; embrace heterogeneity , i.e. allow for multiple locations, protocols, and models; Technical goals: no bottlenecks : scale no less than the interfaced resources no outages : keep failures partial and temporary autonomicity : system reacts and recovers
- 10. Hiding Heterogeneity Heterogeneous resources are virtually accessible in a common ecosystem of resources despite their locations, technologies, and protocol Different communities have access to different views according to the conditions under which the sharing can occur Each community can define its own VRE for a limited timeframe and at no cost for the providers of the resource Several VRE can coexist without interfering each other even by competing for the same resources
- 11. Embracing Heterogeneity Approaches and solutions to achieve interoperability : Blackboard-based asynchronous communication between components in a system one protocol to R/W and one language to specify messages Wrapper/ Mediator-based translates one interface for a component into a compatible interface Adaptor-based provides a unified interface to a set of other components interfaces and encapsulates how this set of objects interact
- 12. Each resource is represented by a profile (metadata) characterising: the interface the state the list of dependencies the run-time status the policies the configuration the pending tasks to execute A Resource profile is published by the resource owner is discovered by the resource consumers asynchronously through a common resource-independent protocol gCube offers a distributed and scalable Information System ( blackboard ) to store, discover, and access resource profiles Interoperability Approaches: Resource Discovery gCube interoperability framework: the solution
- 13. Interoperability Approaches: Content Interoperability[1/2] gCube Open Content Management Architecture (OCMA) Assumption data stored in different storage back-ends diverse locations, models, access types few common primitives: documents, collections, repositories gCube allows to reach content that lies outside system expose content (reachable from) inside system perform coarse-grained as well as fine-grained retrieval, update, and addressing Runtime scalability autonomic read-only state replication, maximize throughput, minimize response time: discovery-time load balancing (through IS) reduce latencies Software plugin-based architecture to reduce development costs (plugins over Storage systems)
- 14. Interoperability Approaches: Content Interoperability[2/2] T 1 T 2 adapts adapts factory gDoc … gDoc Read gDoc Write Content Manager Service ( OCMA Service) Adapts gCube doc model ( gDoc ) to an unbounded number of back-end types
- 15. Interoperability Approaches : Data Discovery gCube offers Several index types Forward indexing, which supports ultra fast lookups on tabular typed metadata; XML indexing, that supports semistructured lookups on content metadata; Textual field indexing, that supports full text and qualified lookups on textual (mainly) metadata; Metadata full text indexing, that enables full text lookups on metadata; Content full text indexing, that enables full text lookups on text extracted by content; Geospatial/temporal indexing, that enables geospatial proximity and coverage queries to be executed over geospatial/temporal metadata; Feature indexing, that enables high-dimension vector indexing, for feature lookup (currently the feature is inactive);
- 16. gCube offers solutions to: Decouple the business domain and infrastructure specific logic from the core “execution” functionality Invocate a wide range of logic components: SOAP and REST WebServices, Shell Scripts, Executable Binaries, POJOs, … Support most of the execution paradigms: batch, map-reduce, synchronous call Bridges key distributed computation technologies: grid (gLite and Globus), Condor, Hadoop Control and monitor the execution of a processing flow Staging of data among different storage providers Streaming data among computation elements Interoperability Approaches : Process Execution [1/2]
- 17. Interoperability Approaches : Process Execution [2/2] By using adaptors that operate on a specific third party language and translate them into native constructs, allow for the creation of complex workflows that exploit several diverse technologies deployed on different infrastructures
- 18. Interoperability Approaches : Security [1/2] gCube offers solutions : To secure access to gCube resources for interoperable external systems (incoming security) To ensure Interoperability of gCube security mechanisms with standards compliant security systems (reuse) To facilitate secure access to external resources for gCube services (outgoing)
- 19. Authz: XACML for authz request/response protocol and policy definition SAML assertions to transport user/service authN information Argus-based approach (EMI Authz framework) having pluggable design to integrate additional PIPs SAML Profile for XACML 2.0 following the OASIS Authorization Interoperability Profile Specification AuthN: Production level SSL/HTTPS support Key- and Trust-Manager Interoperability Approaches : Security [2/2]
- 20. AquaMaps is an application* tailored to predict global distributions of marine species initially designed for marine mammals and subsequently generalised to marine species, that generates color-coded species range maps using a half-degree latitude and longitude blocks by interfacing several databases and repository providers Species Distribution Maps Generation * A lgorithm by Kashner et al. 2006
- 21. AquaMaps execution is based on the gCube Ecological Niche Modelling Suite which allows the extrapolation of known species occurrences Species Distribution Maps Generation to determine environmental envelopes (species tolerances) to predict future distributions by matching species tolerances against local environmental conditions (e.g. climate change and sea pollution) Very large volume of input and output data : HSPEC native range 56,468,301 - HSPEC suitable range 114,989,360 Very large number of computation : One multispecies map computed on 6,188 half degree cells (over 170k) and 2,540 species requires 125 millions computations (Eli E. Agbayani, FishBase Project/INCOFISH WP1, WorlFish Center)
- 22. Time Series Management Offers a set of tools to manage capture statistics Supports the complete TS lifecycle Supports validation, curation, and analysis Provides support for data reallocation Produces uniform data-set
- 23. Time Series and R statistical software integration The main aims are to: provide a complete, fully working, environment for R language give user methods to automatically extract data from the time series he was working on give user the possibility to perform queries on the time series database provide a service distributed on the infrastructure. Multiple instances can be managed on the infrastructure VREs, the distribution being transparent to the users (SaaS model)
- 24. Conclusions gCube System: Stable software being improved over the last 5 years ( end of DILIGENT -> D4Science -> D4ScienceII) gCube offers a variety of patterns, tools, and solutions to delivery interoperability solutions and interconnect Heterogeneous digital content Heterogeneous repository systems Heterogeneous computation platforms to decrease the cost of adoption to deal with several standards
- 25. Questions Time
Editor's Notes
- #6: Scenarios: TS curation 2? 3) the Aquamaps maps expert validation
- #7: VRE resources can be published in the VO at any time by the VRE data managers.
- #8: Interoperability is among the most critical issues to be faced when building systems as "collections" of independently developed constituents (systems on its own) that should co-operate and rely on each other to accomplish larger tasks. Unfortunately, interoperability is a kind of problem that has multiple facets and it is very challenging. Interoperability issues arising whenever two (or more systems) are willing to ''share'' a certain resource (whatever it is) and one of the two systems plays the role of ''provider'' of the resource while the other plays the role of a ''consumer'' of this resource. The multiple facets result from the fact there are multiple barriers hindering the involved systems to ''share'' a common understanding of the resource that is the target of the interoperability scenario. These barriers range from different models of the resource to different protocols and API to access the resource and interact with it, different policies (and policy models) governing the resource consumption, etc.
- #9: Different Interoperability approaches
- #11: Resources accessble from a common ecosystem of resources and different communities can access different Ecosystem views. Communities define their VREs ( trasparentrly from the providers, which could be also Cloud systems) Competition of the same resources btw VREs ( eg. Indexes or Storage)
- #12: Blacboard bases ( Information System) Wrapper /medoator based ( CM) Adaptor-based ( PES adaptor over condor, grid and hadoop) map reduce)
- #14: OCMA is an open, WSRF-compliant architecture for gCube content management services. OCMA defines a design pattern for such services and, by contextualisation of the pattern, their role in a gCube infrastructure. Requirements and Assumptions OCMA acknowledges that gCube is concerned with content that may: be hosted inside or outside a gCube infrastructure; be described with a variety of models, for different media, and with different degrees of structure; be accessed with a variety of protocols; OCMA makes only the following assumptions about content: content is created, accessed, and distributed in units called documents ; documents are grouped in collections ; collections are hosted in local management systems called repositories . Finally, OCMA acknowledges that content management in gCube needs to: embrace heterogeneity , i.e. support simultaneously multiple locations, protocols, and models; hide heterogeneity , i.e. abstract over differences in location, protocol, and model; scale , i.e. retain good throughput under heavy load;
- #15: OCMA is the Architecture, flagship CM sevice A storage back-end R may already offer a native T-interface. In this case OCMA relies on wrapper for R . A storage back-end R may offer a different T'-interface. In this case OCMA relies on adapter for R .
- #22: Composition of: Data access to External infrastructures ( Aquamaps) Data process on Hadoop Data process on glite
- #23: In statistics , signal processing , econometrics and mathematical finance , a time series is a sequence of data points , measured typically at successive times spaced at uniform time intervals. Time series analysis comprises methods for analyzing time series data in order to extract meaningful statistics and other characteristics of the data.
- #24: R is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R. R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.