![4 S. Lynden et al.
and potential applications to highlight how this work can benefit web service-based
architectures used in Grid and Cloud computing.
1.1 Introduction and Background
Standards play a central role in achieving interoperability within distributed envi-
ronments. By having a set of standardized interfaces to access and integrate
geographically distributed data, possibly managed by different organizations that
use different database systems, the work that has to be undertaken to manage and
integrate this data becomes easier. Thus, providing standards to facilitate the access
and integration of database systems on a large scale distributed scale is important.
The Open Grid Forum (OGF)1
is a community-led standards body formed to
promote the open standards required for applied distributed environments such as
Grids and Clouds. The OGF is composed of a number of Working Groups that con-
centrate on producing documents that standardise particular aspects of distributed
environments as OGF recommendations, which are complemented by informational
documents that inform the community about interesting and useful aspects of
distributed computing, experimental documents are more practically based and are
required for the recommendation process and finally community documents inform
and influence the community on practices anticipated to become common in the dis-
tributed computing community. A process has been established [1] that takes these
documents through to publication at the OGF web site. An important aspect of the
OGF recommendation process is that there must be at least two interoperable imple-
mentations of a proposed standard before it can achieve recommendation status.
The interoperability testing is a mandatory step required to finalise the process and
provide evidence of functional, interoperable implementations of a specification.
The Database Access and Integration Services (DAIS) Working Group was estab-
lished relatively early within the lifetime of the OGF, which was at that time entitled
the Global Grid Forum. The focus on Grids had up, to that point, predominantly been
on the sharing of computational resources. DAIS was established to extend the focus
to take data into account in the first instance to incorporate databases into Grids. The
initial development of the DAIS work was guided by an early requirements capture
for data in Grids in GFD.13 [2] as well as the early vision for Grids described by
the Open Grid Services Architecture (OGSA) [3]. The first versions of the DAIS
specification attempted to use this model only, however, much of the focus of the
Grid community changed to the Web Services Resource Framework (WSRF) [4],
and DAIS attempted to accommodate this new family of standards whilst still being
able to use a non-WSRF solution – a requirement coming from the UK e-Science
community [5] – the impact of which is clearly visible in the DAIS specification
documents. The rest of this chapter describes the specifications in more detail.
1
http://www.ogf.org.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-19-320.jpg)
![1 Open Standards for Service-Based Database Access and Integration 5
1.2 The WS-DAI Family of Specification
1.2.1 Overview
The relationship between the WS-DAI (Web Services Database Access and Integra-
tion Services) family of specifications is schematically illustrated in Fig. 1.1. These
provide a set of message patterns and properties for accessing various types of data.
A core specification, the WS-DAI document [6], defines a generic set of interfaces
and patterns which are extended by specifications dealing with particular data
models: WS-DAIR for relational databases [7], WS-DAIX for XML databases [8]
and WS-DAI-RDF(S) for Resource Description Framework (RDF) databases [9].
In WS-DAI, a database is referred to as a data resource. A data resource
represents any system that can act as a source or sink of data. It has an abstract
name which is represented by a URI and an address which shows the location of a
resource. A data access service provides properties and interfaces for describing and
accessing data resources. The address of a resource is a web service endpoint such
as an EndPointReference (EPR) provided by the WS-Addressing [10] specification.
A WSRF data resource provides compatibility with the WS-Resource (WSRF) [4]
specifications. A consumer refers to the application or client that interacts with the
interfaces provided by a data resource.
An important feature introduced by WS-DAI is the support for indirect access
patterns. Typically, web services have a request-response access pattern – this is
referred to as direct data access – where the consumer will receive the requested
data in the response to a request, typically a query, made to a data access service. For
example, passing an XPathQuery message to an XML data access service will result
in a response message containing a set of XML fragments. An operation that directly
inserts, updates or deletes data through a data access service also constitutes a direct
data access. For example, passing an SQL insert statement to a data access service
WS-DAI
Message patterns
Core Interfaces
WS-DAIX
XML Access
WS-DAI-RDF(S)
RDF Access
WS-DAI-RDF(S)-Ontology
Ontology Access
WS-DAI-RDF(S)-Query
Query Access
WS-DAIR
Relational Access
Fig. 1.1 The WS-DAI family of specifications](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-20-320.jpg)
![8 S. Lynden et al.
In this example, the data access service implements the SQLAccess messages
and exposes the SQLDescription properties; more details about the interface and
corresponding properties can be found in [7]. A consumer uses the SQLExecute
message to submit an SQL expression. The associated response message will
contain the results of the SQL execute request. When the SQL expression used is a
SELECT statement, the SQL response will contain the data in a RowSet message
serialized using an implementation-specific data format, for example the XML
WebRowSet [11].
1.2.2.3 Data Factory
Factory messages create a new relationship between a data resource and a data
access service. In this way, a data resource may be used to represent the results of a
query or act as a place holder where data can be inserted. A data factory describes
properties that dictate how a data access service must behave on receiving factory
messages. The factory pattern may involve the creation of a new data resource and
possibly the deployment of a web service to provide access to it (though existing
web services can be re-used for this purpose – DAIS does not specify how this
should be implemented). The WS-DAI specification only sets the patterns that
should be used for extensions to particular types of data.
This ability to derive one data resource from another, or to provide alternative
views of the same data resources, leads to a collection of notionally related data
resources, as illustrated in Fig. 1.3, which again takes an example from the WS-
DAIR specification. The database data access service in this example presents an
SQLFactory interface. The SQLExecuteFactory operation is used to construct a new
derived data resource from the SQL query contained in it. These results are then
made available through an SQLResponseAccess interface which may be available
through the original service or as part of a new data service. Access to the RowSet
resulting from the SQL expression executed by the underlying data resource is made
available through a suitable interface, assuming that the original expression contains
a SELECT statement.
The RowSet could be stored as a table in a relational database or in a form decou-
pled from the database. DAIS does not specify how this should be implemented but
the implementation does have a bearing on the properties of ChildSensitiveToParent
and ParentSensitiveToChild which indicate whether changes in the child data affect
the parent data or changes in the parent data affect the child data, respectively. The
RowSet results are represented as a collection of rows via a data access service
supporting the SQLResponseAccess collection of operations that allow the RowSet
to be retrieved but does not provide facilities for submitting SQL expressions via
the SQLAccess portType.
The Factory interfaces provide a means of minimising data movement when it is
not required in addition to an indirect form of third party data delivery: consumer A
creates a derived data resource available through some specified data service whose
reference can be passed on to consumer B to access.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-23-320.jpg)
![12 S. Lynden et al.
• XUpdateAccess: Allows an XUpdate expression to be executed against an XML
resource, returning the number of updated nodes.
• XPathAccess: Allows the evaluation of XPath expressions across collections of
XML documents represented by an XML resource.
• XMLSequenceAccess: Provides access to an XML sequence created as a result
of an XPath/XQuery query. The GetItems operation of this interface allows the
client to obtain specific subsequences of the overall result.
The following indirect access interfaces are supported:
• XMLCollectionFactory: Provides access to collections and documents in collec-
tions.
• XPathFactory: Provides the XPathQueryFactory that allows new data resources
(supporting the XMLSequenceAccess interface) to be created as the result of an
XPath query.
• XQueryFactory: Provides the XQueryExecuteFactory operation to create new
XMLSequenceAccess data resources as the result of an XQuery query.
1.5 The RDF Extension (WS-DAI-RDF(S))
The RDF is a World Wide Web Consortium (W3C) set of recommendations [12]
focused on the representation and management of metadata. It includes two data
models, RDF and RDF Schema, whose combination is known as RDF(S). The
WS-DAI-RDF(S) extension to this domain provides data access mechanisms for
RDF(S) data, divided into two types based on the style of access: declarative
or programmatic. Hence, the following specifications are in the process of being
defined within DAIS to access RDF(S) data.
1. WS-DAI RDF(S) Querying: This specification provides a query language inter-
face to RDF data based on the W3C SPARQL query language [13] for RDF.
2. WS-DAI RDF(S) Ontology: This specification provides an API style of access
based on ontology handling primitives conforming to the RDF(S) model. These
primitives provide various operations including the possibility of performing
updates to the ontology.
1.5.1 The WS-DAI RDF(S) Querying Specification
The objective of the querying specification is to provide an SPARQL interface to
RDF data. The W3C has defined several related specifications based on SPARQL,
including an XML-based query results format [14] and a protocol for accessing
RDF resources [15]. The WS-DAI-RDF(S) Querying specification, the interaction
patterns of which are illustrated in Fig. 1.5, is defined to be compatible with the](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-27-320.jpg)
![1 Open Standards for Service-Based Database Access and Integration 17
• OGSA-DAI3
is an open-source distributed data access and management system
supporting Web service-based access to data. OGSA-DAI WS-DAIR, an imple-
mentation of the WS-DAIR interfaces using the OGSA-DAI middleware that can
be obtained from the OGSA-DAI SourceForge site.4
1.6.2 WS-DAIX Implementations
• An implementation of WS-DAIX is also available from the OGSA-DAI source
SourceForge site.
1.6.3 WS-DAI-RDF Implementations
• The EC funded ADMIRE (Advanced Data Mining and Integration for Eu-rope)
[www.admire-project.eu/] project has developed an implementation of the WS-
DAI-RDF Querying Specification.
• AIST’s OGSA-DAI-RDF project has developed an implementation of the WS-
DAI-RDF Querying Specification, which can be obtained from.5
The OGF process requires that two independent interoperable implementations
exist before a proposed recommendation can become a full OGF recommendation.
To date, two of the above implementations (WS-DAIR implementations from the
OGSA-DAI and AMGA projects) have been utilised to validate the WS-DAI and
WS-DAIR specifications as reported in [16]. A comparison of the functionality
of these implementations is made in Table 1.1. The performance of the imple-
mentations is dependent on the underlying DBMS being utilised; however, the
overhead incurred by the WS-DAI(R) Web service-based interfaces is similar for
both implementations.6
1.7 Applications
The set of potential areas of application for the DAIS specifications is wide ranging
and some research projects have already become early adopters of them. This
section provides two examples of the application of the WS-DAI-RDF specification.
3
http://ogsadai.org.uk.
4
http://ogsa-dai.sourceforge.net/.
5
http://dbgrid.org.
6
A brief analysis of the performance of the AMGA WS-DAIR implementation and some compar-
isons with the OGSA-DAI implementation can be found under “Design and Implementation of
WS-DAIR for AMGA” available at http://event.twgrid.org/isgc2009/program.htm.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-32-320.jpg)
![18 S. Lynden et al.
Table 1.1 A comparison of the OGSA-DAI and AMGA implementations of WS-DAI and WS-
DAIR
OGSA-DAI AMGA
Apache Axis gSoap
Infrastructure Java 1.4 CCC
SOAP binding rpc/literal Document/literal
Any JDBC-enabled Supports MySQL, SQLLite
Underlying DBMS relational database PostgreSQL, Oracle
SQL (dependent on SQL-92
Supported languages underlying DBMS) AMGA Metadata Language
Yes
Stored procedures (if supported by DBMS) No
WebRowSet
Datasets comma-separated values WebRowSet
SSL, GSI, VOMS
Security features None permission, ACL
ServiceBusyFault
ServiceBusyFault SQL CommunicationArea
Un-supported features GenericQuery (for fault messages)
One of them is the ADMIRE registry, which uses the WS-DAI-RDF specifications
to provide support in a data mining and integration (DMI) context. The second
presents a scenario in which the specifications can be applied to distributed SPARQL
query processing. Other applications that make use of the other specifications have
already been pointed out above, such as the AMGA and OGSA-DAI projects. These
provide additional examples where the WS-DAI and WS-DAIR specifications have
been implemented for the convenience and benefit of their users.
1.7.1 ADMIRE
The ADMIRE7
registry allows a range of DMI components, called processing
elements (PE), to be registered and discovered, together with the set of types, in
the context of their inputs and outputs, that can be handled by those processing
elements. The descriptions used in the registrations contain the data types of the
input and output parameters for each PE and any restrictions associated with these,
such as: the relationships between the inputs and outputs, termination conditions,
are error conditions and all these information are available at the registry in an RDF
format.
In ADMIRE, users create these PEs and register their descriptions in a registry
by means of a register operation as defined in the DISPEL language [17]. Users can
7
EU FP7 ICT 215024 www.admire-project.eu.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-33-320.jpg)
![1 Open Standards for Service-Based Database Access and Integration 19
then retrieve PE descriptions by using SPARQL. By adding a web service layer, the
registry may be accessed by different users at different times in different contexts
(binding the states to the users). The WS-DAI-RDF(S) specification thus provides a
convenient way of providing standardised access to this RDF-based data repository,
and this is what has been achieved by ADMIRE.
1.7.2 Distributed Query Processing
The WS-DAI-RDF(S) specifications allow data integration applications to be
constructed on top of the consistent interfaces provided by WS-DAI-RDF(S) data
resources. When integrating data from distributed data sources, it is necessary to
deal with syntactic heterogeneities that may be present between the interfaces used
to interact with data resources. Furthermore, data retrieval mechanisms must support
delivery mechanisms that allow clients some form of control over the rate at which
data is delivered, especially when scalability is desired.
This is important for grid-based distributed databases, where data is federated
and accessible via service-based interfaces. The standardised interfaces provided
by the WS-DAI-RDF(S) specifications mean that many heterogeneities present
amongst the individual data sources are resolved when performing these tasks. Data
integration may be performed by multiple computational resources, and the WS-
DAI indirect data access pattern can be used to execute sub-queries which result in
the creation of a new data resource for each set of query results. The various data
integration tasks (e.g., joins, unions) that need to be performed can then delegated
to appropriate nodes in a set of computational resources, which are given references
to the created data resources that need to be accessed to perform their allocated
tasks. This therefore allows parallel and distributed query processing to take place
following the approach used by the OGSA-DQP [18] distributed query processor,
which uses OGSA-DAI data resources. The DAIS specification’s operations allow
similar applications to be developed accessing data resource using open standards.
1.8 Conclusions
This chapter has given an overview of the WS-DAI family of specifications that
have been the focus of the DAIS Working Group of the OGF. A core specification
provides a framework which can then be extended to deal with specific types of data.
This process has already been realised for relational, XML and RDF data, and some
initial proposals have been also made for other types of databases. Generally, the
DAIS approach provides a core specification and a flexible framework that allows
extensions if further requirements are specified of the core specification, which may
in turn impact on the other extension specified.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-34-320.jpg)
![20 S. Lynden et al.
This chapter’s review of the interfaces provided by the specifications has focused,
in particular, on the novel use of indirect data access to provide a means of
minimising data movement, allowing derived data resources to be deployed and
exposed at the server side.
These specifications provide a means of abstracting out some of the variability
in the data resources used in distributed environments and presenting uniform
interfaces to specific types of data – for now: relational, XML and RDF data – to
clients. The use of web services to do this ensures a certain degree of programming
language neutrality and portability across different computer systems. For these
reasons, it is expected that the adoption of these specifications will facilitate the
management and integration of data across the distributed environments presented
by Grids and Clouds.
Acknowledgements The authors thank all those people who have participated in the process of
developing and ratifying the DAIS specification documents and OGF for hosting the process.
References
1. Catlett, C., de Laat, C., Martin, D., Newby, G., Skow, D.: Open Grid Forum Document Process
and Requirements [Obsoletes GFD.1] GFD.152. Open Grid Forum, 2009. http://www.ogf.org/
documents/GFD.152.pdf
2. Atkinson, M.P., Dialani, V., Guy, L., Narang, I., Paton, N.W., Pearson, P., Storey, T., Watson
P.: Grid Database Access and Integration: Requirements and Functionalities. GFD-I-13. Open
Grid Forum, 2003. http://www.ogf.org/documents/GFD.13.pdf
3. Foster, I., Kishimoto, H., Savva, A., Berry, D., Djaoui, A., Grimshaw, A., Horn, B., Maciel, F.,
Subramaniam, R., Treadwell, J., Von Reich, J.: The Open Grid Services Architecture, Version
1.0. OGF GFD-I.030. Open Grid Forum, 2005. http://www.ogf.org/documents/GFD.30.pdf
4. Web Service Resource Framework (WSRF) Specifications. OASIS.
http://www.oasis-open.org/committees/tc home.php?wg abbrev=wsrf. Accessed 9 Oct 2010
5. Atkinson, M., De Roure, D., Dunlop, A., Fox, G., Henderson, P., Hey, T., Paton, N., Newhouse,
S., Parastatidis, S., Trefethen, A., Watson, P., Webber, J.: Web Service Grids: An Evolutionary
Approach. Technical report, UK e-Science Institute. http://www.nesc.ac.uk/technical papers/
UKeS-2004-05.pdf. Accessed 9 Oct 2010
6. Antonioletti, M., Atkinson, M., Laws, S., Malaika, S., Paton, N.W., Pearson, D., Riccardi, G.:
Web Services Data Access and Integration (WS-DAI) Specification Version 1.0. OGF GFD.74.
Open Grid Forum, 2006. http://www.ogf.org/documents/GFD.74.pdf
7. Antonioletti, M., Collins, B., Krause, A., Laws, S., Magowan, J., Malaika, S., Paton,
N.W.: Web Services Data Access and Integration The Relational Realisation (WS-DAIR)
Specification Version 1.0. OGF GFD.76. Open Grid Forum, 2006. http://www.ogf.org/
documents/GFD.76.pdf
8. Antonioletti, M., Hastings, S., Krause, A., Langella, S., Laws, S., Malaika, S., Paton, N.W.:
Web Services Data Access and Integration. The XML Realisation (WS-DAIX) Specification
Version 1.0. OGF GFD.75. Open Grid Forum, 2006. http://www.ogf.org/documents/GFD.75.
pdf
9. Antonioletti, M., Aranda, C.B., Corcho, O., Esteban-Gutirrez, M., Gmez-Prez, A., Kojima,
I., Lynden, S., Pahlevi. S.M.: WS-DAI RDF(S) Realization: Introduction, Motivational Use
Cases and Terminologies GFD-I 163. Open Grid Forum, 2009. http://www.ogf.org/documents/
GFD.163.pdf](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-35-320.jpg)
![26 A. Edmonds et al.
Interoperability. Describes how two services can inter-operate on the fly. This
demands a standardised API and protocol (e.g. live migrating a virtual machine
from one host to another, which are in different management domains).
Integration. Describes how a service provider can bring together different tech-
nologies and interconnect them within his domain (e.g. integrate a virtual machine
management tool with an identity management system).
Portability. This is mostly about the porting between service providers. In com-
parison with interoperability, there is no direct connection between the service
provider. This demands that there are standardised data formats which providers
can understand (e.g. porting a virtual machine from one hypervisor to another).
Innovation. Standards have always been started when a field in the IT community
gains popularity, is widely adopted and begins on a path of commoditisation. Next to
interoperability, standards can be a driver for innovation as well as widely adopted
innovations can demand standards.
Reusability. This can be seen on two levels. First the reuse of (legacy) codes through
basic standardised APIs and the reuse of the standard itself in different fields.
2.2.2 The Core Model
The core meta-model [10] for OCCI imposes a general means of handling general
resources, providing semantics for defining the type of a given entity, describing
interdependencies in between different entities, and defining operating character-
istics on them. Although the meta-model aims to ease the implementation burden
by setting a common ground for other OCCI-related specifications, it can be used
as a standalone component in other contexts (e.g. Resource Oriented Architectures
(ROAs)) as well.
The UML class diagram shown (Fig. 2.2) gives an overview of the OCCI core
meta-model. At its heart lies the Resource type. Any resource exposed through
OCCI is a Resource or sub-type thereof. A resource can be for example a virtual
machine, a job in a job submission system, a user, etc. The Resource type
contains a number of common attributes that domain-specific Resource types
inherit. The Resource type is complemented by the Link type which associates
one Resource instance with another. The Link type also contains a number of
common attributes that domain-specific Link types inherit.
Entity is an abstract type which both Resource and Link inherit. Each sub-
type of Entity is identified by a unique Kind instance. The Kind type comprise
the classification system built into the OCCI model. Kind is a specialisation of
Category and introduces additional capabilities in terms of Action types.
2.2.2.1 Classification and Identification
The OCCI model provides a built-in classification system allowing for safe exten-
sion towards domain-specific usage. This system is like a “type system” but with
the possibility of being easily exposed over a text-based protocol.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-41-320.jpg)
![2 OCCI for Data Management 27
Mixin
Kind Action
Entity
−id :URI
−title :String [0..1]
Resource
−summary:String[0..1]
Link
Category
−scheme :URI
−term :String
−title :String [0..1]
−attributes : SetString
cd: Core
target
+ source
*
links
*
actions
kind
*
*
mixins
*
actions
*
*
related
*
*
related
Fig. 2.2 UML class diagram of the OCCI model. The diagram provides an overview of the OCCI
model but is not a standalone definition thereof
The classification system can be summarised with the following key features:
• Each OCCI base type and extension thereof is assigned a unique identifier, a
structural Kind, which allows for dynamic discovery of available types.
• The relationship of structural Kinds is part of the system and thus the inheritance
model is also discoverable.
• The classification system allows non-structural Kinds to be assigned to resource
instances adding new capabilities using a mix-in-like model.
• Tagging of resource instances is supported through mix-in of non-structural
Kinds which have no additional capabilities defined.
• A collection of associated resources is implicitly defined for each structural and
non-structural Kind. That is all resource instances associated with a particular
Kind instance form a collection.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-42-320.jpg)
![2 OCCI for Data Management 29
Overall, the OCCI core meta-model provides a solid foundation for the remote
management of resources offered in an as-a-Service manner, allowing for the devel-
opment of interoperable tools for common tasks including deployment, automatic
scaling and monitoring. The explicit split-out of it allows the leverage of the
developed models, protocols, and APIs in manners not anticipated and to foster
modularity and extensibility for future usage paradigms.
2.2.3 RESTful HTTP Rendering of the OCCI Model
The OCCI Core model which is described in the previous Sect. 2.2.2 is free of any
rendering and forms the base of OCCI. Based upon this model, OCCI describes a
serialisation rendering. This rendering – or serialisation format – is passed on the
wire between client and service, see [11].
OCCI has a default rendering which is text based and uses the HTTP protocol
and implements a ROA, see [14]. In this architecture, a system is modelled as a
set of related resources. ROA’s use Representation State Transfer (REST), see [6],
to cater for client and service interactions. In these interactions, clients request to
perform operations on the state of an individual or set of resources managed by the
service.
HTTP is commonly used in most ROA systems. It provides means to uniquely
identify resources through URIs as well as operating upon them with a set of
general-purpose operations called verbs. These HTTP verbs map loosely to the
resource-related operations of create (POST), retrieve (GET), update (POST, PUT)
and delete (DELETE).
2.2.3.1 Rendering of Resources
Each Resource in the OCCI core model will be rendered as a unique URI (for
example http://example.com/foo). Each resource can be identified uniquely by an
URI and has at least one Category assigned, which defines the type and the
operations that can be performed. This means that from this standpoint a resource
can be almost anything like a Database entry, a Virtual Machine, an Image, etc.
Resources can be linked and actions can be performed upon them. Resource of
the same type (as in have the same Category assigned) can be found under a
certain path relative to the root of the service provider (e.g. all storage devices will
appear under the path /storage – still the path name “storage” is freely defined by
the Service Provider and can do discovered through the Query interface).
Since Categories cannot only be used to define the type of the resource, but
also to tag or group resources, resource can show up under multiple paths. The
following URL hierarchy demonstrates this feature:](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-44-320.jpg)
![2 OCCI for Data Management 31
Body. This is defined as the basic text/plain content type. Other content types also
exist. For example, the information can also be rendered in the HTTP Header or as
HTML (e.g. for browsing the OCCI interface using a Web Browser) by supplying
the appropriate content-type header as specified in the specification.
Rendering of data is done through simple key value associations. Also, more
structural data representations such as JSON of RDFa can easily be added to OCCI.
2.2.4 OCCI for Virtual Machine (Infrastructure) Provisioning
Having described the core model and a way of rendering it on the wire, a concrete
compliment to the core model is now explored [12].
The infrastructure specification extends the core model at two key points:
1. To represent various infrastructure-related resources, it extends Resource
using inheritance.
2. To represent concrete relationships between infrastructural resources, it extends
Link using inheritance.
To represent the main elemental resources found in infrastructure-type services,
OCCI has three specialisations of Resource:
1. Compute: Information processing resources.
2. Network: Interconnection resources.
3. Storage: Information recording resources.
Complimenting these, to allow linkage are:
• NetworkInterface: Represents an L2 client device (e.g. network adapter).
• StorageLink: Represents a link from a Resource to a target Storage Resource.
The relations of these infrastructure resources are shown in the UML diagram
(Fig. 2.3).
When modeling elements, it was found that OCCI needed to support not only
generic cases but also specific cases. This issue was exemplified by Network. It
might be immediately attractive to model all functionalities within this Resource,
including aspects of IP configuration, however, then the model would force certain
technology choices upon implementers. To avoid this, the working group chose to
utilise the OCCI mix-in capabilities to avoid such a situation. Where an implementer
wishes to offer TCP/IP functionality on top of the Network resource, they
can do so by implementing the IPNetworking mix-in. The IPNetworking
mix-in allows to supplement the Network Resource with the necessary TCP/IP
features. Should an implementer wish to offer another type of L3/L4 technology
for example AppleTalk or IPX, then they only need implement a custom network
mix-in.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-46-320.jpg)
![32 A. Edmonds et al.
IPNetworking IPNetworkInterface
Link
(from occi :: core)
Resource
(from occi :: core)
-summary :String[0..1]
StorageLink
Storage NetworkInterface
Network
Compute
cd: Infrastructure
target + source
*
links
Fig. 2.3 Extended OCCI core model showing infrastructure elements
It was the infrastructure model, along with the OCCI core and HTTP rendering
specifications, that aided a successful collaboration that investigated the integration
of two large European Union FP7 research projects, SLA@SOI3
and RESERVOIR.4
The proposed integration was detailed in a subsequent technical paper [13].
2.2.5 Related Standards and Specifications
A guiding principle in OCCI is to make use of existing standards and specifications
where appropriate.
OCCI and the Storage Networking Industry Association’s (SNIA)5
Cloud Data
Management Interface (CDMI) working groups have collaborated together so that
both specifications are interoperable with each other. It states that
“The SNIA Cloud Data Management Interface (CDMI) is the functional interface that
applications will use to create, retrieve, update and delete data elements from the cloud. As
part of this interface the client will be able to discover the capabilities of the cloud storage
offering and use this interface to manage containers and the data that is placed in them. In
addition, meta-data can be set on containers and their contained data elements through this
interface” [16].
3
http://www.sla-at-soi.eu/.
4
http://www.reservoir-fp7.eu/.
5
http://www.snia.org/.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-47-320.jpg)
![2 OCCI for Data Management 33
OCCI and the Distributed Management Task Force’s (DMTF)6
Open Virtual-
ization Format (OVF), see [4], can be easily integrated through the use of the
resource-type Link. Where a provider wishes to supply an OVF representation of
a client’s resource instance(s), they can do so by associating the instance(s) with a
mirror representation, only the serialisation format is OVF.
Other than this, the OCCI working group is closely working together with other
groups inside of the Open Grid Forum. The Distributed Computing Infrastucture
Federation (DCI-fed7
) working group focuses on the creation of models and APIs
for setting up distributed federated computing environments. Other than this, the
OCCI working group uses Standards like those developed by the Distributed
Resource Management Application API (DRMAA8
) working group for common
Job operations on Clusters via the OCCI protocol.
2.3 SLA Assured Provisioning of Database Services
Using OCCI
In today’s service marketplaces including cloud-based ones, there exist basic
limitations in service offerings. Typically, the customer has little say in what is
offered by a service provider and is left with a “take it or leave it” situation. Not
only is the customer faced with such a dilemma, with little possibility of negotiation
but if they do accept the service offering there is little in the way of service
transparency and so detections of service violations are impossible unless that
customer implements custom violation detection systems. The SLA@SOI project
seeks to address these challenges by providing three major benefits:
Predictability and Dependability: The quality characteristics of service can be
predicted and enforced at run-time.
Transparent SLA Management: SLAs defining the exact conditions under which
services are provided/consumed can be transparently managed across the whole
business and IT stack.
Automation: The whole process of negotiating SLAs and provisioning, delivery and
monitoring of services will be automated allowing for highly dynamic and scalable
service consumption.
In this section, a use-case that combines the OCCI model and API with an SLA
management framework to provide an SLA assured database service is described.
In today’s service marketplace, there exists a number of service providers who
offer database services, for example, the Amazon Relational Database Service,9
6
http://www.dmtf.org.
7
http://forge.gridforum.org/sf/projects/dcifed-wg.
8
http://www.drmaa.org.
9
http://aws.amazon.com/rds.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-48-320.jpg)
![34 A. Edmonds et al.
Microsoft SQL Azure10
and Longjump Platform as a Service.11
Other than offering
a basic, non-negotiable, non-machine readable SLA, these service providers do not
offer certain guarantees that particular consumers will require. A case in point is
where a third party service provider wishes to process personal and identifying
information. Many law jurisdictions will require that user-supplied data and the
processed resultant data remain within the protection of that jurisdiction, which
may mean that the physical location of that data must always remain in the country
or region where that jurisdiction has powers to protect. If that data at any one
time falls outside of those defined physical locations due to actions taken by the
service provider that the third party uses, then regardless of knowing or not knowing
about such actions, the third party can be liable under the relevant laws set out by
the jurisdiction. In the use case presented here, an SLA management framework
provides the means to:
1. Customise a service offering.
2. Negotiate on that service offering to the satisfaction of the third party and their
legal responsibilities.
3. Be notified when terms of the agreed service offering deviate and have deviations
logged as an audit trail.
The use case is realised by the third party provisioning the offered database ser-
vice using the OCCI API through the facilities of the SLA manager. OCCI provides
the standard and interoperable means of provisioning the required database service
and the SLA manager provides the means as outlined above. That database service is
realised as a pre-built virtual machine with all the requisite database software
installed and configured, which once provisioned is accessible by the consumer.
The service provider offers means to monitor the agreed terms in the SLA and, in
particular for this use case, allows for the physical location of the virtual machine to
be monitored. This allows the SLA management framework to monitor constantly
the physical location of the virtual machine and in the case that the virtual machine
is migrated to an inappropriate physical location the third party will instantly receive
notification of that event and logs will provide an audit trail.
2.3.1 SLA@SOI SLA Management Framework
SLA@SOI defines a holistic view for the management of SLAs and implements an
SLA management framework that can be easily integrated into a service-oriented
infrastructure (SOI), see [17]. The main innovative features of the project are:
• An automated e-contracting framework
• Systematic grounding of SLAs from the business level down to the infrastructure
10
http://www.microsoft.com/en-us/sqlazure.
11
http://www.longjump.com.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-49-320.jpg)
![36 A. Edmonds et al.
2.3.2 SLA@SOI and OCCI
In this use case, there are two main components that are required for realisation. The
first and most fundamental is a Service Manager that offers a database service.
The Service Manager is the entity that is responsible for providing the client’s
service. Relevant to this use case is that the Service Manager provides database
services as preconfigured virtual machines and that the location of those virtual
machines can be monitored. The SLA@SOI framework makes no assumption on
this Service Manager only that it has an interface:
1. That can create, retrieve, update and delete its managed services.
2. Through which service instance metrics can be listed and retrieved.
The second entity required is the SLA@SOI framework’s SLA Manager. This
is a set of both generic and domain-specific components. What is generic relates
to the management of SLA templates (what a provider offers) and SLA instances
(what a provider runs on their clients behalf and guarantees). The domain-specific
components are those that interact with the particular service manager that provides
the client services. Further details of the SLA@SOI framework and its architecture
can be found, see [18].
The SLA Manager offers to clients one or more SLA Templates, which is
expressed using the SLA@SOI SLA model. Through either a UI or API, the client
can select, customise, negotiate and provision an SLA-guaranteed service. In the
use case scenario, this would entail the third party specifying what physical location
(e.g. region, country) is required for their regulatory compliance.
Once the SLA Manager is acting on the client’s behalf, it first negotiates with the
service manager using the OCCI query interface. The OCCI query interface allows
for the various Resource types to be queried for and interrogated and in particular to
this use case, the locations that a provider can provision their virtual machines. As
an extension to the query interface, SLA@SOI will also allow for per-user quotas to
be queried. Using this extension, an SLA Manager can tell whether a client’s request
will be fully satisfied or not by the current service provider. Having established that
the client’s quota is sufficient, the next step can either take two paths. The first is
that the provisioning of the requested service is done automatically or, second, the
provisioning must be explicitly executed by the client. Where a provisioning request
is executed in one or the other manner, the next responsibility of the SLA Manager
will be to call the provisioning functionality of the Service Manager (relationship
and interaction is shown in Fig. 2.5). This again is looked after by the OCCI API
and an OCCI request from the SLA manager’s domain specific components is sent
to the Service Manager. As soon as the provisioning request is successful, the SLA
manager then begins to monitor the provisioned service, including the location of
the service’s virtual machine. It does this by monitoring the various terms of the
agreed SLA (e.g. QoS metrics).
For the SLA Manager, the major advantage of choosing to implement OCCI
as a means to talk with Service Managers is that in the case where a particular](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-51-320.jpg)
![38 A. Edmonds et al.
The D-Grid Scheduler Interoperability project (DGSI) in the context of the
German D-Grid Initiative13
aims to provide a solution both issues through the devel-
opment of a standards-based protocol between Meta schedulers. DGSI approaches
interoperability DCIs from two sides, namely Activity Delegation (taking care of
the handover of workload from one domain to the other), and Resource Delegation
(taking care of the leasing of resources from one domain to another). Assuming the
DGSI protocols and services in place, the notion of delegation can help to avoid
traditional data management techniques such as decoupled copying, prefetching,
and replication at all.
2.4.1 The Climate Community Use Case
The effects of climate change are one of the major challenges of mankind:
stakeholders of many areas strive for strategies to deal with the consequences of
pollution and man-made changes to the environment. The basis of all decision
making are models of climate processes and the understanding of interplay of the
enormous amount of parameters in them. Since the beginning of industrialisation in
the nineteenth century, Earth System Science, one of the data sciences, investigates
these processes, their chemical formation in the diverse subsystems such as oceans,
atmosphere or biosphere, and their long-term influence on climate.
From those investigations, researchers nowadays possess very detailed insight
into climate development. This rests on the permanent acquisition, cataloging, and
processing of very large (Peta scale) volumes of experimental and model data, as
well as the continuous re-evaluation of scientific results using refined models.
Current information technology provides potent means to accelerate these
processes of data evaluation and simulation. High Performance Computing (HPC)
infrastructure, high speed networks, and modern storage architectures support
archival, preprocessing, selection, and transportation of large data amounts as well
as the computation of highly demanding simulations (e.g. short term weather
forecast or storm track analysis).
For the latter scientific analysis, researchers filter and examine geopotential
heights to track and predict the movement of low-pressure areas over time with
regard to a given climate model. This is essential as storms and cyclones typically
cross such areas [3]. This analysis and simulation is based on long-term acquired
global climate data.
Usually, scientists are only interested in a restricted area for a Stormtrack analysis
and have to reduce the amount of available base data to the region of interest.
Besides a complex combination of several steps, this resorts to either access to a
specific amount of climate data (Fig. 2.6a) or execute simple visualisation workflows
on selected and preprocessed data (Fig. 2.6b).
13
http://www.d-grid.de.](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-53-320.jpg)
![2 OCCI for Data Management 39
Staging
a
b
c
Staging Visualization
Staging Preproc.
Staging Preproc.
Simulation
Fig. 2.6 Three simple workflow examples from C3Grid: (a) selective data download; (b) simple
simulation on input data; (c) simulation on a set of preprocessed input data
Today, they generally have two possibilities to retrieve the desired data: they
either get access to the storage and download the full amount of data (i.e. full
replication is performed) or use proprietary programs to reduce the amount of data
at the storage site and download the desired data set afterwards. In the first case,
the required local storage may simply not be available to the single scientist, or the
providing institution may not be willing to provide an external party with access to
the full archive due to strategical considerations. In the second case, the user needs
to cooperate directly with the data provider, basically via two mechanisms:
1. Having to use tools that are installed, but potentially not known to him,14
or
2. Having to roll out the software on her own, either doing this as part of the batch
processing job or in cooperation with the resource provider.
Obviously, the former is not acceptable from a user’s perspective. The latter
in turn requires extensive manual intervention and additionally necessitates the
acquisition of user rights to retrieve or even locally process the requested data. As
most of the climate data is stored in a distributed way, the procedure often has to
be repeated for several data sites. Furthermore, it leaves intelligent, automatic load
balancing totally to the user, which is generally not desirable. In addition to that, this
traditional approach of application deployment massively hinders cross-community
collaboration, if they rely on different infrastructure technologies: if the user takes
care by himself to deploy the application as part of his computational workload, the
number of resources compatible will usually be very restricted.
14
With the exception of widely accepted and distributed tools such as the Climate Data Operators
[15].](https://image.slidesharecdn.com/1225355-250609164849-cfcd920a/85/Grid-And-Cloud-Database-Management-1st-Edition-Steven-Lynden-54-320.jpg)
Grid And Cloud Database Management 1st Edition Steven Lynden
- 1. Grid And Cloud Database Management 1st Edition Steven Lynden download https://ebookbell.com/product/grid-and-cloud-database- management-1st-edition-steven-lynden-2450710 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Grid And Cloud Database Management 1st Edition Steven Lynden https://ebookbell.com/product/grid-and-cloud-database-management-1st- edition-steven-lynden-4101974 Grid And Cloud Computing And Applications 1st Edition Hamid R Arabnia George A Gravvanis Ashu M G Solo Fernando G Tinetti https://ebookbell.com/product/grid-and-cloud-computing-and- applications-1st-edition-hamid-r-arabnia-george-a-gravvanis-ashu-m-g- solo-fernando-g-tinetti-51607128 Grid And Cloud Computing A Business Perspective On Technology And Applications 1st Edition Katarina Stanoevskaslabeva https://ebookbell.com/product/grid-and-cloud-computing-a-business- perspective-on-technology-and-applications-1st-edition-katarina- stanoevskaslabeva-2618032 Novel Practices And Trends In Grid And Cloud Computing Pethuru Raj Reliance Jio Infocomm Ltd Rjil https://ebookbell.com/product/novel-practices-and-trends-in-grid-and- cloud-computing-pethuru-raj-reliance-jio-infocomm-ltd-rjil-22003990
- 3. Data Management In Cloud Grid And P2p Systems 5th International Conference Globe 2012 Vienna Austria September 56 2012 Proceedings 1st Edition Ibrahima Gueye https://ebookbell.com/product/data-management-in-cloud-grid- and-p2p-systems-5th-international-conference-globe-2012-vienna- austria-september-56-2012-proceedings-1st-edition-ibrahima- gueye-4141354 Data Management In Cloud Grid And P2p Systems 6th International Conference Globe 2013 Prague Czech Republic August 2829 2013 Proceedings 1st Edition Miguel Lirozgistau https://ebookbell.com/product/data-management-in-cloud-grid- and-p2p-systems-6th-international-conference-globe-2013-prague-czech- republic-august-2829-2013-proceedings-1st-edition-miguel- lirozgistau-4333154 Data Management In Cloud Grid And P2p Systems 7th International Conference Globe 2014 Munich Germany September 23 2014 Proceedings 1st Edition Abdelkader Hameurlain https://ebookbell.com/product/data-management-in-cloud-grid- and-p2p-systems-7th-international-conference-globe-2014-munich- germany-september-23-2014-proceedings-1st-edition-abdelkader- hameurlain-4932418 Cloud Computing Smart Grid And Innovative Frontiers In Telecommunications 9th Eai International Conference Cloudcomp 2019 And 4th Eai International Conference Smartgift 2019 Beijing China December 45 2019 And December 2122 2019 1st Ed Xuyun Zhang https://ebookbell.com/product/cloud-computing-smart-grid-and- innovative-frontiers-in-telecommunications-9th-eai-international- conference-cloudcomp-2019-and-4th-eai-international-conference- smartgift-2019-beijing-china-december-45-2019-and- december-2122-2019-1st-ed-xuyun-zhang-22474260 Cloud Grid And High Performance Computing Emerging Applications 1st Edition Emmanuel Udoh https://ebookbell.com/product/cloud-grid-and-high-performance- computing-emerging-applications-1st-edition-emmanuel-udoh-2478328
- 6. Grid and Cloud Database Management
- 7. •
- 8. Sandro Fiore Giovanni Aloisio Editors Grid and Cloud Database Management 123
- 9. Editors Sandro Fiore, Ph.D. Faculty of Engineering Department of Innovation Engineering University of Salento Via per Monteroni 73100 Lecce, Italy and Euro Mediterranean Center for Climate Change (CMCC) Via Augusto Imperatore 16 73100 Lecce, Italy sandro.fiore@unisalento.it Prof. Giovanni Aloisio Faculty of Engineering Department of Innovation Engineering University of Salento Via per Monteroni 73100 Lecce, Italy and Euro Mediterranean Center for Climate Change (CMCC) Via Augusto Imperatore 16 73100 Lecce, Italy giovanni.aloisio@unisalento.it ISBN 978-3-642-20044-1 e-ISBN 978-3-642-20045-8 DOI 10.1007/978-3-642-20045-8 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011929352 ACM Computing Classification (1998): C.2, H.2, H.3, J.2, J.3 c Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: deblik, Berlin Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
- 10. Preface Since the 1960s, database systems have been playing a relevant role in the information technology field. By the mid-1960s, several systems were also available for commercial purposes. Hierarchical and network database systems provided two different perspectives and data models to organize data collections. In 1970, E. Codd wrote a paper called A Relational Model of Data for Large Shared Data Banks, proposing a model relying on relational table structures. Relational databases became appealing for industries in the 1980s, and their wide adoption fostered new research and development activities toward advanced data models like object oriented or the extended relational. The online transaction processing (OLTP) support provided by the relational database systems was fundamental to make this data model successful. Even though the traditional operational systems were the best solution to manage transactions, new needs related to data analysis and decision support tasks led in the late 1980s to a new architectural model called data warehouse. It includes extraction transformation and loading (ETL) primitives and online analytical processing (OLAP) support to analyze data. From OLTP to OLAP, from transaction to analysis, from data to information, from the entity-relationship data model to a star/snowflake one, and from a customer-oriented perspective to a market-oriented one, data warehouses emerged as data repository architecture to perform data analysis and mining tasks. Relational, object-oriented, transactional, spatiotemporal, and multimedia data warehouses are some examples of database sources. Yet, the World Wide Web can be considered another fundamental and distributed data source (in the Web2.0 era it stores crucial information – from a market perspective – about user preferences, navigation, and access patterns). Accessing and processing large amount of data distributed across several coun- tries require a huge amount of computational power, storage, middleware services, specifications, and standards. Since the 1990s, thanks to Ian Foster and Carl Kesselman, grid computing has emerged as a revolutionary paradigm to access and manage distributed, heteroge- neous, and geographically spread resources, promising computer power as easy to access as an electric power grid. The term “resources” also includes the database, v
- 11. vi Preface yet successful attempts of grid database management research efforts started only after 2000. Later on, around 2007, a new paradigm named Cloud Computing brought the promise of providing easy and inexpensive access to remote hardware and storage resources. Exploiting pay per use models, virtualization for resource provisioning, cloud computing has been rapidly accepted and used by researchers, scientists, and industries. Grid and cloud computing are exciting paradigms and how they deal with database management is the key topic of this book. By exploring current and future developments in this area, the book tries to provide a thorough understanding of the principles and techniques involved in these fields. The idea of writing this book dates back to a tutorial on Grid Database Management that was organized at the 4th International Conference on Grid and Pervasive Computing (GPC 2009) held in Geneva (4–8 May 2009). Following up an initial idea from Ralf Gerstner (Springer Senior Editor Computer Science), we decided to act as editors of the book. We invited internationally recognized experts asking them to contribute on challenging topics related to grid and cloud database management. After two review steps, 16 chapters have been accepted for publication. Ultimately, the book provides the reader with a collection of chapters dealing with Open standards and specifications (Sect. 1), Research efforts on grid database management (Sect. 2), Cloud data management (Sect. 3), and some Scientific case studies (Sect. 4). The presented topics are well balanced, complementary, and range from well-known research projects and real case studies to standards and specifications as well as to nonfunctional aspects such as security, performance, and scalability, showing up how they can be effectively addressed in grid- and cloud- based environments. Section 1 discusses the open standards and specifications related to grid and cloud data management. In particular, Chap. 1 presents an overview of the WS-DAI family of specifications, the motivation for defining them, and their relationships with other OGF and non-OGF standards. Conversely, Chap. 2 outlines the OCCI specifications and demonstrates (by presenting three interesting use cases) how they can be used in data management-related setups. Section 2 presents three relevant research efforts on grid-database management systems. Chapter 3 provides a complete overview on the Grid Relational Catalog (GRelC) Project, a grid database research effort started in 2001. The project’s main features, its interoperability with gLite-based production grids, and a relevant show- case in the environmental domain are also presented. Chapter 4 provides a complete overview about the OGSA-DAI framework, the main components for the distributed data management via workflows, the distributed query processing, and the most relevant security and performance aspects. Chapter 5 gives a detailed overview of the architecture and implementation of DASCOSA-DB. A complete description of novel features, developed to support typical data-intensive applications running on a grid system, is also presented.
- 12. Preface vii Section 3 provides a wide overview on several cloud data management topics. Some of them (from Chaps. 6 to 8) specifically focus only on database aspects, whereas the remaining ones (from Chaps. 9 to 12) are wider in scope and address more general cloud data management issues. In this second case, the way these concepts apply to the database world is clarified through some practical examples or comments provided by the authors. In particular, Chap. 6 proposes a new security technique to measure the trustiness of the cloud resources. Through the use of the metadata of resources and access policies, the technique builds the privilege chains and binds authorization policies to compute the trustiness of cloud database management. Chapter 7 presents a method to manage the data with dirty data and obtain the query results with quality assurance in the dirty data. A dirty database storage structure for cloud databases is presented along with a multilevel index structure for query processing on dirty data. Chapter 8 examines column-oriented databases in virtual environments and provides evidence that they can benefit from virtualization in cloud and grid computing scenarios. Chapter 9 introduces a Windows Azure case study demonstrating the advantages of cloud computing and how the generic resources offered by cloud providers can be integrated to produce a large dynamic data store. Chapter 10 presents CloudMiner, which offers a cloud of data services running on a cloud service provider infrastructure. An example related to database management exploiting OGSA-DAI is also discussed. Chapter 11 defines the requirements of e-Science provenance systems and presents a novel solution (addressing these requirements) named the Vienna e-Science Provenance System (VePS). Chapter 12 examines the state of the art of workload management for data-intensive computing in clouds. A taxonomy is presented for workload management of data-intensive computing in the cloud and the use of the taxonomy to classify and evaluate current workload management mechanisms. Section 4 presents a set of scientific use cases connected with Genomic, Health, Disaster monitoring, and Earth Science. In particular, Chap. 13 explores the implementation of an algorithm, often used to analyze microarray data, on top of an intelligent runtime that abstracts away the hard parts of file tracking and scheduling in a distributed system. This novel formulation is compared with a traditional method of expressing data parallel computations in a distributed environment using explicit message passing. Chapter 14 describes the use of Grid technologies for satellite data processing and management within the international disaster monitoring projects carried out by the Space Research Institute NASU- NSAU, Ukraine (SRI NASU-NSAU). Chapter 15 presents the CDM ActiveStorage infrastructure, a scalable and inexpensive transparent data cube for interactive analysis and high-resolution mapping of environmental and remote sensing data. Finally, Chap. 16 presents a mechanism for distributed storage of multidimensional EEG time series obtained from epilepsy patients on a cloud computing infrastructure (Hadoop cluster) using a column-oriented database (HBase). The bibliography of the book covers the essential reference material. The aim is to convey any useful information to the interested readers, including researchers actively involved in the research field, students (both undergraduate and graduate), system designers, and programmers.
- 13. viii Preface The book may serve as both an introduction and a technical reference for grid and cloud database management topics. Our desire and hope is that it will prove useful while exploring the main subject, as well as the research and industries efforts involved, and that it will contribute to new advances in this scientific field. Lecce Sandro Fiore February 2010 Giovanni Aloisio
- 14. Contents Part I Open Standards and Specifications 1 Open Standards for Service-Based Database Access and Integration ..................................................... 3 Steven Lynden, Oscar Corcho, Isao Kojima, Mario Antonioletti, and Carlos Buil-Aranda 2 Open Cloud Computing Interface in Data Management-Related Setups .............................................. 23 Andrew Edmonds, Thijs Metsch, and Alexander Papaspyrou Part II Research Efforts on Grid Database Management 3 The GRelC Project: From 2001 to 2011, 10 Years Working on Grid-DBMSs............................................................. 51 Sandro Fiore, Alessandro Negro, and Giovanni Aloisio 4 Distributed Data Management with OGSA–DAI ....................... 63 Michael J. Jackson, Mario Antonioletti, Bartosz Dobrzelecki, and Neil Chue Hong 5 The DASCOSA-DB Grid Database System.............................. 87 Jon Olav Hauglid, Norvald H. Ryeng, and Kjetil Nørvåg Part III Cloud Data Management 6 Access Control and Trustiness for Resource Management in Cloud Databases ......................................................... 109 Jong P. Yoon 7 Dirty Data Management in Cloud Database............................. 133 Hongzhi Wang, Jianzhong Li, Jinbao Wang, and Hong Gao ix
- 15. x Contents 8 Virtualization and Column-Oriented Database Systems............... 151 Ilia Petrov, Vyacheslav Polonskyy, and Alejandro Buchmann 9 Scientific Computation and Data Management Using Microsoft Windows Azure ................................................. 169 Steven Johnston, Simon Cox, and Kenji Takeda 10 The CloudMiner ............................................................ 193 Andrzej Goscinski, Ivan Janciak, Yuzhang Han, and Peter Brezany 11 Provenance Support for Data-Intensive Scientific Workflows ......... 215 Fakhri Alam Khan and Peter Brezany 12 Managing Data-Intensive Workloads in a Cloud ....................... 235 R. Mian, P. Martin, A. Brown, and M. Zhang Part IV Scientific Case Studies 13 Managing and Analysing Genomic Data Using HPC and Clouds .... 261 Bartosz Dobrzelecki, Amrey Krause, Michal Piotrowski, and Neil Chue Hong 14 Grid Technologies for Satellite Data Processing and Management Within International Disaster Monitoring Projects ........................................................ 279 Nataliia Kussul, Andrii Shelestov, and Sergii Skakun 15 Transparent Data Cube for Spatiotemporal Data Mining and Visualization ........................................................... 307 Mikhail Zhizhin, Dmitry Medvedev, Dmitry Mishin, Alexei Poyda, and Alexander Novikov 16 Distributed Storage of Large-Scale Multidimensional Electroencephalogram Data Using Hadoop and HBase................ 331 Haimonti Dutta, Alex Kamil, Manoj Pooleery, Simha Sethumadhavan, and John Demme Index .............................................................................. 349
- 16. Part I Open Standards and Specifications
- 17. •
- 18. Chapter 1 Open Standards for Service-Based Database Access and Integration Steven Lynden, Oscar Corcho, Isao Kojima, Mario Antonioletti, and Carlos Buil-Aranda Abstract The Database Access and Integration Services (DAIS) Working Group, working within the Open Grid Forum (OGF), has developed a set of data access and integration standards for distributed environments. These standards provide a set of uniform web service-based interfaces for data access. A core specification, WS-DAI, exposes and, in part, manages data resources exposed by DAIS-based services. The WS-DAI document defines a core set of access patterns, messages and properties that form a collection of generic high-level data access interfaces. WS-DAI is then extended by other specifications that specialize access for specific types of data. For example, WS-DAIR extends the WS-DAI specification with interfaces targeting relational data. Similar extensions exist for RDF and XML data. This chapter presents an overview of the specifications, the motivation for defining them and their relationships with other OGF and non-OGF standards. Current implementations of the specifications are described in addition to some existing S. Lynden AIST, Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8568, Japan e-mail: steven.lynden@aist.go.jp O. Corcho C. Buil-Aranda Ontology Engineering Group, Departamento de Inteligencia Artificial, Facultad de Informática, Universidad Politécnica de Madrid, Boadilla del Monte, 28660, Madrid, Spain e-mail: ocorcho@fi.upm.es; cbuil@fi.upm.es I. Kojima () Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8568, Japan e-mail: kojima@ni.aist.go.jp M. Antonioletti EPCC, The University of Edinburgh, JCMB, The Kings Buildings, Mayfield Road, Edinburgh EH9 3JZ, UK e-mail: Mario.Antonioletti@ed.ac.uk S. Fiore and G. Aloisio (eds.), Grid and Cloud Database Management, DOI 10.1007/978-3-642-20045-8 1, © Springer-Verlag Berlin Heidelberg 2011 3
- 19. 4 S. Lynden et al. and potential applications to highlight how this work can benefit web service-based architectures used in Grid and Cloud computing. 1.1 Introduction and Background Standards play a central role in achieving interoperability within distributed envi- ronments. By having a set of standardized interfaces to access and integrate geographically distributed data, possibly managed by different organizations that use different database systems, the work that has to be undertaken to manage and integrate this data becomes easier. Thus, providing standards to facilitate the access and integration of database systems on a large scale distributed scale is important. The Open Grid Forum (OGF)1 is a community-led standards body formed to promote the open standards required for applied distributed environments such as Grids and Clouds. The OGF is composed of a number of Working Groups that con- centrate on producing documents that standardise particular aspects of distributed environments as OGF recommendations, which are complemented by informational documents that inform the community about interesting and useful aspects of distributed computing, experimental documents are more practically based and are required for the recommendation process and finally community documents inform and influence the community on practices anticipated to become common in the dis- tributed computing community. A process has been established [1] that takes these documents through to publication at the OGF web site. An important aspect of the OGF recommendation process is that there must be at least two interoperable imple- mentations of a proposed standard before it can achieve recommendation status. The interoperability testing is a mandatory step required to finalise the process and provide evidence of functional, interoperable implementations of a specification. The Database Access and Integration Services (DAIS) Working Group was estab- lished relatively early within the lifetime of the OGF, which was at that time entitled the Global Grid Forum. The focus on Grids had up, to that point, predominantly been on the sharing of computational resources. DAIS was established to extend the focus to take data into account in the first instance to incorporate databases into Grids. The initial development of the DAIS work was guided by an early requirements capture for data in Grids in GFD.13 [2] as well as the early vision for Grids described by the Open Grid Services Architecture (OGSA) [3]. The first versions of the DAIS specification attempted to use this model only, however, much of the focus of the Grid community changed to the Web Services Resource Framework (WSRF) [4], and DAIS attempted to accommodate this new family of standards whilst still being able to use a non-WSRF solution – a requirement coming from the UK e-Science community [5] – the impact of which is clearly visible in the DAIS specification documents. The rest of this chapter describes the specifications in more detail. 1 http://www.ogf.org.
- 20. 1 Open Standards for Service-Based Database Access and Integration 5 1.2 The WS-DAI Family of Specification 1.2.1 Overview The relationship between the WS-DAI (Web Services Database Access and Integra- tion Services) family of specifications is schematically illustrated in Fig. 1.1. These provide a set of message patterns and properties for accessing various types of data. A core specification, the WS-DAI document [6], defines a generic set of interfaces and patterns which are extended by specifications dealing with particular data models: WS-DAIR for relational databases [7], WS-DAIX for XML databases [8] and WS-DAI-RDF(S) for Resource Description Framework (RDF) databases [9]. In WS-DAI, a database is referred to as a data resource. A data resource represents any system that can act as a source or sink of data. It has an abstract name which is represented by a URI and an address which shows the location of a resource. A data access service provides properties and interfaces for describing and accessing data resources. The address of a resource is a web service endpoint such as an EndPointReference (EPR) provided by the WS-Addressing [10] specification. A WSRF data resource provides compatibility with the WS-Resource (WSRF) [4] specifications. A consumer refers to the application or client that interacts with the interfaces provided by a data resource. An important feature introduced by WS-DAI is the support for indirect access patterns. Typically, web services have a request-response access pattern – this is referred to as direct data access – where the consumer will receive the requested data in the response to a request, typically a query, made to a data access service. For example, passing an XPathQuery message to an XML data access service will result in a response message containing a set of XML fragments. An operation that directly inserts, updates or deletes data through a data access service also constitutes a direct data access. For example, passing an SQL insert statement to a data access service WS-DAI Message patterns Core Interfaces WS-DAIX XML Access WS-DAI-RDF(S) RDF Access WS-DAI-RDF(S)-Ontology Ontology Access WS-DAI-RDF(S)-Query Query Access WS-DAIR Relational Access Fig. 1.1 The WS-DAI family of specifications
- 21. 6 S. Lynden et al. will result in a response message indicating how many tuples have been inserted. For indirect data access, a consumer will not receive the results in the response to the request made to a data access service. Instead, the request to access data is processed with the results being made available to the consumer indirectly as a new data resource, possibly through a different data service that supports a different set of interfaces. This is useful, for instance, to hold results at the service side minimising any unnecessary data movement. The type and behaviour of the new data resource are determined by the data access service and the configuration parameters passed in with the original request. This indirect access behaviour is different from the request-response style of behaviour found in typical web service interactions. 1.2.2 The Core Specification (WS-DAI) The WS-DAI specification, also referred to as the core specification, groups interfaces into the following functional categories: • Data description: Metadata about service and data resource capabilities. • Data access: Direct access interfaces. • Data factory: Indirect access interfaces. It is important to note that data access and data factory operations wrap existing query languages to specify what data is to be retrieved, inserted or modified in the underlying data resources. The DAIS specifications do not define new query lan- guages nor do they do any processing on the incoming queries nor do they provide a complete abstraction of the underlying data resource – for instance, you have to know that the data service you are interacting with wraps a relational database to send SQL queries to it. The benefit of DAIS is that it provides a set of operations that will function on an underlying data resource without requiring knowledge of the native connection mechanisms for that type of database. This makes it easier to build client interfaces that will use DAIS services to talk to different types of databases. These interface groupings provide a framework for the data service interfaces and the properties that describe, or modify, the behaviour of these interfaces that can then be extended to define interfaces to access particular types of data, as is done by the WS-DAIR, WS-DAIX and WS-DAI-RDF(S) documents. 1.2.2.1 Data Description Data Description provides the metadata that represents the characteristics of the database and the service that wraps it. The metadata are available as properties that can be read and sometimes modified. If WSRF is used, the WSRF mechanisms can be used to access and modify properties otherwise operations are available to do this for non-WSRF versions of WS-DAI. For instance, the message GetDataResource- PropertyDocument will retrieve metadata that includes the following information:
- 22. 1 Open Standards for Service-Based Database Access and Integration 7 1. AbstractNames: A set of abstract names for the data resources that are available through that data services. Abstract names are unique and persistent name for a data resource represented as URIs. 2. ConcurrentAccess: A flag indicating whether the data service provides concur- rent access or not. 3. DataSetMap: Can be used to retrieve XML Schema representing the data formats that the data service can return the results in. 4. LanguageMap: Shows the query languages that are supported by the data service. Note that DAIS does not require the service to validate the query language that is being used; improper languages will be detected by the underlying data resource. 5. Readable/Writable: A flag indicating whether the data service provides read and write capabilities to the resource. For instance, if the service were providing access to a digital archive it would clearly only have read-only access. This property is meant to describe the underlying characteristics of the data resource rather than authorization to access the resource. 6. TransactionInitiaton: Information about the transactional capabilities of the underlying data resource. Using this information, a user can understand the database and service capabili- ties provided by that data service. The property set can be extended to accommodate particular properties pertaining to access to specific types of data, for example, relational, XML and RDF. 1.2.2.2 Data Access Data access collects together messages that can directly access or modify the data represented by a data access service along with the properties that describe the behaviour of these access messages, as illustrated in Fig. 1.2, which depicts a use case where the WS-DAIR interfaces are used. Consumer Database Data Access Service SQLAccess Relational Database SQLResponse SQLExecute ( Data ResourceAbstractName, DatasetFormatURI, SQLExpression ) SQLDescription: Readable Writeable ConcurrentAccess TransactionInitiation TransactionIsolation Etc. Fig. 1.2 Data access example
- 23. 8 S. Lynden et al. In this example, the data access service implements the SQLAccess messages and exposes the SQLDescription properties; more details about the interface and corresponding properties can be found in [7]. A consumer uses the SQLExecute message to submit an SQL expression. The associated response message will contain the results of the SQL execute request. When the SQL expression used is a SELECT statement, the SQL response will contain the data in a RowSet message serialized using an implementation-specific data format, for example the XML WebRowSet [11]. 1.2.2.3 Data Factory Factory messages create a new relationship between a data resource and a data access service. In this way, a data resource may be used to represent the results of a query or act as a place holder where data can be inserted. A data factory describes properties that dictate how a data access service must behave on receiving factory messages. The factory pattern may involve the creation of a new data resource and possibly the deployment of a web service to provide access to it (though existing web services can be re-used for this purpose – DAIS does not specify how this should be implemented). The WS-DAI specification only sets the patterns that should be used for extensions to particular types of data. This ability to derive one data resource from another, or to provide alternative views of the same data resources, leads to a collection of notionally related data resources, as illustrated in Fig. 1.3, which again takes an example from the WS- DAIR specification. The database data access service in this example presents an SQLFactory interface. The SQLExecuteFactory operation is used to construct a new derived data resource from the SQL query contained in it. These results are then made available through an SQLResponseAccess interface which may be available through the original service or as part of a new data service. Access to the RowSet resulting from the SQL expression executed by the underlying data resource is made available through a suitable interface, assuming that the original expression contains a SELECT statement. The RowSet could be stored as a table in a relational database or in a form decou- pled from the database. DAIS does not specify how this should be implemented but the implementation does have a bearing on the properties of ChildSensitiveToParent and ParentSensitiveToChild which indicate whether changes in the child data affect the parent data or changes in the parent data affect the child data, respectively. The RowSet results are represented as a collection of rows via a data access service supporting the SQLResponseAccess collection of operations that allow the RowSet to be retrieved but does not provide facilities for submitting SQL expressions via the SQLAccess portType. The Factory interfaces provide a means of minimising data movement when it is not required in addition to an indirect form of third party data delivery: consumer A creates a derived data resource available through some specified data service whose reference can be passed on to consumer B to access.
- 24. 1 Open Standards for Service-Based Database Access and Integration 9 Consumer Database Data Access Service SQLFactory Relational Database Reference to SQLResponse Data Service SQLExecuteFactory (Data ResourceAbstractName, PortTypeQName ConfigurationDocument SQLExpression) SQLDescription: Readable Writeable ConcurrentAccess TransactionInitiation TransactionIsolation Etc. SQL Response Data Access Service SQLResponseAccess SQLResponseDescr- iption: .. Rowset GetRowset (Data ResourceAbstractName, RowsetNumber) Row Set Fig. 1.3 Data factory example The data resources derived by means of the Factory-type interfaces are referred to as data service managed resources as opposed to the externally managed resources which are database management systems exposed by the data services. Clearly, the creation of these derived data resources will consume resources, thus resulting in operations such as DestroyDataResource being provided. Soft state lifetime management of data resources is not supported by WS-DAI unless WSRF is used. 1.3 The Relational Extension (WS-DAIR) Relational database management systems offer well-understood, widely used tools embodied by the ubiquitous SQL language for expressing database queries. As a natural result of this, the DAIS working group focused on producing the WS-DAIR extensions which defines properties and operations defined to deal with relational data. A brief overview of these extensions is given here, starting with the properties
- 25. 10 S. Lynden et al. defined by WS-DAIR to extend the basic set of data resource properties defined by WS-DAI: • SQLAccessDescription: Defines properties required to describe the capabilities of a relational resource in terms of its ability to provide access to data via the SQL query language. • SQLResponseDescription: Defines a set of properties to describe the result of an interaction with a relational data resource using SQL. For example, the number of rows updated, the number of result sets returned and – or any error messages generated when the SQL expression was executed. • SQLRowSetDescription: Defines properties describing the results returned by an SQL SELECT statement against a relational database, including the schema used to represent the query result and the number of rows that exist. The following direct access interfaces are defined by WS-DAIR: • SQLAccess: Provides operations for retrieving SQLAccessDescription proper- ties (although for implementations that use WSRF should be able to employ the methods defined there as well) and executing SQL statements (via a SQLExecuteRequest message). • SQLResponseAccess: Provides operations for processing the responses from SQL statements, for example, retrieval of SQLRowsets, SQL return values and output parameters. • SQLRowSetAccess: Provides access to a set of rows through a GetTuples operation. • SQLResponseFactory: Provides access to the results returned by an SQL state- ment. For example, the SQLRowsetFactory operation can be used to create a new data resource supporting the SQLRowset interface. Example XML representations of an SQLExecuteRequest and a corresponding response message are shown in Fig. 1.4. 1.4 The XML Extension (WS-DAIX) The growing popularity of XML databases and the availability of expressive query languages such as XQuery means that the provision of an extension to WS-DAI to cater for XML databases. Work on WS-DAIX was undertaken in addition to the WS-DAIR effort from the start. A key difference to the relational specification is that XML databases may support a number of different query languages that need to be catered for: XQuery, XUpdate and XPath, although XQuery can, in effect, encompass the capabilities of XUpdate and XPath. The following property sets are defined by WS-DAIX: • XMLCollectionDescription: Provides properties describing an XML collection, such as the number of documents and the presence of an XML schema against which documents are validated.
- 26. 1 Open Standards for Service-Based Database Access and Integration 11 wsdair: SQLExecuteRequest xmlns:wsdai=http://www.ggf.org/namespaces/2005/12/WS-DAI xmlns:wsdair=http://www.ggf.org/namespaces/2005/12/WS-DAIR wsdai:DataResourceAbstractNamewsdai:EmployeeDB/wsdai:DataResourceAbstractName wsdai:DatasetFormatURIhttp://java.sun.com/xml/ns/jdbc/wsdai:DatasetFormatURI wsdair:SQLExpression wsdair:ExpressionSELECT name,age FROM persons/wsdair:Expression /wsdair:SQLExpression /wsdair:SQLExecuteRequest SQLExecuteRequest Message SQLExecuteResponse Message SQL Query Expression wsdair:SQLExecuteResponse xmlns:wsdai=http://www.ggf.org/namespaces/2005/12/WS-DAI xmlns:wsdair=http://www.ggf.org/namespaces/2005/12/WS-DAIR” xmlns:wrs=http://java.sun.com/xml/ns/jdbc wsdai:DatasetFormatURI http://java.sun.com/xml/ns/jdbc /wsdai:DatasetFormatURI wsdai:DatasetData wrs:webRowSet properties commandselect name, age from persons/command ... /properties metadata ... /metadata data currentRow columnValueJenkins/columnValuecolumnValue26/columnValue /currentRow currentRow columnValueRogers/columnValuecolumnValue35/columnValue /currentRow currentRow columnValueWalsh/columnValuecolumnValue42/columnValue /currentRow /data /wrs:webRowSet /wsdai:DatasetData /SQLExcuteResponse WebRowSet-encoded Dataset Data (tuples) WS-DAI core Properties Fig. 1.4 An SQLExecuteRequest/response example (direct access) • XMLSequenceDescription: Describes an XML sequence, usually created as the result of an XPath or XQuery expression. Specifically, a property to define the length of the sequence is provided. It should be noted that no extra properties are defined to describe data resources with XPath, XUpdate or XQuery capabilities as the WS-DAI-defined properties such as LanguageURI, DatasetFormatURI, etc. are adequate for this purpose. The following direct data access interfaces are supported: • XMLCollectionAccess: Provides access to an XML collection via operations supporting addition/removal or documents and sub-collections. • XQueryAccess: Allows the evaluation of XQuery expressions across collections of XML documents represented by an XML resource.
- 27. 12 S. Lynden et al. • XUpdateAccess: Allows an XUpdate expression to be executed against an XML resource, returning the number of updated nodes. • XPathAccess: Allows the evaluation of XPath expressions across collections of XML documents represented by an XML resource. • XMLSequenceAccess: Provides access to an XML sequence created as a result of an XPath/XQuery query. The GetItems operation of this interface allows the client to obtain specific subsequences of the overall result. The following indirect access interfaces are supported: • XMLCollectionFactory: Provides access to collections and documents in collec- tions. • XPathFactory: Provides the XPathQueryFactory that allows new data resources (supporting the XMLSequenceAccess interface) to be created as the result of an XPath query. • XQueryFactory: Provides the XQueryExecuteFactory operation to create new XMLSequenceAccess data resources as the result of an XQuery query. 1.5 The RDF Extension (WS-DAI-RDF(S)) The RDF is a World Wide Web Consortium (W3C) set of recommendations [12] focused on the representation and management of metadata. It includes two data models, RDF and RDF Schema, whose combination is known as RDF(S). The WS-DAI-RDF(S) extension to this domain provides data access mechanisms for RDF(S) data, divided into two types based on the style of access: declarative or programmatic. Hence, the following specifications are in the process of being defined within DAIS to access RDF(S) data. 1. WS-DAI RDF(S) Querying: This specification provides a query language inter- face to RDF data based on the W3C SPARQL query language [13] for RDF. 2. WS-DAI RDF(S) Ontology: This specification provides an API style of access based on ontology handling primitives conforming to the RDF(S) model. These primitives provide various operations including the possibility of performing updates to the ontology. 1.5.1 The WS-DAI RDF(S) Querying Specification The objective of the querying specification is to provide an SPARQL interface to RDF data. The W3C has defined several related specifications based on SPARQL, including an XML-based query results format [14] and a protocol for accessing RDF resources [15]. The WS-DAI-RDF(S) Querying specification, the interaction patterns of which are illustrated in Fig. 1.5, is defined to be compatible with the
- 28. 1 Open Standards for Service-Based Database Access and Integration 13 Consumer RDF(S) Database Data Access Service SPARQLAccess SPARQLExecuteResponse SPARQLExecute (Data ResourceAbstractName, DatasetFormatURI SPARQLQueryRequest) SPARQLAccess Description Direct Access RDF(S) Database Data Access Service SPARQLFactory SPARQLAccess Description Indirect Access ResultsSet Data Access Service TriplesSet Data Access Service SPARQLItems Description SPARQL ResultsSetAccess SPARQL TriplesSetAccess Construct/ Describe Select/ Ask Consumer Results GetResults (Start Position ResultsCount) GetTriples (Start Position, ResultsCount) Results Reference to data access service SPARQLExecuteFactory (Data ResourceAbstractName PortTypeQName ConfigurationDocument SPARQLQueryRequest) Fig. 1.5 Overview of WS-DAI RDF(S) querying specification W3C standards (e.g. by supporting the SPARQL query language and the XML results format) while also benefitting from the WS-DAI approach. For example, indirect access is not supported by the W3C SPARQL protocol, meaning that when using the SPARQL protocol all query results are returned directly to the consumer
- 29. 14 S. Lynden et al. accessing the service. In contrast, WS-DAI-RDF(S) allows the consumer to control the retrieval of query results, a feature that can be extremely useful in certain scenarios, such as when retrieving large result sets. 1.5.1.1 Indirect Access Using TriplesSetAccess and ResultsSetAccess SPARQL has four query forms: CONSTRUCT, DESCRIBE, SELECT and ASK. The first and second forms return an RDF graph as a query result (CONSTRUCT returns an RDF graph constructed by substituting variables in query patterns, while DESCRIBE returns an RDF graph that describes the resources found). Other representations also exist but the important thing is that they are modeled as triples. For this purpose, the WS-DAI-RDF(S) specification introduces a TriplesSetAccess interface to provide access to the results. In contrast to these two forms, the results of the other two forms are not RDF graphs: SELECT returns variables bound during the matching of an RDF graph against a basic graph pattern specified in the query; ASK returns a boolean value indicating whether there is a match for a query pattern. The WS-DAI-RDF(S) specification introduces a ResultsSetAccess interface to access the results of these query forms, based on the SPARQL Result Set XML Format specification. 1.5.2 The WS-DAI RDF(S) Ontology Specification The object of the WS-DAI-RDF(S) Ontology access specification is to provide an integral access mechanism for RDF(S) sources that goes beyond the retrieval capabilities offered by the querying specification, whilst providing a simple but complete set of functionalities that abstract the most general necessities a consumer may have when accessing with RDF(S) data sources. To achieve this objective, the specification proposes a model-based access mechanism for accessing RDF(S) sources at the conceptual level, that is, an access mechanism that revolves around the concepts and semantics defined by the RDF(S) model. Thus, the specification details a set of ontology handling primitives for dealing with such models, hiding the syntactic aspects of RDF(S) and transparently exploiting its semantics. 1.5.2.1 Data Resources The WS-DAI-RDF(S) Ontology specification differentiates several types of RDF(S) data resources, each of them provided to allow access to, and manage elements of RDF(S) sources at different levels of granularity. They can be divided into two groups: • Placeholders for built-in RDF(S) classes (Resource, Class, Property, Statement, Container and List data resources): These data resources provide class-oriented views of an RDF(S) resource.
- 30. 1 Open Standards for Service-Based Database Access and Integration 15 • Convenience abstractions (RepositoryCollection and Repository data resources), for RDF(S) sources that contain more than a resource. 1.5.2.2 Interfaces for Direct and Indirect Access To interact with the data resources described above, several interfaces are provided in the WS-DAI-RDF(S) Ontology specification. The first group is for the direct access interfaces: • RepositoryCollectionAccess: Provides access to the repositories of a collection. • RepositoryAccess: Provides access to the repository content, offering function- ality for managing the repository at RDF(S) resource level. • ResourceAccess: Provides access to a particular RDF(S) resource, concentrating in those aspects common to every resource: property value management, resource description, etc. • ClassAccess: Provides access to particular RDF(S) resources that are an RDF(S) class, focusing on the data that is specific to RDF(S) classes: class hierarchy traversal, instance retrieval, etc. • PropertyAccess: Provides access to particular RDF(S) resources that are RDF(S) properties, focusing on the data that is specific to RDF(S) properties: range and domain management, property hierarchy traversal, etc. • StatementAccess: Provides access to particular RDF(S) resources that are RDF(S) statements reified triples, not the triples themselves focusing on the management of the components that set up the reification. • ListAccess and ListIteratorAccess: Provides access to particular RDF(S) resources that are RDF collections (List), focusing on the management of the members of a collection, as well as, the structure of the collection. • ContainerAccess and ContainerIteratorAccess: Provides access to particular RDF(S) resources that are RDF(S) containers, focusing on the management of the members of the container, as well as the structure of the container, regardless the its specific type. • AltAccess: Provides access to particular RDF(S) containers that are of the particular alt type. There are also indirect access interfaces: • RepositoryCollectionFactory: Provides access to the repositories in a collection. • RepositoryFactory: Provides access to the repository content. • ListFactory: Provides access to the contents of an RDF collection. • ContainerFactory: Provides access to the contents of a container. Finally, due to the large number of operations the aim will be to incrementally introduce the different levels of functionality described previously through three different profiles documents, schematically illustrated in Fig. 1.6. These will provide support for the different types of use case, of increasing complexity, with basic RDF support, RDF Schema support and, finally, full RDF support. It is envisaged that,
- 31. 16 S. Lynden et al. WS-DAI-RDF(S) Ontology Specification StatementAccess ContainerAccess ContainerFactory ContainerIterator AltAccess ListAccess ListFactory ListIterator Profile 2: Full RDF(S) Support Profile 1: RDF Schema Support Profile 0: Basic RDF Support Statement Data Resource Container Data Resource List Data Resource ClassAccess PropertyAccess Class Data Resource Property Data Resource RepositoryCollectionAccess RepositoryCollectionFactory RepositoryAccess RepositoryFactory ResourceAccess RepositoryCollection Data Resource Repository Data Resource Resource Data Resource Fig. 1.6 Profile documents for the WS-DAI-RDF(S) ontology specification like a Russian doll, implementation of a given level of profile will also require the previous levels to also be implemented. 1.6 Implementations Implementations of the specifications are important for a number of reasons – first, they serve to debug and test the specifications during their development. Second, they provide examples to potential adopters of the specifications in use, allowing easier implementations to be constructed by developers. Third and most importantly, implementations are necessary to promote the specifications, allow them to become widely recognised, and foster adoption, a factor by which the success of specifications will ultimately by judged. Several implementations of the DAIS specifications have been developed to serve as experimental platforms during the specification development process and following that, implementations have also been produced as part of research projects developing applications of the specifications. The following is a list of the implementations that have been made public to date. 1.6.1 WS-DAIR Implementations • AMGA2 is a metadata catalogue compliant with the EGEE grid environment.The implementation of the metadata catalogue provides various interfaces, including a SOAP WS-DAIR compliant implementation. 2 http://amga.web.cern.ch/amga/.
- 32. 1 Open Standards for Service-Based Database Access and Integration 17 • OGSA-DAI3 is an open-source distributed data access and management system supporting Web service-based access to data. OGSA-DAI WS-DAIR, an imple- mentation of the WS-DAIR interfaces using the OGSA-DAI middleware that can be obtained from the OGSA-DAI SourceForge site.4 1.6.2 WS-DAIX Implementations • An implementation of WS-DAIX is also available from the OGSA-DAI source SourceForge site. 1.6.3 WS-DAI-RDF Implementations • The EC funded ADMIRE (Advanced Data Mining and Integration for Eu-rope) [www.admire-project.eu/] project has developed an implementation of the WS- DAI-RDF Querying Specification. • AIST’s OGSA-DAI-RDF project has developed an implementation of the WS- DAI-RDF Querying Specification, which can be obtained from.5 The OGF process requires that two independent interoperable implementations exist before a proposed recommendation can become a full OGF recommendation. To date, two of the above implementations (WS-DAIR implementations from the OGSA-DAI and AMGA projects) have been utilised to validate the WS-DAI and WS-DAIR specifications as reported in [16]. A comparison of the functionality of these implementations is made in Table 1.1. The performance of the imple- mentations is dependent on the underlying DBMS being utilised; however, the overhead incurred by the WS-DAI(R) Web service-based interfaces is similar for both implementations.6 1.7 Applications The set of potential areas of application for the DAIS specifications is wide ranging and some research projects have already become early adopters of them. This section provides two examples of the application of the WS-DAI-RDF specification. 3 http://ogsadai.org.uk. 4 http://ogsa-dai.sourceforge.net/. 5 http://dbgrid.org. 6 A brief analysis of the performance of the AMGA WS-DAIR implementation and some compar- isons with the OGSA-DAI implementation can be found under “Design and Implementation of WS-DAIR for AMGA” available at http://event.twgrid.org/isgc2009/program.htm.
- 33. 18 S. Lynden et al. Table 1.1 A comparison of the OGSA-DAI and AMGA implementations of WS-DAI and WS- DAIR OGSA-DAI AMGA Apache Axis gSoap Infrastructure Java 1.4 CCC SOAP binding rpc/literal Document/literal Any JDBC-enabled Supports MySQL, SQLLite Underlying DBMS relational database PostgreSQL, Oracle SQL (dependent on SQL-92 Supported languages underlying DBMS) AMGA Metadata Language Yes Stored procedures (if supported by DBMS) No WebRowSet Datasets comma-separated values WebRowSet SSL, GSI, VOMS Security features None permission, ACL ServiceBusyFault ServiceBusyFault SQL CommunicationArea Un-supported features GenericQuery (for fault messages) One of them is the ADMIRE registry, which uses the WS-DAI-RDF specifications to provide support in a data mining and integration (DMI) context. The second presents a scenario in which the specifications can be applied to distributed SPARQL query processing. Other applications that make use of the other specifications have already been pointed out above, such as the AMGA and OGSA-DAI projects. These provide additional examples where the WS-DAI and WS-DAIR specifications have been implemented for the convenience and benefit of their users. 1.7.1 ADMIRE The ADMIRE7 registry allows a range of DMI components, called processing elements (PE), to be registered and discovered, together with the set of types, in the context of their inputs and outputs, that can be handled by those processing elements. The descriptions used in the registrations contain the data types of the input and output parameters for each PE and any restrictions associated with these, such as: the relationships between the inputs and outputs, termination conditions, are error conditions and all these information are available at the registry in an RDF format. In ADMIRE, users create these PEs and register their descriptions in a registry by means of a register operation as defined in the DISPEL language [17]. Users can 7 EU FP7 ICT 215024 www.admire-project.eu.
- 34. 1 Open Standards for Service-Based Database Access and Integration 19 then retrieve PE descriptions by using SPARQL. By adding a web service layer, the registry may be accessed by different users at different times in different contexts (binding the states to the users). The WS-DAI-RDF(S) specification thus provides a convenient way of providing standardised access to this RDF-based data repository, and this is what has been achieved by ADMIRE. 1.7.2 Distributed Query Processing The WS-DAI-RDF(S) specifications allow data integration applications to be constructed on top of the consistent interfaces provided by WS-DAI-RDF(S) data resources. When integrating data from distributed data sources, it is necessary to deal with syntactic heterogeneities that may be present between the interfaces used to interact with data resources. Furthermore, data retrieval mechanisms must support delivery mechanisms that allow clients some form of control over the rate at which data is delivered, especially when scalability is desired. This is important for grid-based distributed databases, where data is federated and accessible via service-based interfaces. The standardised interfaces provided by the WS-DAI-RDF(S) specifications mean that many heterogeneities present amongst the individual data sources are resolved when performing these tasks. Data integration may be performed by multiple computational resources, and the WS- DAI indirect data access pattern can be used to execute sub-queries which result in the creation of a new data resource for each set of query results. The various data integration tasks (e.g., joins, unions) that need to be performed can then delegated to appropriate nodes in a set of computational resources, which are given references to the created data resources that need to be accessed to perform their allocated tasks. This therefore allows parallel and distributed query processing to take place following the approach used by the OGSA-DQP [18] distributed query processor, which uses OGSA-DAI data resources. The DAIS specification’s operations allow similar applications to be developed accessing data resource using open standards. 1.8 Conclusions This chapter has given an overview of the WS-DAI family of specifications that have been the focus of the DAIS Working Group of the OGF. A core specification provides a framework which can then be extended to deal with specific types of data. This process has already been realised for relational, XML and RDF data, and some initial proposals have been also made for other types of databases. Generally, the DAIS approach provides a core specification and a flexible framework that allows extensions if further requirements are specified of the core specification, which may in turn impact on the other extension specified.
- 35. 20 S. Lynden et al. This chapter’s review of the interfaces provided by the specifications has focused, in particular, on the novel use of indirect data access to provide a means of minimising data movement, allowing derived data resources to be deployed and exposed at the server side. These specifications provide a means of abstracting out some of the variability in the data resources used in distributed environments and presenting uniform interfaces to specific types of data – for now: relational, XML and RDF data – to clients. The use of web services to do this ensures a certain degree of programming language neutrality and portability across different computer systems. For these reasons, it is expected that the adoption of these specifications will facilitate the management and integration of data across the distributed environments presented by Grids and Clouds. Acknowledgements The authors thank all those people who have participated in the process of developing and ratifying the DAIS specification documents and OGF for hosting the process. References 1. Catlett, C., de Laat, C., Martin, D., Newby, G., Skow, D.: Open Grid Forum Document Process and Requirements [Obsoletes GFD.1] GFD.152. Open Grid Forum, 2009. http://www.ogf.org/ documents/GFD.152.pdf 2. Atkinson, M.P., Dialani, V., Guy, L., Narang, I., Paton, N.W., Pearson, P., Storey, T., Watson P.: Grid Database Access and Integration: Requirements and Functionalities. GFD-I-13. Open Grid Forum, 2003. http://www.ogf.org/documents/GFD.13.pdf 3. Foster, I., Kishimoto, H., Savva, A., Berry, D., Djaoui, A., Grimshaw, A., Horn, B., Maciel, F., Subramaniam, R., Treadwell, J., Von Reich, J.: The Open Grid Services Architecture, Version 1.0. OGF GFD-I.030. Open Grid Forum, 2005. http://www.ogf.org/documents/GFD.30.pdf 4. Web Service Resource Framework (WSRF) Specifications. OASIS. http://www.oasis-open.org/committees/tc home.php?wg abbrev=wsrf. Accessed 9 Oct 2010 5. Atkinson, M., De Roure, D., Dunlop, A., Fox, G., Henderson, P., Hey, T., Paton, N., Newhouse, S., Parastatidis, S., Trefethen, A., Watson, P., Webber, J.: Web Service Grids: An Evolutionary Approach. Technical report, UK e-Science Institute. http://www.nesc.ac.uk/technical papers/ UKeS-2004-05.pdf. Accessed 9 Oct 2010 6. Antonioletti, M., Atkinson, M., Laws, S., Malaika, S., Paton, N.W., Pearson, D., Riccardi, G.: Web Services Data Access and Integration (WS-DAI) Specification Version 1.0. OGF GFD.74. Open Grid Forum, 2006. http://www.ogf.org/documents/GFD.74.pdf 7. Antonioletti, M., Collins, B., Krause, A., Laws, S., Magowan, J., Malaika, S., Paton, N.W.: Web Services Data Access and Integration The Relational Realisation (WS-DAIR) Specification Version 1.0. OGF GFD.76. Open Grid Forum, 2006. http://www.ogf.org/ documents/GFD.76.pdf 8. Antonioletti, M., Hastings, S., Krause, A., Langella, S., Laws, S., Malaika, S., Paton, N.W.: Web Services Data Access and Integration. The XML Realisation (WS-DAIX) Specification Version 1.0. OGF GFD.75. Open Grid Forum, 2006. http://www.ogf.org/documents/GFD.75. pdf 9. Antonioletti, M., Aranda, C.B., Corcho, O., Esteban-Gutirrez, M., Gmez-Prez, A., Kojima, I., Lynden, S., Pahlevi. S.M.: WS-DAI RDF(S) Realization: Introduction, Motivational Use Cases and Terminologies GFD-I 163. Open Grid Forum, 2009. http://www.ogf.org/documents/ GFD.163.pdf
- 36. 1 Open Standards for Service-Based Database Access and Integration 21 10. Gudgin, M., Hadley, M., Rogers, T.: Web Services Addressing 1.0 – Core. W3C Recommendation. World Wide Web Consortium, 2006. http://www.w3.org/TR/ws-addr-core 11. Bruce, J.: JSR-000114 JDBC RowSet Implementations, 07 April 2004. http://jcp.org/ aboutJava/communityprocess/final/jsr114 12. The Resource Description Framework (RDF) Specifications, (last visited on 10/10/10). http:// www.w3.org/standards/techs/rdf#w3c all 13. Prud’hommeaux, E., Seaborne, A.: SPARQL Query Language for RDF. W3C Recommenda- tion. World Wide Web Consortium, 2008. http://www.w3.org/TR/rdf-sparql-query 14. Beckett, D., Broekstra, J.: SPARQL Query Results XML Format – W3C Recommendation. World Wide Web Consortium, 15 January 2008. http://www.w3.org/TR/rdf-sparql-XMLres 15. Grant Clark, K., Feigenbaum, L., Torres, E.: SPARQL Protocol for RDF. W3C Recommendation. World Wide Web Consortium, 15 January 2008. http://www.w3.org/ TR/rdf-sparql-protocol 16. Lynden, S., Antonioletti, M., Jackson, M., Ahn, S.: WS-DAI and WS-DAIR Implementations – Experimental Document GFD.160. Open Grid Forum, 2009. http://www.ogf.org/documents/ GFD.160.pdf 17. Atkinson, M., Brezany, P., Krause, A., van Hemert, J., Janciak, I., Yaikhom, G.: DISPEL: Grammar and Concrete Syntax, version 1.0. Deliverable report D1.7, the ADMIRE Project, February 2010. http://www.admire-project.eu/docs/ADMIRE-D1.7-research-prototypes.pdf 18. Dobrzelecki, B., Krause, A., Hume, A., Grant, A., Antonioletti, M., Alemu, T., Atkinson, M., Jackson, M., Theocharopoulos, E.: Integrating distributed data sources with OGSADAI DQP and Views. Phil. Trans. Roy. Soc. A 368, 4133–4145 (2010)
- 37. •
- 38. Chapter 2 Open Cloud Computing Interface in Data Management-Related Setups Andrew Edmonds, Thijs Metsch, and Alexander Papaspyrou Abstract The Cloud community is a vivid group of people who drive the ideas of Cloud computing into different fields of Information Technology. This demands for standards to ensure interoperability and avoid vendor lock-in. Since such standards need to satisfy many requirements, use cases, and applications, they need to be extremely flexible and adaptive. The Open Cloud Computing Interface (OCCI) family of specifications aims to achieve this goal: originally developed for the deployment of infrastructure Clouds, it can also be used in different service and deployment models. This article will outline the OCCI specifications and demonstrate how they can be used in data management-related setups. Not only can OCCI be easily integrated but it can also be used to deploy data-centric applications (which are secured by SLAs), support data-awareness in scheduling, as well as directly interface with data management tools in a PaaS-based manner. To demonstrate this, three use cases are discussed in this article. 2.1 Introduction Next to traditional HPC and Grid computing, Cloud computing has become a new driver for the global IT market. The overall idea is to deliver a service to the customer. Instead of traditionally boxing and shipping of software products, A. Edmonds () Intel Ireland Branch, Collinstown Industrial Park, Leixlip, County Kildare, Ireland e-mail: andrewx.edmonds@intel.com T. Metsch Platform Computing GmbH, Europaring 60, 40878 Ratingen, Germany e-mail: tmetsch@platform.com A. Papaspyrou Technische Universität Dortmund, Institut für Roboterforschung, 44221 Dortmund, Germany e-mail: alexander.papaspyrou@tu-dortmund.de S. Fiore and G. Aloisio (eds.), Grid and Cloud Database Management, DOI 10.1007/978-3-642-20045-8 2, © Springer-Verlag Berlin Heidelberg 2011 23
- 39. 24 A. Edmonds et al. software is now delivered as a service to the customer directly. This change in use of computing services changes the IT landscape drastically – not only will data centers most probably transform into service providers but also the way service providers and customers interact will change. One example is billing in all businesses where a Pay-per-Use model can be easily established. The next major change in this area will be the management of data: starting with the idea of moving compute resources to the data (data-aware scheduling) as an obvious step also the way how data is treated in the Cloud (manipulation of data – NoSQL vs. Relational Databases vs. Virtual Disc Images) will evolve. Countless other opportunities such as signing, tracing changes and movement of data are still ahead of us. Since many customers move into the cloud the deployment of their data and the applications becomes very important to them. Still, most Cloud computing providers currently focus on providing Infrastructure-as-a-Service (IaaS)1 but this might change as the industry moves its focus into the idea of providing Platform-as- a-Service (PaaS) where services are constructed on a higher (non-OS, but rich API) level to provide services surrounding the data. Still, the underlying technology is evolving: standards are being developed and technologies emerge (like virtualisation). As such, there is a demand for ensuring clean interfaces and protocols which are easy to use and can be used for multiple kinds of service offerings to prevent a vendor lock-in. In the context of these developments, the Open Cloud Computing Interface (OCCI) working group works towards forming such a standard. The OCCI family of specifications can be used for IaaS and PaaS offerings. In this paper, it is demonstrated how OCCI can be used in data-centric setups for IaaS and PaaS offerings. To this end, a setup is described in which Virtual Machines (containing Databases etc.) can be deployed in a Cloud environment while ensuring certain Service Level Agreements (SLAs). Another use case demoes the ability of OCCI for moving compute resource towards large datasets. The last scenario works (in contrast to the former two) towards a PaaS scenario: it shows a Key-Value store implementation over OCCI. The purpose of these use cases is to show the need for an interoperable Cloud interface/protocol which can be used in all layers of the Cloud stack. Furthermore, it demonstrates that OCCI provides flexible usage models for a very heterogeneous field of scenarios in the broader field of data management in the Cloud. The rest of the paper is organised as follows: in Sect. 2.2, the OCCI family of specification is introduced. Next, three use cases for the application of OCCI are exemplified in Sects. 2.3–2.5. Finally, the paper concludes with a summary of achievements and shows future work. 1 Usually in the form of virtual machines.
- 40. 2 OCCI for Data Management 25 Service Provider Domain Resource Management Framework Resources OCCI Service Consumer most interoperable least interoperable Proprietary API HTTP Fig. 2.1 OCCI and its position in the service provider context 2.2 Open Cloud Computing Interface OCCI is an effort driven by a working group in the standards track of the Open Grid Forum.2 It strives to create an open, interoperable protocol and API for the Cloud. The group started with a clear focus on provisioning IaaS but later extended the focus to include other layers in the Cloud stack as well. The following diagram (Fig. 2.1) shows where OCCI fits in the service provider context. The OCCI protocol can be used for integration, ensuring interoperability and portability between service providers. Proprietary APIs can be used alongside OCCI in the case that other features than those of OCCI are maintained. The specification strives to be very easy, flexible and extensible. Therefore, it is broken into different modules. It starts with a module describing the core models. Another module describes how this model can be mapped and rendered using a HTTP/REST approach. The third module describes the infrastructure entities and how they related to the core model. 2.2.1 Motivation for Standards Main driver for standards in the past has been interoperability. This is still a fundamental part of what standards want to achieve. Still there are nuances in the term interoperability which are important and need to be looked upon separately: 2 http://www.ogf.org/.
- 41. 26 A. Edmonds et al. Interoperability. Describes how two services can inter-operate on the fly. This demands a standardised API and protocol (e.g. live migrating a virtual machine from one host to another, which are in different management domains). Integration. Describes how a service provider can bring together different tech- nologies and interconnect them within his domain (e.g. integrate a virtual machine management tool with an identity management system). Portability. This is mostly about the porting between service providers. In com- parison with interoperability, there is no direct connection between the service provider. This demands that there are standardised data formats which providers can understand (e.g. porting a virtual machine from one hypervisor to another). Innovation. Standards have always been started when a field in the IT community gains popularity, is widely adopted and begins on a path of commoditisation. Next to interoperability, standards can be a driver for innovation as well as widely adopted innovations can demand standards. Reusability. This can be seen on two levels. First the reuse of (legacy) codes through basic standardised APIs and the reuse of the standard itself in different fields. 2.2.2 The Core Model The core meta-model [10] for OCCI imposes a general means of handling general resources, providing semantics for defining the type of a given entity, describing interdependencies in between different entities, and defining operating character- istics on them. Although the meta-model aims to ease the implementation burden by setting a common ground for other OCCI-related specifications, it can be used as a standalone component in other contexts (e.g. Resource Oriented Architectures (ROAs)) as well. The UML class diagram shown (Fig. 2.2) gives an overview of the OCCI core meta-model. At its heart lies the Resource type. Any resource exposed through OCCI is a Resource or sub-type thereof. A resource can be for example a virtual machine, a job in a job submission system, a user, etc. The Resource type contains a number of common attributes that domain-specific Resource types inherit. The Resource type is complemented by the Link type which associates one Resource instance with another. The Link type also contains a number of common attributes that domain-specific Link types inherit. Entity is an abstract type which both Resource and Link inherit. Each sub- type of Entity is identified by a unique Kind instance. The Kind type comprise the classification system built into the OCCI model. Kind is a specialisation of Category and introduces additional capabilities in terms of Action types. 2.2.2.1 Classification and Identification The OCCI model provides a built-in classification system allowing for safe exten- sion towards domain-specific usage. This system is like a “type system” but with the possibility of being easily exposed over a text-based protocol.
- 42. 2 OCCI for Data Management 27 Mixin Kind Action Entity −id :URI −title :String [0..1] Resource −summary:String[0..1] Link Category −scheme :URI −term :String −title :String [0..1] −attributes : SetString cd: Core target + source * links * actions kind * * mixins * actions * * related * * related Fig. 2.2 UML class diagram of the OCCI model. The diagram provides an overview of the OCCI model but is not a standalone definition thereof The classification system can be summarised with the following key features: • Each OCCI base type and extension thereof is assigned a unique identifier, a structural Kind, which allows for dynamic discovery of available types. • The relationship of structural Kinds is part of the system and thus the inheritance model is also discoverable. • The classification system allows non-structural Kinds to be assigned to resource instances adding new capabilities using a mix-in-like model. • Tagging of resource instances is supported through mix-in of non-structural Kinds which have no additional capabilities defined. • A collection of associated resources is implicitly defined for each structural and non-structural Kind. That is all resource instances associated with a particular Kind instance form a collection.
- 43. 28 A. Edmonds et al. 2.2.2.2 Categorisation The Category type comprises the basis of the identification mechanism used by the OCCI classification system. Instances of the Category type are only used to identify Action types. All other uses of Category properties are managed through its sub-type Kind. A Category is uniquely identified by concatenating the categorisation scheme with the category term, for example http://example.com/category/scheme#term. This is done to enable discovery of Category definitions in text-based renderings such as HTTP. Sub-types of Category such as Kind inherit this property. 2.2.2.3 Kind Relationships The OCCI base types Resource and Link extend Entity. This together with any further sub-typing implies a hierarchy of related structural Kind instances. The Kind relationships thus mirror the type inheritance structure of the OCCI model and any extension thereof. In an example where a domain-specific “Custom Compute Resource” is a sub- type, the OCCI infrastructure type Compute, which in turn is a sub-type of the Resource type, four related structural Kinds would be involved. One or more Entity instances associated with the same Kind, automatically form a collection, and each Kind identifies a collection consisting of all Entity instances of it. For example, an instance of the Resource type will always be associated with the structural Kind (http://scheme.ogf.org/occi/core#resource) and thus part of the collection implied by the Kind. Collections are, by definition of the core model, navigable and support the following operations: • Retrieve the whole collection. • Retrieve a specific item in a collection. • Retrieve a subset of a collection. 2.2.2.4 Discovery In addition to that, Kinds and Category instances a particular service provider support can be discovered. By examining these instances a client is enabled to deduce the following information: • The Entity sub-types available from a service provider, including domain- specific extensions. • The attributes associated with each Entity sub-type. • The invocable operations, that is Actions, defined for each Entity sub-type. • Additional mix-ins or tags, that is non-structural Kinds, applicable to Entity sub-type instances.
- 44. 2 OCCI for Data Management 29 Overall, the OCCI core meta-model provides a solid foundation for the remote management of resources offered in an as-a-Service manner, allowing for the devel- opment of interoperable tools for common tasks including deployment, automatic scaling and monitoring. The explicit split-out of it allows the leverage of the developed models, protocols, and APIs in manners not anticipated and to foster modularity and extensibility for future usage paradigms. 2.2.3 RESTful HTTP Rendering of the OCCI Model The OCCI Core model which is described in the previous Sect. 2.2.2 is free of any rendering and forms the base of OCCI. Based upon this model, OCCI describes a serialisation rendering. This rendering – or serialisation format – is passed on the wire between client and service, see [11]. OCCI has a default rendering which is text based and uses the HTTP protocol and implements a ROA, see [14]. In this architecture, a system is modelled as a set of related resources. ROA’s use Representation State Transfer (REST), see [6], to cater for client and service interactions. In these interactions, clients request to perform operations on the state of an individual or set of resources managed by the service. HTTP is commonly used in most ROA systems. It provides means to uniquely identify resources through URIs as well as operating upon them with a set of general-purpose operations called verbs. These HTTP verbs map loosely to the resource-related operations of create (POST), retrieve (GET), update (POST, PUT) and delete (DELETE). 2.2.3.1 Rendering of Resources Each Resource in the OCCI core model will be rendered as a unique URI (for example http://example.com/foo). Each resource can be identified uniquely by an URI and has at least one Category assigned, which defines the type and the operations that can be performed. This means that from this standpoint a resource can be almost anything like a Database entry, a Virtual Machine, an Image, etc. Resources can be linked and actions can be performed upon them. Resource of the same type (as in have the same Category assigned) can be found under a certain path relative to the root of the service provider (e.g. all storage devices will appear under the path /storage – still the path name “storage” is freely defined by the Service Provider and can do discovered through the Query interface). Since Categories cannot only be used to define the type of the resource, but also to tag or group resources, resource can show up under multiple paths. The following URL hierarchy demonstrates this feature:
- 45. 30 A. Edmonds et al. /compute/123 /storage/discABC /database/tableXYZ /nosql/entry_1 /my-linux-vms/123 // links to /compute/123 /my-datasets/discABC // links to /storage/discABC /my-datasets/tableXYZ // links to /database/tableXYZ /my-datasets/entry_1 // links to /nosql/entry_1 This very flexible system allows that the OCCI model can be used for several use cases including for Data Management operations. 2.2.3.2 Discovery of Capabilities Through a Query Interface One of the main features of OCCI is that clients can discover the capabilities of the service provider through a standardised query interface. This is important since OCCI is designed for extensibility. To query the capabilities of a Service provider implementing OCCI, the Client needs to do a HTTP GET on the URI /-/. This Category management URL allows the client to get a listing of all categories supported by the provider. Should the provider allow and support client- created categories, then this URL endpoint must support the creation of user categories as well. 2.2.3.3 Linking and Performing Actions on Resources Each of these resources can be linked with other resources. Links again are RESTful resources and have a source and a target attribute. Each link resource is bound to a category identifying it as a link. Next to linking, some type specific actions can be performed. The set of possible actions is defined by the Category of the resource. Actions are triggered by adding a fragment to the URI of the resource indicating which action should be triggered (e.g. http://example.com/foo;actionDshutdown). Parameters of the action are described in the HTTP message. 2.2.3.4 Use of HTTP Features The HTTP rendering of OCCI makes use of many HTTP features. This includes for example HTTP headers for Versioning and all Authentication features. OCCI does not explicitly define those but makes use of those features. Next to these basic features, OCCI also makes use of the Content-Type defini- tions. At a minimum, all information for OCCI resources is transferred in the HTTP
- 46. 2 OCCI for Data Management 31 Body. This is defined as the basic text/plain content type. Other content types also exist. For example, the information can also be rendered in the HTTP Header or as HTML (e.g. for browsing the OCCI interface using a Web Browser) by supplying the appropriate content-type header as specified in the specification. Rendering of data is done through simple key value associations. Also, more structural data representations such as JSON of RDFa can easily be added to OCCI. 2.2.4 OCCI for Virtual Machine (Infrastructure) Provisioning Having described the core model and a way of rendering it on the wire, a concrete compliment to the core model is now explored [12]. The infrastructure specification extends the core model at two key points: 1. To represent various infrastructure-related resources, it extends Resource using inheritance. 2. To represent concrete relationships between infrastructural resources, it extends Link using inheritance. To represent the main elemental resources found in infrastructure-type services, OCCI has three specialisations of Resource: 1. Compute: Information processing resources. 2. Network: Interconnection resources. 3. Storage: Information recording resources. Complimenting these, to allow linkage are: • NetworkInterface: Represents an L2 client device (e.g. network adapter). • StorageLink: Represents a link from a Resource to a target Storage Resource. The relations of these infrastructure resources are shown in the UML diagram (Fig. 2.3). When modeling elements, it was found that OCCI needed to support not only generic cases but also specific cases. This issue was exemplified by Network. It might be immediately attractive to model all functionalities within this Resource, including aspects of IP configuration, however, then the model would force certain technology choices upon implementers. To avoid this, the working group chose to utilise the OCCI mix-in capabilities to avoid such a situation. Where an implementer wishes to offer TCP/IP functionality on top of the Network resource, they can do so by implementing the IPNetworking mix-in. The IPNetworking mix-in allows to supplement the Network Resource with the necessary TCP/IP features. Should an implementer wish to offer another type of L3/L4 technology for example AppleTalk or IPX, then they only need implement a custom network mix-in.
- 47. 32 A. Edmonds et al. IPNetworking IPNetworkInterface Link (from occi :: core) Resource (from occi :: core) -summary :String[0..1] StorageLink Storage NetworkInterface Network Compute cd: Infrastructure target + source * links Fig. 2.3 Extended OCCI core model showing infrastructure elements It was the infrastructure model, along with the OCCI core and HTTP rendering specifications, that aided a successful collaboration that investigated the integration of two large European Union FP7 research projects, SLA@SOI3 and RESERVOIR.4 The proposed integration was detailed in a subsequent technical paper [13]. 2.2.5 Related Standards and Specifications A guiding principle in OCCI is to make use of existing standards and specifications where appropriate. OCCI and the Storage Networking Industry Association’s (SNIA)5 Cloud Data Management Interface (CDMI) working groups have collaborated together so that both specifications are interoperable with each other. It states that “The SNIA Cloud Data Management Interface (CDMI) is the functional interface that applications will use to create, retrieve, update and delete data elements from the cloud. As part of this interface the client will be able to discover the capabilities of the cloud storage offering and use this interface to manage containers and the data that is placed in them. In addition, meta-data can be set on containers and their contained data elements through this interface” [16]. 3 http://www.sla-at-soi.eu/. 4 http://www.reservoir-fp7.eu/. 5 http://www.snia.org/.
- 48. 2 OCCI for Data Management 33 OCCI and the Distributed Management Task Force’s (DMTF)6 Open Virtual- ization Format (OVF), see [4], can be easily integrated through the use of the resource-type Link. Where a provider wishes to supply an OVF representation of a client’s resource instance(s), they can do so by associating the instance(s) with a mirror representation, only the serialisation format is OVF. Other than this, the OCCI working group is closely working together with other groups inside of the Open Grid Forum. The Distributed Computing Infrastucture Federation (DCI-fed7 ) working group focuses on the creation of models and APIs for setting up distributed federated computing environments. Other than this, the OCCI working group uses Standards like those developed by the Distributed Resource Management Application API (DRMAA8 ) working group for common Job operations on Clusters via the OCCI protocol. 2.3 SLA Assured Provisioning of Database Services Using OCCI In today’s service marketplaces including cloud-based ones, there exist basic limitations in service offerings. Typically, the customer has little say in what is offered by a service provider and is left with a “take it or leave it” situation. Not only is the customer faced with such a dilemma, with little possibility of negotiation but if they do accept the service offering there is little in the way of service transparency and so detections of service violations are impossible unless that customer implements custom violation detection systems. The SLA@SOI project seeks to address these challenges by providing three major benefits: Predictability and Dependability: The quality characteristics of service can be predicted and enforced at run-time. Transparent SLA Management: SLAs defining the exact conditions under which services are provided/consumed can be transparently managed across the whole business and IT stack. Automation: The whole process of negotiating SLAs and provisioning, delivery and monitoring of services will be automated allowing for highly dynamic and scalable service consumption. In this section, a use-case that combines the OCCI model and API with an SLA management framework to provide an SLA assured database service is described. In today’s service marketplace, there exists a number of service providers who offer database services, for example, the Amazon Relational Database Service,9 6 http://www.dmtf.org. 7 http://forge.gridforum.org/sf/projects/dcifed-wg. 8 http://www.drmaa.org. 9 http://aws.amazon.com/rds.
- 49. 34 A. Edmonds et al. Microsoft SQL Azure10 and Longjump Platform as a Service.11 Other than offering a basic, non-negotiable, non-machine readable SLA, these service providers do not offer certain guarantees that particular consumers will require. A case in point is where a third party service provider wishes to process personal and identifying information. Many law jurisdictions will require that user-supplied data and the processed resultant data remain within the protection of that jurisdiction, which may mean that the physical location of that data must always remain in the country or region where that jurisdiction has powers to protect. If that data at any one time falls outside of those defined physical locations due to actions taken by the service provider that the third party uses, then regardless of knowing or not knowing about such actions, the third party can be liable under the relevant laws set out by the jurisdiction. In the use case presented here, an SLA management framework provides the means to: 1. Customise a service offering. 2. Negotiate on that service offering to the satisfaction of the third party and their legal responsibilities. 3. Be notified when terms of the agreed service offering deviate and have deviations logged as an audit trail. The use case is realised by the third party provisioning the offered database ser- vice using the OCCI API through the facilities of the SLA manager. OCCI provides the standard and interoperable means of provisioning the required database service and the SLA manager provides the means as outlined above. That database service is realised as a pre-built virtual machine with all the requisite database software installed and configured, which once provisioned is accessible by the consumer. The service provider offers means to monitor the agreed terms in the SLA and, in particular for this use case, allows for the physical location of the virtual machine to be monitored. This allows the SLA management framework to monitor constantly the physical location of the virtual machine and in the case that the virtual machine is migrated to an inappropriate physical location the third party will instantly receive notification of that event and logs will provide an audit trail. 2.3.1 SLA@SOI SLA Management Framework SLA@SOI defines a holistic view for the management of SLAs and implements an SLA management framework that can be easily integrated into a service-oriented infrastructure (SOI), see [17]. The main innovative features of the project are: • An automated e-contracting framework • Systematic grounding of SLAs from the business level down to the infrastructure 10 http://www.microsoft.com/en-us/sqlazure. 11 http://www.longjump.com.
- 50. 2 OCCI for Data Management 35 • Exploitation of virtualisation technologies at infrastructure level for SLA enforcement • Advanced engineering methodologies for creation of predictable and manageable services The accompanying diagram (Fig. 2.4) illustrates the anticipated SLA manage- ment activities throughout the business/IT stack. Customer Service Provider Business Use Procurement Business Assessment SLA (Re-)Negotiation Monitoring / Arbritration Business Assessment SLA Orchestration / Transformation / Aggregation Resource Consumption Forecasting Monitoring Adjustment Alerting Service Demand SLA Service Demand Forecasting Infrastructure Provider Provisioning Mapping Software Provider Virtual Physical SOI Contracting / Sales SOA Fig. 2.4 SLA@SOI overview
- 51. 36 A. Edmonds et al. 2.3.2 SLA@SOI and OCCI In this use case, there are two main components that are required for realisation. The first and most fundamental is a Service Manager that offers a database service. The Service Manager is the entity that is responsible for providing the client’s service. Relevant to this use case is that the Service Manager provides database services as preconfigured virtual machines and that the location of those virtual machines can be monitored. The SLA@SOI framework makes no assumption on this Service Manager only that it has an interface: 1. That can create, retrieve, update and delete its managed services. 2. Through which service instance metrics can be listed and retrieved. The second entity required is the SLA@SOI framework’s SLA Manager. This is a set of both generic and domain-specific components. What is generic relates to the management of SLA templates (what a provider offers) and SLA instances (what a provider runs on their clients behalf and guarantees). The domain-specific components are those that interact with the particular service manager that provides the client services. Further details of the SLA@SOI framework and its architecture can be found, see [18]. The SLA Manager offers to clients one or more SLA Templates, which is expressed using the SLA@SOI SLA model. Through either a UI or API, the client can select, customise, negotiate and provision an SLA-guaranteed service. In the use case scenario, this would entail the third party specifying what physical location (e.g. region, country) is required for their regulatory compliance. Once the SLA Manager is acting on the client’s behalf, it first negotiates with the service manager using the OCCI query interface. The OCCI query interface allows for the various Resource types to be queried for and interrogated and in particular to this use case, the locations that a provider can provision their virtual machines. As an extension to the query interface, SLA@SOI will also allow for per-user quotas to be queried. Using this extension, an SLA Manager can tell whether a client’s request will be fully satisfied or not by the current service provider. Having established that the client’s quota is sufficient, the next step can either take two paths. The first is that the provisioning of the requested service is done automatically or, second, the provisioning must be explicitly executed by the client. Where a provisioning request is executed in one or the other manner, the next responsibility of the SLA Manager will be to call the provisioning functionality of the Service Manager (relationship and interaction is shown in Fig. 2.5). This again is looked after by the OCCI API and an OCCI request from the SLA manager’s domain specific components is sent to the Service Manager. As soon as the provisioning request is successful, the SLA manager then begins to monitor the provisioned service, including the location of the service’s virtual machine. It does this by monitoring the various terms of the agreed SLA (e.g. QoS metrics). For the SLA Manager, the major advantage of choosing to implement OCCI as a means to talk with Service Managers is that in the case where a particular
- 52. 2 OCCI for Data Management 37 Client SLA Manager Service Manger UI API negotiate provision manage Fig. 2.5 Relationship between SLA manager and service manager Service Manager cannot satisfy the provisioning of the requested service resources due to insufficient client quotas or unsuitable virtual machine deployment locations, the SLA Manager can, with the necessary logic implemented, look up the next registered service provider and seek to have the remaining service resources provisioned there, without any need for Service Manager protocol or API changes. Such functionality makes an excellent case for SLA-mediated cloud brokerage use cases. 2.4 On-Demand Data-Aware Provisioning of Services A different application area for OCCI appears in the context of traditional community-based Distributed Computing Infrastructures (DCIs): modern research more and more relies on cross-institutional, cross-project data processing. In many communities, scientists quickly state the requirement to enable exchange of information beyond traditional boundaries such as project collaborations or long- term Virtual Organisations. Rather than that, a more flexible, more agile approach is expected. This development poses a major challenge not only for the management of data itself (i.e. ensuring authentication and authorisation, planning distribution and replication, and tracking provenance), but also for the management of its computation: workload needs to run close to the data in most cases (since data is usually large), but the compute infrastructure available in the direct proximity of the data may not necessarily provide the correct environment. That is, applications to process the data might be missing, the operating system does not match the application requirements, or – on a higher level – certain services needed for data analysis and manipulation have not been deployed on-site. Beside, many communities run their own, proprietary workload management software, tailored to the specific needs of their users. As such, it is usually not an option to require a central system, often referred to as a meta scheduler12 for all users of all communities. Rather than that, additional technology needs to be incorporated, which allows dynamic federation of planning domains depending on the current demand. 12 Mostly found in the context of traditional Grid Computing.
- 53. 38 A. Edmonds et al. The D-Grid Scheduler Interoperability project (DGSI) in the context of the German D-Grid Initiative13 aims to provide a solution both issues through the devel- opment of a standards-based protocol between Meta schedulers. DGSI approaches interoperability DCIs from two sides, namely Activity Delegation (taking care of the handover of workload from one domain to the other), and Resource Delegation (taking care of the leasing of resources from one domain to another). Assuming the DGSI protocols and services in place, the notion of delegation can help to avoid traditional data management techniques such as decoupled copying, prefetching, and replication at all. 2.4.1 The Climate Community Use Case The effects of climate change are one of the major challenges of mankind: stakeholders of many areas strive for strategies to deal with the consequences of pollution and man-made changes to the environment. The basis of all decision making are models of climate processes and the understanding of interplay of the enormous amount of parameters in them. Since the beginning of industrialisation in the nineteenth century, Earth System Science, one of the data sciences, investigates these processes, their chemical formation in the diverse subsystems such as oceans, atmosphere or biosphere, and their long-term influence on climate. From those investigations, researchers nowadays possess very detailed insight into climate development. This rests on the permanent acquisition, cataloging, and processing of very large (Peta scale) volumes of experimental and model data, as well as the continuous re-evaluation of scientific results using refined models. Current information technology provides potent means to accelerate these processes of data evaluation and simulation. High Performance Computing (HPC) infrastructure, high speed networks, and modern storage architectures support archival, preprocessing, selection, and transportation of large data amounts as well as the computation of highly demanding simulations (e.g. short term weather forecast or storm track analysis). For the latter scientific analysis, researchers filter and examine geopotential heights to track and predict the movement of low-pressure areas over time with regard to a given climate model. This is essential as storms and cyclones typically cross such areas [3]. This analysis and simulation is based on long-term acquired global climate data. Usually, scientists are only interested in a restricted area for a Stormtrack analysis and have to reduce the amount of available base data to the region of interest. Besides a complex combination of several steps, this resorts to either access to a specific amount of climate data (Fig. 2.6a) or execute simple visualisation workflows on selected and preprocessed data (Fig. 2.6b). 13 http://www.d-grid.de.
- 54. 2 OCCI for Data Management 39 Staging a b c Staging Visualization Staging Preproc. Staging Preproc. Simulation Fig. 2.6 Three simple workflow examples from C3Grid: (a) selective data download; (b) simple simulation on input data; (c) simulation on a set of preprocessed input data Today, they generally have two possibilities to retrieve the desired data: they either get access to the storage and download the full amount of data (i.e. full replication is performed) or use proprietary programs to reduce the amount of data at the storage site and download the desired data set afterwards. In the first case, the required local storage may simply not be available to the single scientist, or the providing institution may not be willing to provide an external party with access to the full archive due to strategical considerations. In the second case, the user needs to cooperate directly with the data provider, basically via two mechanisms: 1. Having to use tools that are installed, but potentially not known to him,14 or 2. Having to roll out the software on her own, either doing this as part of the batch processing job or in cooperation with the resource provider. Obviously, the former is not acceptable from a user’s perspective. The latter in turn requires extensive manual intervention and additionally necessitates the acquisition of user rights to retrieve or even locally process the requested data. As most of the climate data is stored in a distributed way, the procedure often has to be repeated for several data sites. Furthermore, it leaves intelligent, automatic load balancing totally to the user, which is generally not desirable. In addition to that, this traditional approach of application deployment massively hinders cross-community collaboration, if they rely on different infrastructure technologies: if the user takes care by himself to deploy the application as part of his computational workload, the number of resources compatible will usually be very restricted. 14 With the exception of widely accepted and distributed tools such as the Climate Data Operators [15].
- 55. Other documents randomly have different content
- 56. to pry into their doings. The night being clear she found her way out of the scrambly lane, leading up from Boslow to the highroad, scampered on as fast as she could, and never stopped till she reached the top of Dry Carn. There she sat down a minute, that she might recover her breath, to pass quickly over the road near Carn Kenidzek and down the Gump, as everybody then (as now) dreaded that haunted track; indeed, few go near that wisht place, about the turn of night, without hearing, if not seeing, the Old One and his hounds, hunting among the rocks for any restless spirits that might have strayed so far away from the churchyard—their only place of safety—or some other frightful apparitions, fighting and howling round the carn, or fleeing over the downs. She 'jailed' away—down the hill, as fast as she could lay foot to ground, thinking to be home by the kitchen fire in a quarter of an hour, and went far enough, as she thought, to have reached Pendeen gate twice over. Then she feared that she might have got into a wrong bridle-path over the downs, or that Piskey was playing her a trick, because, turn whichever way she would, the road appeared to be before her. After going on for a long while, she saw light and heard music, at no great distance. Thinking then that she must have kept too much on her left and be near some house on the road to Church-town, where they were getting in tune for the dancing on Feasten Monday night, she went over the downs, straight towards the light, feeling ready for a jig, and stopped more than once to 'try her steps,' as the lively old dancing tunes kept sounding in her ears. But, instead of arriving at a house, as she expected to, in passing round some high rocks, which hid the light a moment, she came, all at once, on a level green, surrounded by furze and rocks, and there, a few yards before her, saw troops of 'small people' holding a fair, or belike it might be their feasten market. Scores of little standings all in a row, were covered with trinkets, such as knee and shoe buckles of silver and gold, glistening with
- 57. Cornish diamonds; pins, with jewelled heads; brooches, rings, bracelets, and strings of crystal beads, figured with green and red, or blue and gold; and scores of other pretty things quite new to An' Pee; who, not to disturb the small folks till she had seen all that was doing, crept along softly in the rear of the standings, till she stood opposite a company of dancers; hundreds of them linked hand in hand, after the old bonfire-dance fashion, were whirling round so fast that it made her head light to look at them. Small as they were—none more than two feet high, and rather slender in make—they were all decked out like old-fashioned gentry —the little men in three-cocked hats and feathers; full, square- skirted, blue coats, stiff with buckram and gay with lace and buttons; vest, breeches, and stockings of a lighter hue; and their dainty little shoes fastened with diamond clasps. Some few, who were rigged more like soldiers or huntsmen, wore either jet-black or russet-coloured riding boots. An' Pee said that she couldn't name the colours of the little ladies' dresses, which were of all the hues of summer's blossoms. The vain little things, to make themselves look the taller, had their powdered hair turned up on pads and dressed with flowers, lace, and ribbons to an extraordinary height for such dolls of things. Their gay gowns were very long-waisted, and their skirts so distended by hoops that they looked just as broad as they were long. Their shoes of velvet or satin, were high-heeled and pointed at the toes. The men were much darker complexioned than the women, yet they were all very good looking, with sparkling dark eyes, well-shaped noses, sweet little mouths, and dimpled cheeks and chins. Not one among them, that she saw, had a spotty face or purple-top nose, because they drink nothing stronger than honey-dew. Some, to be sure, appeared to be rather aged, yet, all were sprightly, merry, and gay. In the dancers' ring stood a May-pole about three yards high, all wreathed with flowers. Where they got them, that time of the year, to make their garlands, was a wonder. The pipers, standing in their midst, played lively old dance tunes that are now but seldom heard,
- 58. and An' Pee never felt more inclined for a dance in her life than when she heard their cheery music; but how could she reel round among such little beings and have a jig without kicking them down? 'The women,' she always said, 'were the sauciest little creatures that one ever seed; she was most ashamed to look at them—tossing up their heels, forwards and backwards, higher than their heads, and kicking off the men's hats, as they capered round and round.' Every now and then, one would unlock her hands and, breaking out of the ring, take a leap right over the men's heads, perch on the May-pole, and there spin round, on her toe, like a whirligig. There were lights about in all directions—lanthorns no bigger than gun-pop (fox-glove) flowers, hanging in rows along the standings, and rushlights, in paper cups like tulips, shone among the gingerbread-nuts, comfits, candied angelica, peppermint-drops, and more enticing things that are seen in any other fair. She thought, too, that all the glow-worms in creation had gathered together near the fair-ground, to help to light it up. Yet, with all these lights, there was such a shimmer over everything that the old dame got bewildered at times and could never see anything so plainly as she wished. At no great distance from the dancers there was a wrestling ring, where many little ladies were looking on, betting on their favourites and helping them with their good wishes and applause. Farther on, some were shooting with bows and arrows at a target. Others were playing at keals (bowls). Every here and there the lilly-bangers (raffle-keepers) with their tables and dice kept a great noise calling out, 'Come hither, sweet ladies and gentlemen, and try your luck! One in, two in, three in; who will make four in for this nice cake?' Farther off, nearly out of sight, a great number were 'hurling to the gold' (goal). She knew what was going on from hearing the old cry of 'Well done, Santusters, one and all, comrades; fair play is good play' and, every now and then she saw the little hurling-ball, as it was cast from side to side, shine like a shooting star. By that means they contrived to hurl by night.
- 59. All games, which used to be played at fairs and merry-makings, were there carried on. Still, great part of the small folks diverted themselves in parading up and down, on the green, between the standings and dancing-ground, examining the pretty things displayed. They didn't seem to have any money amongst them to buy anything, yet they often bartered their trinkets and changed them from stall to stall. The old woman determined to have some of the pretty things glistening before her, but, among so much that was beautiful, she couldn't make up her mind what to take. Whilst An' Pee was considering, she saw approach the standing a little lady, tired with dancing, leaning her head on her partner, who with his arm round her waist supported her steps. The gentleman taking from the hands of a little dame who kept the stall a golden goblet of the size and shape of a poppy head (capsule) held it to the faint lady's lips. Sipping the contents she recovered in an instant, and, choosing a fan, made of a few goldfinch feathers stuck into a pearl handle, her partner took a pair of diamond buckles from his knees and placed them on the standing by way of pledge. The little couple having tripped off again to the dance, An' Pee thought how well the bright little buckles would look, fixed as brooches, on her Sunday's cap- ribbon or in her neckatee, and determined to secure them at once, fearing they might be gone with the next small body that saw them. As there was nothing that she could so readily turn inside-out, and drop on them, as one of her gloves, which reached to her elbows, she drew off one, inside-out, and dropped it, as it seemed to her, right on the buckles. Her hand nearly touched them; but, in trying to grasp them under her glove, a palm of pins or needles, so small that she didn't notice them, stuck into her fingers, and she cried out, 'Oh! Cuss 'e! You little buccas.' That instant all the lights went out, and all the fair, and most of the small people, vanished like shadows among the rocks or sunk into the earth, like muryans (ants) into their holes. Yet many of the frolicsome sprights were still about her, as she soon found to her cost.
- 60. Whilst she was still stooping, and groping for her glove and the buckles, she felt a great number of the small tribe—a score or more —leap on her back, neck, and head. At the same time others, tripping up her heels, laid her flat on the ground and rolled her over and over. More than once, when her face was uppermost, she caught a glimpse of Piskey, all in rags as usual, mounted on a year- old colt, his toes stuck in the mane, holding a rush in his hand to guide it. There he sat, putting on the smaller sprights to torment her, making a tee-hee-hee and haw-haw-haw, with his mouth open from ear to ear. When she spread out her arms and squeezed herself down, that they shouldn't turn her over, they would squeak and grunt in trying to lift her; but all her endeavours to hinder their game were of no use. Somehow or other over she went, and every time they turned her face downwards some of the small fry would jump on her back and there jig away with 'heel-and-toe' from her head to her feet. In the pitch and pass of their three-handed reels, it was who and who should get on her stays; the steel and whalebone in that, she supposed, served them as a springing-board. In the finishing off of their double shuffles they would leap more than three times their height, turn a summersault over each others' heads, and so make the pass. An' Pee twisted her head on one side, saw what they were at, and tried to beat them off with her stick, but they got it from her hand, laid it across her waist, and mounting on it astride, as many as could, bobbed up and down, singing, 'See-saw-see, Lie still, old Peepan Pee. See-saw-see, Upon old Peepan Pee, Who should better ride than we? See-saw-see.' The old woman, not to be beaten with such imps, tossed back her feet to kick them off; then they held her legs doubled back and pulled off her shoes; some jumped up and balanced themselves on
- 61. her upturned toes, whilst others pricked at, and tickled, the soles of her feet till she fell into fits of crying and laughing by turns. Pee was almost mad with their torment, when, by good luck, she remembered to have heard that the adder-charm was powerful to drive away all mischievous sprights. She had no sooner pronounced the words than they all fled screeching down the hill, Piskey galloping after; they left her lying on a bed of furze, near a large rock. She got on her feet, and, looking round, saw, by the starlight of a clear frosty morning, that the place to which she had been piskey- led was near the bottom of the Gump; that the level spot of green on which the small people held their fair, and carried on their games, was almost surrounded by high rocks, and was no larger over than the Green-court or walled garden in front of Pendeen house; yet, when the fair was on it, through the sprights' illusions, this green spot seemed like a three-acre field. An' Pee only found her stick. The basket, tied to her arm, was empty and broken to pieces. She paced the ground over and round, in hope of finding her hat and shoes, and above all her glove, and the precious buckles under it. Giving over at length her fruitless search, with the help of her stick she hobbled, barefooted and bare- headed, down the hill and reached Pendeen gate. 'Now thank the powers,' said she, as she passed through it and slammed it behind her, 'I shall be a-bed and sleepan in a few minutes.' Though An' Pee knew that Piskey had played her many tricks that night, and she thought he might be still dogging her footsteps, yet she was so bewildered that, until too late, it never came into her head to turn some of her clothing inside out, and now, so near home, she defied him to lead her astray. Inside Pendeen gate there is one road leading to the mansion and another which goes down to the mill. Between them there were two
- 62. or three acres of ground, which had probably never been cleared or cultivated, as there were several large rocks remaining on it and brakes of furze, seldom cut, because the old Squire, or his family, had stocked this piece of rough ground with fancy breeds of tame rabbits, and the wild ones which came among them from not being chased or shot at, became so tame that they continued their frisky gambols, without showing any signs of fear when persons passed near them; and, for the pleasure of seeing the bunnies sport, furze was allowed to grow here and there over great part of this ground. In passing to the house An' Pee avoided the stony road and walked on the green, because her poor bare feet were cut and sore. Now hundreds of times—drunk and sober—on the darkest nights she had gone along the grass beside the bridle-paths, without once missing her way to the Green-court gate. Yet, that Hallan Eve she, somehow, went too far from the road, got in on the grassy patches between the furze, and, before she knew that she had missed her way, found herself down by the mill-road. She followed up that track, and in making a new attempt to reach the house, she again got among the furze and wandered about on the patches of green between them for hours without coming to either road. Yet, as usual, with piskey-led persons the path appeared either before or close beside her, until, tired out, she lay down to wait for day and fell asleep. The Squire and all his household were very much concerned because of the old woman's absence, well knowing that no ordinary matter would keep her from home on the feasten tide. During the night the servants had been sent to the villages round, to inquire if anyone had seen her in Penzance or on the road, but no tidings were obtained of her. The Squire rose by break of day and called up his servants to hunt for her. In passing along the road towards the gate, only a few yards from the house, he heard somebody snoring in a brake of furze bordering on the path, and there he found his housekeeper very ragged and torn. Some say he discovered her by finding on the road her knitting-work, with the yarn hanging to it,
- 63. and, by taking up the yarn, he went by it till he found the dame with some of the ball in her pocket. However that may be, he roused her with great difficulty, and, without opening her eyes, she said, 'I wan't turn out to please anybody till I've had my morning nap; so go away, go, and shut my chamber door!' At length her master, having brought her to her senses, helped her up and asked what made her take up her lodgings on the cold ground? In passing slowly along, and stopping awhile at the Green-court gate, she told him of her mishaps. The Squire didn't think one half of what she said could be true; indeed he questioned whether she had been to Penzance at all, and thought it quite as likely that she had stayed tippling at the cove till near dark, starting for town, had missed her way, and, wandering over the Gump, had there, or where he found her, fallen asleep and dreamt great part of what she told him. 'Belike Pee,' said the Squire, as she was about to go down the Green-court steps, 'what you took at the cove had something to do with rising the spirits you saw.' 'Oh! you misbelieving man,' cried she, turning round, and holding towards him her uplifted hands, 'if I like a drop of good liquor to cheer my heart, now and then, I never took so much as to do me harm in all my born days; and, leave me tell 'e, that with all your learning, and doubting, you know but little about the 'small people.' There es more taking place in the region of spirits, as I've heard the parson say, than you can learn from your books, and for want of faith, I fear me you will never be enlightened. Yet as sure as my name is Penelope Tregeer, I seed, heard, and what is more I felt, all that I now tell 'e.' 'Go in and sleep the spirits out of thy noddle, that thou mayest be in time to see about the feasten dinner,' said the Squire, as he
- 64. turned away, and took his favourite morning's walk to the cove. When he came in, after a turn round the cliff and up by the mill, he found the old woman, never the worse for her journey, busy preparing the feasten fare, and the ladies and gentlemen of his family, and numerous visitors, at an early breakfast that they might have time to proceed to church in grand state on the feasten day. Pendeen of Old. Capt. Peter, having taken a pull from the pewter pot, continued with —Believe me, comrades, Pendeen didn't then look wisht at feasten tides nor at any other time, when one saw, (and smelt, too), the sweet scent of turf-smoke that curled up from chimney stacks, which now look down sorrowfully on cold hearths; and one saw fair faces peering through the casements, numbers of ladies and gentlemen walking about the garden alleys and courts of the old mansion, or when the cry of hounds and the winding of the horn echoeing through the house, called one and all to the hunt at early morn. And, I can but think, he continued, how strangers visiting Pendeen for the first time, after riding over miles of open downs with scarce a dwelling in sight, must have been surprised when they caught the first glimpse of the noble old seat, which is only seen when close at hand, and the track of rich cultivated land between it and the sea; it must have appeared to them like a place raised by enchantment, as we hear of in old stories. And the old masons, who took pride in their art and did their work truly, were right to bestow such labour on the beautiful chimney stacks of the old mansion, because they are there first seen, and from parts where little else of the house is visible; and the first sight, like first love, is never forgotten, mates. Capt. Peter paused, drained the pewter pot, which had stood foaming before him, handed it to the cheerful old landlady to be
- 65. replenished, and took a smoke. A tinner, who sat by the fire knocking the ashes out of his pipe, said, whilst he cut up his roll- tobacco, rubbed it in the palm of his hand, and refilled:— I don't understand very well Capen what is meant by enchantment, only that it's something strange and wonderful. Now, to my mind, the greatest wonder about the place is the Vow. One end of it we know is within a few yards of the mansion, but no one knows where the other is to be found. Ef there be any truth in old traditions about that cavern, adit, fougou, or whatever it may be called, it runs for a great distance (some say miles), yet most people believe that the eastern end was once open at the cove. Others will have it that old tinners, who lived before part of the roof had fallen in, travelled in it for ten times the distance from the house to the cove, and burned more than a pound of candles without finding the end. They always returned frightened, and what they saw to scare them they could never be got to tell. Perhaps the Spirit of the Vow, that many have seen at the entrance, in the appearance of a tall lady, dressed in white, with a red rose in her mouth, at all seasons of the year, may take a more fearful form within the cavern. Who can tell, he continued, but that money and treasures may have been secreted there in troublesome times of old, and I wonder why the Squire don't have the mystery about the Vow cleared up; there can't be much of the roof fallen in, and, for my part, I'd willingly give all my time, out of core for a month to help clear away the rubbish and take the venture upon shares. I am very much of thy mind, my dear, Capt. Peter replied, Ef the Squire would give us leave we'd pitch cost as soon as the feast is over, and I don't think we should find there many spirits to frighten us away. I believe that many of the fearful stories about the Vow were invented by smugglers. When the fair trade was in its glory the Vow was a convenient place for storage, and I think that the smugglers, who didn't want any faint hearts, with weak heads and
- 66. long tongues, to come near them, invented many fearful stories to scare such away. One never finds any so fond of prying into other people's business as the foolish ones, or 'Grammer's weak children,' as we say. How Piskey Left Boslow.
- 67. Tells how the drudging goblin sweat To earn his cream bowl duly set, When in one night, ere glimpse of morn, His shadowy flail hath thresh'd the corn. Milton. No doubt, said the tinner after a pause, Piskey threshed the corn and did other odd jobs for the old man of Boslow, as long as he lived, and they said that after his death he worked some time for the old widow, till he took his departure from the place about three score years ago. Some say— Stop a minute, my son, I can tell 'e a story about that, said Capt. Peter, taking the pipe from his mouth, and holding up his finger: —One night, when the hills were covered with snow and winter had come severely, the old widow of Boslow left in the barn for Piskey a larger bowl than usual of gerty milk (boiled milk, thickened with pillas, or oatmeal). Being clear moonlight she took a turn round the town-place, stopped at the barn-door, and looked in to see if Piskey were come to eat his supper while it was hot. The moonlight shone through a little window right on the barn-boards, and there, sitting on a sheaf of oats, she saw Piskey eating his gerty milk very hearty. He soon emptied his wooden bowl, and scraped it with the wooden spoon as clean as if it had been washed out. Having placed the 'temberan dish and spoon' in a corner, he stood up and patted and stroked his stomach, and smacked his lips in a way that was as much as to say, 'that's good of 'e old dear; see ef I don't thresh well for 'e to-night.' But when Piskey turned round, the old woman was sorry to see that he had nothing on but rags and a very little of them. 'How poor Piskey must suffer with the cold,' she thought and said to herself, 'to pass great part of his time out among the rushes in the boggy moors or on the downs with this weather—his legs all naked, and a very holey breeches. I'll pitch about it at once, and
- 68. make the poor fellow a good warm suit of home-spun. We all know ragged as Piskey es, he's so proud that he won't wear cast-off clothes, or else he should have some of my dear old man's—the Lord rest him.' No sooner thought than she begun; and, in a day or two, made a coat and breeches, knitted a pair of long sheep's-black stockings, with garters, and a nightcap, knitted too. When night came the old woman placed Piskey's new clothes, and a bowl of gerty milk on the barn-boards, where the moonlight would shine on them to show them best. A few minutes after leaving the barn she came back to the door, opened its upper part a little, and, looking in, saw Piskey standing up, eating his milk, and squinting at the clothes at the same time. Laying down his empty bowl he took the new breeches on the tip of his hand-staff, carried it to the window, and seeing what it was, put it on over his rags, dragged on the stockings, and gartered them, donned coat and cap, then jumped over the barn-boards, and capered round the barn, like a fellow light in the head, singing, 'Piskey fine and Piskey gay, Piskey now will run away.' And, sure enow, run away he did; for when he came round to the door opening into the mowhay he bolted out and took himself off without as much as saying, 'I wish 'e well 'till I see again' to the old woman, who stood outside the other door looking at am. Piskey never came back and the old woman of Boslow died that winter.
- 69. An Overseer and a Parish Clerk of St. Just about sixty years ago. It was no wonder if persons coming from Penzance to Pendeen of a dark night should miss their way and think themselves piskey-led, said the tinner. There was neither bridge nor house in the place called New Bridge before wheel carriages were in use, and the only St. Just road from Penzance this side of Cardew Water was a mere bridle-path or rather a great number of horse tracks, often crossing each other and twisting about far and wide round rocks and intervening patches of furze, over miles of open downs and boggy moors, with no hedges near the road to keep it within bounds. When one track was worn too deep it was never repaired, as there was plenty of room to make a new one. Bridges then were few, and for the most part made by placing flat slabs to rest on the stepping-stones in some of the deepest streams, for the convenience of foot passengers. These old foot-bridges were ugly things to cross by night and the stepping- stones were worse. We have all heard about the old stepping-stones in Nancherrow Water, said the tinner, who finished the foregoing story, how, after day-down, no one could pass over them in going to Church-town without some mishap, and no person would venture to return that way until daybreak. Shortly before the first bridge was built there, one of the overseers was a farmer who lived in the North of St. Just. Few persons then could either write or read, except one here and
- 70. there, who passed for a great scholar if he could sign his name and read a chapter in the Psalter without much spelling. The overseer not knowing how to write or cipher, kept the accounts of his monthly disbursements on the dairy-door, in round o's for shillings and long chalks for pence. The last Saturday of each month he took the dairy- door on his back and carried it to Church-town, that the clerk might enter his accounts in the parish book. One Saturday, in the season when days are short and streams high, the overseer couldn't make out his accounts and reach Nancherrow Water before dark; and, in passing, with the door on his back, over the wet and slippery stones, he lost his balance, and fell into the stream. By good luck the door was under, and floated him down to a place where the water spread out shallow and there he landed, but all the accounts were washed out. 'Tis said that the overseer's mishap was the reason why the first bridge was built over Nancherrow Water. I can tell 'e another sad case, said the Capt. We elderly folks have all heard of Uncle Will Ben, who was the parish clerk and the best fiddler in the parish, a little before I was born, and everybody says he was what we call a 'peathy old fellow, with plenty of gumption.' One Feasten Monday Uncle Will was rather late in going to Church- town with his fiddle, in a case, under his arm, to play during the night in a public house. Being Feasten Monday, like enough he had stopped to take a drop at neighbours' houses on the road; however, in crossing Nancherrow Water, his foot slipped from the stepping- stones and his fiddle fell from under his arm into the water, floated down the stream and in under a high bank where it was caught in some brambles. A gentleman riding through the water, saw Uncle Will a little below trying to get at something with his stick, and asked what was the matter. Uncle Will told him of his mishap. 'I pity your case,' the horseman replied, and rode on. 'I don't care a cuss for the case if I'd only got my fiddle,' replied Uncle Will.
- 71. This gave rise to the saying which is still often heard, 'I don't care a cuss for the case, if I'd only got the fiddle,' as Uncle Will Ben said. This old jewel of a parish clerk and fiddler said many other things which are still remembered and used as every-day sayings. It was the custom then for the great farmers to invite the parson and clerk to supper on goolthise (harvest-home) day, and the sexton usually came to work and see his reverend master safe home. Often all three came in time to lend a hand about the corn carrying. If two farmers had their goolthise on the same day the parson and sexton favoured one and the clerk the other. It happened, one day, when Uncle Will came alone early in the morning to help, and to enjoy the feast, that the weather was very lowering, and such was the fear of rain coming before the corn was in ricks, and thatched, that the carrying was continued all day for dear life, without stopping to take any other breakfast or dinner than such snacks as the corn carriers could catch, when there were more trusses round the ricks than the builders could put away for some time. The corn was then, except on a few large farms where ox-wains were just coming into use, all carried on the horses' backs, and the chasers, as they called the leaders who kept the trusses steady on the horses, were fond of coming in together that they might have a race back to the field, made the mowers work very irregular; it was gallop and stop half the time. That day, however, all worked with such a will that the corn was in and thatched in good time before the rain came. The supper being served, the clerk, in the absence of the parson, was asked to say grace. Uncle Will hesitated a moment; then, rising, he said, 'Thank God we have carried all the corn and had very fine weather; so here's grace for breakfast, dinner, and supper together.' Yet what is usually known as Uncle Will Ben's grace, is, 'God bless the meat and now let's eat!' Another saying accredited to Uncle Will—that 'Job had patience, but Job never had such a splat of black petates in his life'—is owing to
- 72. An' Mary, his wife, having been a parson's daughter from upwards, and 'brought up like a lady' as he was fond of saying sometimes. When Will was a young and smart militia man, and An' Mary a girl in her teens, he fell in love with her and she fell in love with him, and came with him to St. Just. In their time potatoes were just coming into use; gentlemen and some farmers planted a few in their gardens as a curious vegetable to be used on extraordinary occasions. Will Ben, not to be behind the fashion, had a small spot planted in his garden. When his potatoes were high enough for hoeing Will told his wife Mary, who kept the garden in order, to hoe the 'splat of petates,' and be sure to hoe them clean. When William came in from his work in the fields, he said, 'Well Mary, hast a hoed the petates?' 'Yes, William dear, and hoed them nice and clean; just go out and look at them whilst I take up the supper.' 'William dear' went into the garden, but he saw no potatoe-plants, for Mary had cut them all out of the ground, not knowing them from weeds. 'Dear William' came in swearing on his wife for hoeing up all the precious petates, telling her that it had been ten times better for him if he had wedded the sexton's dafter, as she would have made a better farmer's wife. An' Mary (who, as I have heard say, was always a dear gentle soul) only replied, 'Sweet William, have patience and they will grow again. Remember Job, William dear, and think, cheeld vean, how he had patience.' 'Oh! d——n Job,' replied sweet William, 'don't tell me about Job. Job never had such a splat of black petates in his life!' And now, my dears, said Capt. Peter, holding up a pot of foaming ale, here's health and luck to 'e all, my hearties, and a merry Feasten-tide to 'one and all.' There's no sense in being miserable, and, for my part, old as I am, I'd go ten miles this night to dance to the music of as good a fiddler and as honest a man as Uncle Will Ben.
- 74. The Fairy Master, or Bob o' the Carn. Out steps some Faery, with quick motion, And tells him wonders of some flowrie vale. Marston. UST fifty years ago, one Tom Treva lived on a small lone tenement near the foot of Carn Kenidjack hill. He had a large family and disliked for any of them to go in service. The boys, as they grew up, worked in the mines, and helped about the tillage of their few acres of crofts 'out of core.' The eldest daughter, Grace, remained at home to assist her mother, who took pride in making her handy in doing all such simple work as was required in their humble household. But as it was hard for them to make both ends meet, the poor girl had no best clothes except such as were made out of old gowns which had belonged to her grandmother. These were very gay, to be sure, yet so old-fashioned that other maidens, who worked at the mines and procured more modish dresses, wouldn't be seen anywhere from home with Grace and her grammer's old gowns. She didn't much mind their company, however. Her mother and 'the boys' (her brothers) promised her, year after year, that against the next Feasten-tide, if they could only lay by a few shillings, they would buy her as smart a rig-out as any of the proud hussies could show. But, with so many mouths to be fed, it was hard for them to save a farthing. So tides came and went, and Grace had nothing for bettermost wear that was fit to be seen in Church-town or anywhere else from home, so the bal- maidens said, and they wouldn't be seen going to preaching or to games with her; yet she didn't mind it much, and seemed
- 75. contented enough to stay at home, in the evenings listening to old stories related by her father and others who gathered round his hearth, because they, too, were not rich or smart enow to follow the fashions then upsetting all old customs among such Santusters as 'got a sturt to bal.' Grace would go about her work, indoors and out, singing like a lark. She was nearly sixteen, when a cousin of about her own age, who had been away only a year in farmer's service, a few miles off, came to see them the next Feast, dressed out quite like a lady, to Grace's seeming; for she wore a blue shining dress and earrings, and necklaces of red, green, and yellow beads that she changed more than once a day, or wore them altogether, while the flowers in her bonnet were the admiration of all beholders. I should be glad, cousin Grace, said she, to put thee up to Church-town to the fiddler a Monday night, and wish I had only brought home one of my frocks for thee to wear; but really, cheeld, grammer's old gowns would make thee a laughing-stock to the youngsters, and not one of them would dance with us. Go thee way'st in service, cheeld, that thee may'st get a stock of clothes fit to be seen in, and a sweetheart that thee west soon want to have as well as other maidens! But the Lord help thee and the young fellow who would come a courting and take thee to Morvah Fair even in that old rorey-torey gown, with red and blue flowers so large that the birds are nesting in them. Grace became very dissatisfied after this vision of grandeur, and never gave her mother any peace till she consented for her to go in service next summer. She was the more ready to let Grace try her fortune away, as other daughters were growing up to help her. So, during winter, she and her mother spun and knitted for dear life that they might earn a few extra shillings to provide changes of under clothing against she set out to look for service. For weeks Grace had been going round saying good-bye to the neighbours, and she rose one fine morning and gave the last kiss, and said, I wish 'e well, for the last time, to all the family round. Her father, on parting, charged her not to go more than a day's
- 76. journey from home, and be sure to keep far away from Penzance or any town, for fear she should be kidnapped, and they should nevermore see her. He told her how strange sailors, that frequented such places, often prowled about for miles, and no maiden was safe within their reach. Grace promised to be on her guard, took her fardel, and started on her journey towards the southern parishes where gentlemen farmers lived. On her way she thought upon what her smart cousin had told her to go over to the other side of the country, get into good farmer's service, where she might soon qualify herself to live in a gentleman's house and get higher wages. She had advised her not to pay much heed to what old folks said in their fears, about conjurors, witches, small people, and such like, that are seldom met with now-a-days. Up here amongst the hills you know but little of the world, said the cousin, and your old drolls arn't altogether to be believed. Grace couldn't help going out of her way a little to take a last look of Carn Kenidjack, where she had passed many happy hours, for youngsters were accustomed to meet there of Sunday afternoons to play about amongst the rocks or listen to old folks' stories. Then she went on with a pretty good heart till she reached high ground, from which she could only just see the smoke curling over the house-tops below. She turned round, took a farewell look, her eyes blinded with tears; then she went a little farther and sat down on a rock by the road- side, to have a good cry and ease her heart. She wept aloud to think she was going to an unknown country to live amongst strangers—that she might nevermore behold her parents and old playmates. But still, determined to go on, even if she went as far as daylight would take her, she dried her eyes with her apron; and, looking up, she saw standing close beside her a very nice-looking gentleman. He wished her good morrow and asked why she wept. Oh, sir, I have left home, she replied, and am on the road to a strange country to look for service.
- 77. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com