The Moving Code Project - Matthias Müller

Faculty of Environmental Sciences, Geoinformation Systems

The Moving Code Project –
Sharing Computing
Components on the Web

Matthias Müller

We like to move it, move it! – the "Moving Code" idea
A piece of implemented
computational logic:
- Well defined
- Tested
- Useful for others
- Robust
- … worth publishing

Workstation GIS

Web Service

Cloud machine

2

The roots of service-oriented processing …
 Think of a processing tool as a service („get me a
haircut“), but not necessarily a Web Service
 Services are well-defined
 Services are hopefully robust

 Having WPS process descriptions for processing
functions is a first major step to compare different
functions interface-wise
 Top-down approach: First agree on a convenient
process interface between service provider and
consumer, then start / adjust implementation

3

Use cases
 Process sensitive data (which cannot be transported
over a network due to legal reasons)
 Publish well-defined processes from scientific research
(imagine impact factor for well-defined, wellprogrammed, well-tested models!)
 Approach the scaling issue in a distributed environment
(it is not economic to re-implement new function in every
product / library)
 Sharing new processing functions in an interopeable
fashion without having to provide a WPS server

4

Save bandwidth – move processes close to the data

Data: 5GB

Coded process:
5 KB

5

Requirements for code exchange across systems

1. What functionality is provided?
2. What software environment is required to run the code?
3. What hardware is required for robust execution?
4. What are the legal conditions for use?
5. (What is a proper packaging mechanism for shipping?)
7

Multi-level description approach
 Adopted from service models in cloud environents
 SaaS – Software as a Service  Algorithmic functionality
 PaaS – Platform as a Service  Req‘d. software environment
 IaaS – Infrastructure as a Service  Req‘d. hardware resources

 Plus exploitation rights
 Related standards
SaaS

OGC WPS Standard: Process Descriptions

PaaS

(URI registry)

IaaS

OCCI (Open Cloud Computing Infrastructure Spec.)

Exploitation Rights

Creative Commons Modular license

8

1. Functional Contracts – SaaS
 WPS process descriptions can be used to describe
process intefaces (without actually using the WPS
protocol)






What are the parameters of a process
What are they for
Are they Inputs, wich or Outputs
Are they mandatory or optional (cardinality in general)
Are specific about the required data format (e.g. GeoJSON,
GeoTiff, …)

9

2. Contracted Platforms - PaaS
 There is a limited number of well known, coarse-grained
APIs in the Geospatial Domain







GDAL (which usually includes Numpy)
GRASS
ArcGIS
R
Matlab
…

 Probably 90% of the people will use these APIs to
produce their compuational logic
 Business models of PaaS providers can support predefined platform configurations
10

3. Contracted Infrastructure – IaaS
 Service-oriented operation means a robust execution of
a service under a broad range of conditions
 … is a prerequisite to operate on large data volumes
 Making algorithms fit for large amounts of data is not an
easy task but worth the effort
 Contracted infrastructure for code packages are the
expected hardware requirements for a single execution
 This also helps to check the feasibility of concurrent executions
on the same machine

11

Different implementations, same functional contracts

SaaS
Functionality
Interface

Process

PaaS
Platform B

Platform A
Interpreters
Dependencies

IaaS

Platform ...

#!/bin/bash

System 1

System 2

System 3

System 4

System ...

Architecture
Memory
Processor

12

4. Exploitation Rights
 Implementations will surely have associated usage rights
or lincensing conditions
 Creative Commons is a widely used standardized
approach – also for software licenses
… but there is also a need to support other types of open
source or commercial licenses
 Possibilities: Use structures CC approach to licensing –
or – reference „traditional“ license
 Moving Code currently uses URLs to reference license
documents but preferes CC
13

5. Packaging Mechanism – Workspaces and Containers
Structured ZIP-Archive
Package description (XML)

Workspace

A transferable
directory structure

Executable
A well-known container
format (.py, .jar, …)
Additional elements
(e.g. constant data, fine-grained
dependencies)

14

Towards a Web archive for self-contained, well-defined
processing functions

Geoprocessing Repository / „Appstore“
15

MovingCode Framework – Components
Process Repository /
„Appstore“
/ web-based toolbox

mc-schema
Descriptive schema
and packaging format

52N WPS Framework

mc-runtime
API for reading, writing
and executing MCpackages

Use a catalogue, download
and use directly …

MCPlugin

MCPlugin

<other products>

16

Delivery mechanism – Atom Feeds

 Can be subscribed to
 Human-readable
 Web-Browsers
 RSS-Readers /
Mailclients
 Machine-readable
content
17

Package Description: Schema

wps::ProcessDescription
(inherits from WPS specification)

mvc::ContractedFunctionality

implements

mvc::Workspace
+ workspaceRoot [1]
+ executableLocation [1]
+ containerType [1]
+ executionParameters [1..*]

liable to

...
(inherits from other standard)

complies

mvc::ContractedPlatform
+ runtimeComponent [1..*]

complies
mvc::ContractedInfrastructure
(inherits from OCCI specification)

mvc::ExploitationRights
+ CreativeCommonsLicense [0..1]
+ TraditionalLicenseDocument [0..1]

18

Package Creation: Do it top down
Contract-first: Start with the WPS interface description

Specify container format and required software platform
(we have tested: Any command line, JAVA, ArcToolbox)

Map the I/O parameters from the WPS description to the container‘s
interface

Specify hardware and license

Package and publish

19

Summary / Remarks / Further thoughts
 Relies on well-defined or „standardized“ environments
 Easy to require but hard to enforce
 Needs a common agreement on container types
(command line interfaces are come out-of-the box)
 Packaging scheme for reusable processing logic relies
as far as possible on open standards
 Sandbox testing of new packages (replicate runtime
environment)
 Generate visibility for scientific models and analysis
functions
20

The Moving Code Project - Matthias Müller

More Related Content

What's hot (20)

Similar to The Moving Code Project - Matthias Müller (20)

More from Luis_de_Sousa (7)

The Moving Code Project - Matthias Müller