Session 3-Distributed System Principals

Distributed System
Principles

Miron Livny
Center for High Throughput Computing (CHTC)
Computer Sciences Department
University of Wisconsin-Madison
miron@cs.wisc.edu

Principals
not

Technologies
(or buzz words)

www.cs.wisc.edu/~miron

Benefits to Science
› Democratization of Computing – “you do not
have to be a SUPER person to do SUPER
computing.” (accessibility)
› Speculative Science – “Since the resources
are there, lets run it and see what we get.”
(unbounded computing power)
› Function and/or data shipping – “Find the
image that has a red car in this 3 TB
collection.” (computational/data mobility)

Batch vs. Interactive
› We started with personal computers
› Then we had computers that were managed by
batch systems and supported multi-programming
› The demand for interactive computing led to time-
sharing systems
› Then we fell in love with personal computers and
got rid of (almost) all batch systems
› When we wanted to deliver on the promise of
computational grids we rediscovered batch
systems and Job Control Languages (JCL)

2,000 years ago we
had the words of
Koheleth
son of David king
in Jerusalem

The words of Koheleth son of David, king in
Jerusalem ….

Only that shall happen
Which has happened,
Only that occur
Which has occurred;
There is nothing new
Beneath the sun!

Ecclesiastes Chapter 1 verse 9


In other words …
Re-labeling, buzz words or hype do not
deliver as the problems we are facing
are fundamental and silver bullet
technologies are pipe-dreams.
This is true for computing in general
and distributed computing in particular


Data and Processing
Bringing the right data to the
processing unit and shipping the results
back to the consumer is a fundamental
(hard) problem. The properties of a
distributed system make data
placement harder.


Identity Management
Tracking the identity of the owner of a
request for service is a fundamental
(hard) problem, the de-centralized
nature of distributed computing makes
it harder.


Debugging
Trouble shooting software in a
production environment is a
fundamental (hard) problem, the
heterogeneous nature of a distributed
system makes it harder.


Unhappy customers …
› Can not find my job … it vanished
› My job is stuck for ever in the job queue …
› The input data was not there …
› Failed to ship the results back …
› The remote site refused to run my job …
› My job was supposed to run for 2 hours but
it ran for two days …
› I have no idea what went wrong with my job
…

In the words of the
CIO of Hartford Life
Resource: What do you expect to gain from
grid computing? What are your main goals?

Severino: Well number one was scalability. …

Second, we obviously wanted scalability
with stability. As we brought more servers
and desktops onto the grid we didn’t make
it any less stable by having a bigger
environment.
The third goal was cost savings. One of the most …

We do not have

performance problems

we have

functionality problems

35 years ago we had

Distributed
Processing
Systems

Claims for “benefits” provided by
Distributed Processing Systems
P.H. Enslow, “What is a Distributed Data Processing
System?” Computer, January 1978

e High Availability and Reliability
y High System Performance
o Ease of Modular and Incremental Growth
a Automatic Load and Resource Sharing
n Good Response to Temporary Overloads
Easy Expansion in Capacity and/or Function


One of the early computer networking designs, the ALOHA network
was created at the University of Hawaii in 1970 under the leadership
of Norman Abramson. Like the ARPANET group, the ALOHA network
was built with DARPA funding. Similar to the ARPANET group, the
ALOHA network was built to allow people in different locations to
access the main computer systems. But while the ARPANET used
leased phone lines, the ALOHA network used packet radio.
ALOHA was important because it used a shared medium for
transmission. This revealed the need for more modern contention
management schemes such as CSMA/CD, used by Ethernet. Unlike the
ARPANET where each node could only talk to a node on the other end,
in ALOHA everyone was using the same frequency. This meant that
some sort of system was needed to control who could talk at what
time. ALOHA's situation was similar to issues faced by modern
Ethernet (non-switched) and Wi-Fi networks.
This shared transmission medium system generated interest by
others. ALOHA's scheme was very simple. Because data was sent via a
teletype the data rate usually did not go beyond 80 characters per
second. When two stations tried to talk at the same time, both
transmissions were garbled. Then data had to be manually resent.
ALOHA did not solve this problem, but it sparked interest in others,
most significantly Bob Metcalfe and other researchers working at
Xerox PARC. This team went on to create the Ethernet protocol.

Definitional Criteria for a Distributed
Processing System
P.H. Enslow and T. G. Saponas “”Distributed and
Decentralized Control in Fully Distributed Processing
Systems” Technical Report, 1981

mMultiplicity of resources
r Component interconnection
o Unity of control
t System transparency
n Component autonomy

Multiplicity of resources
The system should provide a number
of assignable resources for any type
of service demand. The greater the
degree of replication of resources,
the better the ability of the system
to maintain high reliability and
performance

Component interconnection
A Distributed System should include
a communication subnet which
interconnects the elements of the
system. The transfer of information
via the subnet should be controlled by
a two-party, cooperative protocol
(loose coupling).

Unity of Control
All the component of the system
should be unified in their desire to
achieve a common goal. This goal will
determine the rules according to
which each of these elements will be
controlled.


System transparency
From the users point of view the set
of resources that constitutes the
Distributed Processing System acts
like a “single virtual machine”.
When requesting a service the user
should not require to be aware of the
physical location or the instantaneous
load of the various resources

Component autonomy
The components of the system, both the
logical and physical, should be autonomous
and are thus afforded the ability to refuse
a request of service made by another
element. However, in order to achieve the
system’s goals they have to interact in a
cooperative manner and thus adhere to a
common set of policies. These policies
should be carried out by the control
schemes of each element.

Many Distributed Challenges

› Race Conditions…
› Name spaces …
› Distributed ownership …
› Heterogeneity …
› Object addressing …
› Data caching …
› Object Identity …
› Trouble shooting …
› Circuit breakers …

26 years ago I
wrote a Ph.D. thesis –

“Study of Load Balancing
Algorithms for Decentralized
Distributed Processing
Systems”
http://www.cs.wisc.edu/condor/doc/livny-dissertation.pdf


BASICS OF A M/M/1
SYSTEM
Expected # of customers
λ is 1/(1-ρ), where
(ρ = λ/µ) is the
utilization
When utilization is 80%,
you wait on the average 4 units
µ
for every unit of service


BASICS OF TWO M/M/1
SYSTEMS
λ λ you wait on the average 4 units
for every unit of service

25% of the time a customer is
waiting for service while
µ µ
a server is idle


Wait while Idle (WwI)
in m*M/M/1

1 m = 20

m = 10
Prob
(WwI) m=5

m=2
0
0 Utilization 1

“ … Since the early days of mankind the
primary motivation for the establishment of
communities has been the idea that by being
part of an organized group the capabilities
of an individual are improved. The great
progress in the area of inter-computer
communication led to the development of
means by which stand-alone processing sub-
systems can be integrated into multi-
computer ‘communities’. … “
Miron Livny, “ Study of Load Balancing Algorithms for
Decentralized Distributed Processing Systems.”,
Ph.D thesis, July 1983.


12 years ago we had

“The Grid”


The Grid: Blueprint for a New
Computing Infrastructure
Edited by Ian Foster and Carl Kesselman
July 1998, 701 pages.

The grid promises to fundamentally change the way we
think about and use computing. This infrastructure will
connect multiple regional and national computational
grids, creating a universal source of pervasive
and dependable computing power that
supports dramatically new classes of applications. The
Grid provides a clear vision of what computational
grids are, why we need them, who will use them, and
how they will be programmed. www.cs.wisc.edu/~miron

“ … We claim that these mechanisms, although
originally developed in the context of a cluster
of workstations, are also applicable to
computational grids. In addition to the
required flexibility of services in these grids,
a very important concern is that the system
be robust enough to run in “production mode”
continuously even in the face of component
failures. … “
Miron Livny & Rajesh Raman, "High Throughput Resource
Management", in “The Grid: Blueprint for
a New Computing Infrastructure”.


High Throughput Computing
We first introduced the distinction between
High Performance Computing (HPC) and High
Throughput Computing (HTC) in a seminar at
the NASA Goddard Flight Center in July of
1996 and a month later at the European
Laboratory for Particle Physics (CERN). In
June of 1997 HPCWire published an interview
on High Throughput Computing.


Why HTC?
For many experimental scientists, scientific
progress and quality of research are strongly
linked to computing throughput. In other words,
they are less concerned about instantaneous
computing power. Instead, what matters to them
is the amount of computing they can harness over
a month or a year --- they measure computing
power in units of scenarios per day, wind patterns
per week, instructions sets per month, or crystal
configurations per year.


High Throughput Computing
is a
24-7-365
activity
FLOPY ≠ (60*60*24*7*52)*FLOPS


Obstacles to HTC

› Ownership Distribution (Sociology)
› Customer Awareness (Education)
(Robustness)
› Size and Uncertainties
(Portability)
› Technology Evolution
(Technology)
› Physical Distribution


The Three OSG Cornerstones
Need to be
National harmonized into a well
integrated whole.

Campus Community
]

“ … Grid computing is a partnership between
clients and servers. Grid clients have more
responsibilities than traditional clients, and
must be equipped with powerful mechanisms
for dealing with and recovering from
failures, whether they occur in the context
of remote execution, work management, or
data output. When clients are powerful,
servers must accommodate them by using
Douglas Thain & Miron Livny, "Building Reliable Clients and Servers",
careful protocols.… “
in “The Grid: Blueprint for a New Computing
Infrastructure”,2nd edition


The Ethernet Protocol

IEEE 802.3 CSMA/CD - A truly
distributed (and very effective)
access control protocol to a
shared service.
♥ Client responsible for access control
♥ Client responsible for error detection
♥ Client responsible for fairness


Introduction

“The term “the Grid” was coined in the mid 1990s to denote a proposed
distributed computing infrastructure for advanced science and
engineering [27]. Considerable progress has since been made on the
construction of such an infrastructure (e.g., [10, 14, 36, 47]) but the term
“Grid” has also been conflated, at least in popular perception, to embrace
everything from advanced networking to artificial intelligence. One might
wonder if the term has any real substance and meaning. Is there really a
distinct “Grid problem” and hence a need for new “Grid technologies”? If so,
what is the nature of these technologies and what is their domain of
applicability? While numerous groups have interest in Grid concepts and
share, to a significant extent, a common vision of Grid architecture, we do not
see consensus on the answers to these questions.”

“The Anatomy of the Grid - Enabling Scalable Virtual Organizations” Ian
Foster, Carl Kesselman and Steven Tuecke 2001.


Global Grid Forum (March 2001)
The Global Grid Forum (Global GF) is a community-initiated forum of individual
researchers and practitioners working on distributed computing, or
"grid" technologies. Global GF focuses on the promotion and development of
Grid technologies and applications via the development and documentation of
"best practices," implementation guidelines, and standards with an emphasis on
rough consensus and running code.
Global GF efforts are also aimed at the development of a broadly based
Integrated Grid Architecture that can serve to guide the research, development,
and deployment activities of the emerging Grid communities. Defining such an
architecture will advance the Grid agenda through the broad deployment and
adoption of fundamental basic services and by sharing code among different
applications with common requirements.
Wide-area distributed computing , or "grid" technologies, provide
the foundation to a number of large scale efforts utilizing the global Internet to
build distributed computing and communications infrastructures..


Summary
“We have provided in this article a concise statement of the “Grid
problem,” which we define as controlled
resource
sharing and coordinated resource use in
dynamic, scalable virtual organizations. We
have also presented both requirements and a framework for a Grid
architecture, identifying the principal functions required to enable
sharing within VOs and defining key relationships among these
different functions.”

“The Anatomy of the Grid - Enabling Scalable Virtual Organizations” Ian
Foster, Carl Kesselman and Steven Tuecke 2001.


What makes an

“O”
a

“VO”?

Client
Master

Server
Worker

Being a Master
Customer “delegates” task(s) to the master
who is responsible for:
l Obtaining allocation of resources
l Deploying and managing workers on allocated
resources
n Delegating work unites to deployed workers
d Receiving and processing results
o Delivering results to customer


Master must be …
› Persistent – work and results must be safely recorded on
non-volatile media
› Resourceful – delegates “DAGs” of work to other masters
› Speculative – takes chances and knows how to recover from
failure
› Self aware – knows its own capabilities and limitations
› Obedience – manages work according to plan
› Reliable – can mange “large” numbers of work items and
resource providers
› Portable – can be deployed “on the fly” to act as a “sub
master”

Master should not do …

› Predictions …
› Optimal scheduling …
› Data mining …
› Bidding …
› Forecasting …


Never assume that
what you know
is still true and that
what you ordered
did actually happen.

Resource Allocation
(resource -> customer)
vs.
Work Delegation
(job -> resource)


Resource Allocation
A limited assignment of the “ownership” of
a resource
m Owner is charged for allocation regardless of
actual consumption
o Owner can allocate resource to others
t Owner has the right and means to revoke an
allocation
g Allocation is governed by an “agreement”
between the client and the owner
n Allocation is a “lease”
“ Tree of allocations www.cs.wisc.edu/~miron

Garbage collection
is the
cornerstone
of
resource allocation


Work Delegation
A limited assignment of the
responsibility to perform the work
o Delegation involved a definition of these
“responsibilities”
s Responsibilities my be further delegated
Delegation consumes resources
m Delegation is a “lease”
“ Tree of delegations

Focus of grids has been
job delegation,
clouds are all about
resource allocation
(provisioning)


Cloud Computing
› Part of the mix of computing resources and
services that needs to be integrated into
our computing framework
› Defines a cost model for computing
capabilities
›Promotes Virtualization technologies
› Encourages “on-the-fly” deployment of
software stacks


Distributed system
architectures and scenarios

Miron Livny
Center for High Throughput Computing (CHTC)
Computer Sciences Department
University of Wisconsin-Madison
miron@cs.wisc.edu

The 1994 Worldwide Condor Flock
Delft Amsterdam
3
30
200 10

3

3
3
Madison
Warsaw
10
10
Geneva Dubna/Berlin


Every Community
can benefit from the
services of

Matchmakers!
eBay is a matchmaker

Why? Because ...
.. someone has to bring together
community members who have
requests for goods and services with
members who offer them.
e Both sides are looking for each other
g Both sides have constraints
r Both sides have preferences


Being a Matchmaker

› Symmetric treatment of all parties
› Schema “neutral”
› Matching policies defined by parties
› “Just in time” decisions
› Acts as an “advisor” not “enforcer”
› Can be used for “resource allocation”
and “job delegation”

Leads to a
“bottom up”
approach to building
and operating HTC
‘communities’

My jobs should run …
› … on my laptop if it is not connected to the
network
› … on my group resources if my certificate
expired
› ... on my campus resources if the meta
scheduler is down
› … on my national resources if the trans-
Atlantic link was cut by a submarine
* Same for “my resources”

Routing Jobs from
UW Campus Grid to OSG
HEP matchmaker
CS matchmaker
GLOW matchmaker

condor_submit
Grid
JobRouter

schedd globus
(Job caretaker) gatekeeper

Combining both worlds:
•simple, feature-rich local mode
•when possible, transform to
condor
grid job for traveling globally gridmanager

HTC on the Internet (1993)
Retrieval of atmospheric temperature and
humidity profiles from 18 years of data
from the TOVS sensor system.
n 200,000 images
n ~5 on Condor pools at the
Executed minutes per image University of Washington,
University of Wisconsin and NASA. Controlled by DBC
(Distributed Batch Controller). Execution log visualized by
DEVise


U of Washington U of Wisconsin NASA

Jobs per Pool
(5000 total)

Exec time
vs.
Turn around

Time line
(6/5-6/9)


The NUG30 Quadratic
Assignment Problem (QAP)
30 30
min
p∈∏ ∑∑
i=1 j=1
aijbp(i)p(j)


NUG30 Personal Grid …
Managed by one Linux box at Wisconsin

Flocking: -- the main Condor pool at Wisconsin (500 processors)

-- the Condor pool at Georgia Tech (284 Linux boxes)
-- the Condor pool at UNM (40 processors)
-- the Condor pool at Columbia (16 processors)
-- the Condor pool at Northwestern (12 processors)
-- the Condor pool at NCSA (65 processors)
-- the Condor pool at INFN Italy (54 processors)

Glide-in: -- Origin 2000 (through LSF ) at NCSA. (512 processors)
-- Origin 2000 (through LSF) at Argonne (96 processors)

Hobble-in: -- Chiba City Linux cluster (through PBS) at Argonne
(414 processors).

Solution Characteristics.
Scientists 4
Workstations 1
Wall Clock Time 6:22:04:31
Avg. # CPUs 653
Max. # CPUs 1007
Total CPU Time Approx. 11 years
Nodes 11,892,208,412
LAPs 574,254,156,532
Parallel Efficiency 92%

The NUG30 Workforce
System
Condor crash
Upgrade

Workers
Application
Upgrade


The Condor ‘way’
of resource management –

Be matched,
claim (+maintain),
and then delegate


Job Submission Options
› leave_in_queue = <ClassAd Boolean Expression>
› on_exit_remove = <ClassAd Boolean Expression>
› on_exit_hold = <ClassAd Boolean Expression>
› periodic_remove = <ClassAd Boolean Expression>
› periodic_hold = <ClassAd Boolean Expression>
› periodic_release = <ClassAd Boolean Expression>
› noop_job = <ClassAd Boolean Expression>

www.cs.wisc.edu/condor

DAGMan
startD
3

1
3
starter
schedD
1 2 3 4 5 6
4
Globus
shadow 5
1
3 EC2
grid 6
4 5
GAHP- 4
manager startD
EC2 DAGMan

schedD
starter

schedD
6
BLAH- 5 Globus
EC2 shadow
NorduGrid
grid
manager GAHP
Globus

schedD
GAHP- GAHP
NorduGrid

Condor 6 www.cs.wisc.edu/~miron
GAHP
Condor

Overlay Resource Managers
Ten years ago we introduced the concept of Condor
glide-ins as a tool to support ‘just in time scheduling’
in a distributed computing infrastructure that
consists of recourses that are managed by
(heterogeneous) autonomous resource managers. By
dynamically deploying a distributed resource manager
on resources provisioned by the local resource
managers, the overlay resource manager can
implement a unified resource allocation policy.


MM
PSE or User Condor C-app
Local
MM
SchedD (Condor G)

Grid Tools
MM MM MM
SchedD SchedD SchedD
(Condor C) (Condor C) (Condor C)

MM MM
LSF PBS Condor Condor
StartD StartD StartD
(Glide-in) (Glide-in) (Glide-in)

Remote
C-app
G-app C-app
G-app C-app
G-app
C-app

Managing Job Dependencies
15 years ago we introduced a simple language and a
scheduler that use Directed Acyclic Graphs (DAGs)
to capture and execute interdependent jobs. The
scheduler (DAGMan) is a Condor job and interacts
with the Condor job scheduler (SchedD) to run the
jobs.

DAGMan has been adopted by the Laser
Interferometer Gravitational Wave Observatory
(LIGO) Scientific Collaboration (LSC).


Example of a LIGO Inspiral
DAG

Use of Condor by the LIGO
Scientific Collaboration

• Condor handles 10’s of millions of jobs per year
running on the LDG, and up to 500k jobs per DAG.
• Condor standard universe check pointing widely
used, saving us from having to manage this.
• At Caltech, 30 million jobs processed using 22.8
million CPU hrs. on 1324 CPUs in last 30 months.
• For example, to search 1 yr. of data for GWs from
the inspiral of binary neutron star and black hole
systems takes ~2 million jobs, and months to run on
several thousand ~2.6 GHz nodes.

A proven computational protocol for genome-wide predictions and
annotations of intergenic bacterial sRNA-encoding genes
NCBI FTP
ORFs
t/rRNAs IGRExtract3
genomes
ROI IGRs All other IGRs
RNAMotif FindTerm TransTerm
BLAST
Terminators
Conservation
Known sRNAs,
riboswitches sRNAPredict

BLAST sRNA candidates FFN parse

TFBS BLAST BLAST
matrices ORFs flanking ORFs flanking
Paralogues candidates known sRNAs
Homology QRNA
to known
Patser BLAST
o
2 conservation
TFBSs sRNA Annotate Synteny

Annotated candidate sRNA-encoding genes

Using SIPHT, searches for sRNA-
encoding genes were conducted in
556 bacterial genomes (936
replicons)

This kingdom-wide search:

• was launched with a single command line and required no further user
intervention

• consumed >1600 computation hours and was completed in < 12 hours

• involved 12,710 separate program executions

• yielded 125,969 predicted loci, inlcluding ~75% of the 146 previously
confirmed sRNAs and 124,925 previously unannotated candidate genes

•The speed and ease of running SIPHT allow kingdom-wide searches to be repeated often
incorporating updated databases; the modular design of the SIPHT protocol allow it to be
easily modified to incorporate new programs and to execute improved algorithms

Customer requests:

Place y = F(x) at L!
System delivers.


Simple plan for y=F(x)L

› Allocate (size(x)+size(y)+size(F)) at
SEi
› Place x from SEj at SEi
› Place F on CEk
› Compute F(x) at CEk
› Move y fromCompute Element (CE)
Storage Element (SE);
SEi at L
› Release allocated space at SEi

Customer requests:

Place y@S at L!
System delivers.


Basic step for y@SL
› Allocate size(y) at L,
› Allocate resources (disk bandwidth,
memory, CPU, outgoing network
bandwidth) on S
› Allocate resources (disk bandwidth,
memory, CPU, incoming network
bandwidth) on L
› Match S and L


Or in other words,
it takes two (or more)
to Tango
(or to place an object)!


When the
“source” plays “nice”
it “asks”
in advance
for permission to
place an object at the
“destination”


Match!

I am S and I am L and
am looking I have what
for L to it takes to
place a file place a file

The SC’05 effort

Joint with the
Globus GridFTP team


Stork controls
number of
outgoing
connections

Destination
advertises
incoming
www.cs.wisc.edu/~mironconnections

A Master Worker
view of the same
effort


Master

Files
Worker

For
Workers

When the
“source” does not
play “nice”,
destination must
protect itself


Managed Object Placement
Management of storage space and bulk
data transfers plays a key role in the end-
to-end effectiveness of many scientific
applications:
n Object Placement operations must be treated as “first class”
tasks and explicitly expressed in the work flow
i Fabric must provide services to manage storage space
i Object Placement schedulers and matchmakers are needed
Object Placement and computing must be coordinated
Smooth transition of Compute/Placement interleaving across
software layers and granularity of compute tasks and object
size
a Error handling and garbage collection


How can we accommodate
an unbounded
need for computing and
an unbounded
amount of data with
an unbounded
amount of resources?

Session 3-Distributed System Principals

More Related Content

Similar to Session 3-Distributed System Principals (20)

More from ISSGC Summer School (20)

Recently uploaded (20)

Session 3-Distributed System Principals