Introduction to glideinWMS

glideinWMS for users

Introduction to glideinWMS
by Igor Sfiligoi (UCSD)

CERN, Dec 2012 glideinWMS Intro 1

Scope of this talk

This talk provides a
user perspective of glideinWMS
for users with previous experience
with Grid computing.

It does not provide much detail
but concentrates on the concepts
behind the system instead.


The problem(s)
● Users have many ● Resources provided
jobs that must be run by O(100) Grid sites
● Each user has
multiple tasks at once

● How do we schedule them to get the results
in the shortest amount of time?
● Assuming one result per task
● How do we treat all users in a fair way?
● Independently of how many jobs they submit


glideinWMS approach
● Separates
● Resource provisioning from Never the
● Resource scheduling user jobs!

● In practice, sends out pilot jobs to the Grid
● And creates an “overlay batch system”
● Pilot jobs get ownership of the Grid slots
● At least for a limited time Grid Site

Grid Site
Overlay
● Known also as batch
Grid Site
system
the pilot approach
Grid Site
Grid Site


Job scheduling
● Once we have the overlay batch system,
job scheduling works like in any dedicated B.S.
● We own the B.S. and can set the
job scheduling policies
● glideinWMS based on HTCondor
● So can do whatever HTCondor can do
● Which is quite flexible
– But also nothing more... HTCondor
(formerly knowns as Condor)
● HTCondor-based pilots are is a widely used
batch system.
usually called glideins
Thus glideinWMS stands for “glidein based Workflow Management System”
More details
later on.

Creating the overlay
i.e. resource provisioning

● glideinWMS will grow and shrink the
overlay B.S. automatically
● No human intervention needed
● Expansion based on user jobs in the queue
● The more jobs, the faster it will try to grow
● Since not all jobs can run at all sites,
different attempted growth rates for different sites
● Shrinks automatically if resources unused
● Again, based
on user jobs in the queue Each glidein should run
at least one user job,
but will try to run many
if the Grid slot is long enough

glideinWMS in a picture

Grid Site Grid Site

Grid site
glideinWMS
HTCondor
CPU Handler

User Job
HTCondor
Job
Repository


From the user point of view
● Users see just a “regular” HTCondor system
● Just a dynamic one
● However
● Have to be aware of the resource provisioning logic
– No native HTCondor tools to help with this
● Debugging system problems much harder
– Again, no native HTCondor tools to help with this
– Most likely question a user will ask is
“Why is my job not starting?”


A few more details
● So, glideinWMS is really HTCondor++
● First you have to understand how HTCondor works
● If you do, 99% of the problems are solved

● HTCondor is composed of 3 logical pieces
● Submit nodes – keep the job queue(s)
● Execute nodes – owns and operates a resource
● A central manager – glues the other two categories
together and executes policies


HTCondor in a picture

Execute node

Central manager Execute node
Submit node
Execute node
Condor
Submit node
Execute node
Submit node
Execute node
Condor Condor


Even more details
● The actual work quanta are
● Jobs – on the submit node, typically many/node
● Slots – on the execute node, typically only a few /node
● While internally implemented differently,
jobs and slots are conceptually very similar
● Both describe a logical entity
“ClassAd” in
● Both have attributes describing it HTCondor speak
● Both have requirements
● HTCondor policy engine is really all about
matchmaking jobs to slots


Matchmaking
● Jobs that are not running
(i.e. are “idle” in HTCondor speak)
will be matched against Slots
that don't yet run anything
(i.e. are “Unclaimed” in HTCondor speak)
● Requirements expressions can (and usually do)
reference attributes in the other ClassAd
● Both sides must evaluate to True for a match
● Although we encourage all logic to reside in the
Slot Requirement in glideinWMS setups
(more on this later on)


A non-technical example

Buyer Ad Pet Ad
MyType = “Buyer” MyType = “Pet”
TargetType = “Pet” TargetType = “Buyer”
Requirements = Requirements =
(PetType == “Dog”) && DogLover == True
(Price <= AcctBalance) && PetType = “Dog”
(Size == "Large"||Size == "Very Large")
Color = “Brown”
AcctBalance = 1000
Price = 75
DogLover = True
Breed = "Saint Bernard"
LegalName = “Curly Howard”
... Size = "Very Large"
...

Buyer ~= Job Dog == Resource ~= Slot


Matching order
● Most of the time, there are
way more idle jobs than Unclaimed slots
● So order is important
● Two policies
1) Jobs from highest priority user first
2) Priority-FIFO policy for jobs of the same user
● User priority based on usage
● The more resources you use, the lower the priority
(with priority recovery over time)
● But some users may be marked as “more important”
(priority multipliers and group quotas)


The glideinWMS part
● i.e. The layer on top of HTCondor that
finds resources where to start the
“Execute node” daemons
● i.e. glideins
● Composed of two parts:
● Glidein Factory – The abstraction layer
● VO Fronend – The brain


Glidein Factory
● The splitting in two allows for the i.e. serve
many VO FEs
Glidein Factory to be generic
● I.e can (and should) be shared between many VOs
● The G.F. is really just an abstraction layer
● It insulates the VO from the provisioning details
– e.g. knowing the name of the Grid CE and relative RSL
● Allows new technology to be added seamlessly
(e.g. Clouds)
● It also provides a troubleshooting service
● The factory operators are supposed to
address any Grid related problems they observe


The VO Frontend
● The name may be misleading
● It is really the “matchmaker of Grid resources”
● Introduces a new quanta
● Entry – logical equivalent of a “queue at a site”
Basic working block of a G.F.
● The VO Frontend
1) Matches idle Jobs to Entries
2) Instructs the affected G.F. to
increase or decrease the number
of glideins on that Entry
Thus regulates the
resource provisioning


Updated glideinWMS picture
G.F.
+3
VO FE Grid
G.F.
+1
Execute node

Central manager Execute node
Submit node
Execute node
Condor
Submit node
Execute node
Submit node
Execute node
Condor Condor


VO FE Matchmaking logic
● Based on Job attributes
● Jobs don't have “FE-specific requirements”
● The exact matchmaking policy
depends on the VO FE instance
Will describe CMS policies in a different talk
● glideinWMS has 2 level matchmaking
● Once in the FE, then in the HTCondor C.M.
● Recommended to avoid explicit
“HTCondor requirements” in the Job ClassAd
– The glideins should set “Slot requirements” based on the
same attributes used by the VO FE, instead
Since VO FE configures the glideins

What is the user to do?
0) Learn how to use HTCondor
1) Learn what the VO FE policy is
2) Create the HTCondor submit file (i.e. JDL)
containing the necessary attributes
3) Submit jobs
4) Wait for the results to come back
5) Rinse and repeat (from (2))


This is it

● Hopefully you have a high level view of
the system now
● More details in separate talks


Pointers
● glideinWMS Home Page
http://tinyurl.com/glideinWMS
● HTCondor Home Page
http://research.cs.wisc.edu/htcondor/
● HTCondor support
htcondor-users@cs.wisc.edu
htcondor-admin@cs.wisc.edu
● glideinWMS support
glideinwms-support@fnal.gov


Acknowledgments
● The creation of this document was sponsored
by grants from the US NSF and US DOE,
and by the University of California system


Introduction to glideinWMS

More Related Content

Viewers also liked (17)

Similar to Introduction to glideinWMS (20)

More from Igor Sfiligoi (20)

Recently uploaded (20)

Introduction to glideinWMS