ITPerformanceMeasurementStandards

IT PERFORMANCE MEASUREMENT STANDARDS

November 2015

Developed through collaboration of Higher Education IT professionals

Mission Statement

The mission of the Consortium for the Establishment of IT Performance Standards (CEITPS) is to provide
royaltyfree standards for Information Technology (IT) performance measures. The Consortium is initially
focused on information technology performance measures in higher education, but the standards published
may (should) be applicable to other industries that employ information technologies.

Metrics are increasingly becoming a universal tool used for improving information technology services or
products. One barrier to the successful use of metrics in higher education IT circles is the absence of a
common language and a lack of agreed upon standards for those measures. A set of standards for IT
performance metrics will enhance the ability of organizations to improve the performance and the value or
quality of their services and products. Additionally, standards for determining the quality of services or
products will offer organizations the opportunity to fairly and accurately benchmark with their peers.

Thanks to all our contributors

Name Institution
Richard Coffey Argonne National Laboratory
Mary Therese Durr Boston College
Matt Davidman CDW Government
Ronak Shah CDW Government
Pete Hoffswell Davenport University
Donna Petherbridge NC State University
Michael Budzik Purdue University
Kim IveyBourne The University of North Carolina at Greensboro
Joshua Jones University of North Dakota
Donald Padgett University of Notre Dame
Martin Klubeck University of Notre Dame
Michelle Sorensen University of Notre Dame
Paul Drake University of Notre Dame

Our Call To Action:
Six years ago members of the IT Metrics Constituent Group formed a notforprofit organization, Consortium
for the Establishment of IT Performance Standards (CEITPS) to create standards for how universities and
colleges measure IT Performance in Higher Ed.  The idea was to allow higher education institutions to
compare their performance with other higher education institutions.  This is not currently possible because
Higher Ed doesn't use the same language and the performance measures are not standardized.

During the EDUCAUSE Annual Conference 2015 (Indianapolis, IN October 2730th) members of CEITPS
plan to take a large step in creating those standards, culminating in publication. With the support of the
EDUCAUSE Annual Conference committee we'll have dedicated space for multiple working sessions for
creating standards.  Over the course of two days (10/28 10/29) we will draft, review, edit, and finalize the
standards.

This first draft was created by attendees at the annual Educause Conference, 2015.  As a result of our call
to action we succeeded at drafting 21 standards covering 7 focus areas.  Those standards are documented
here for your review.  We will publish these standards on Monday, Nov 9th, making them available for free to
the public.  Your name and your institution will be listed as a contributor if you helped write a standard or if
you (during this week) provide feedback.  Your input can be as little as supporting the submissions (tell us
they’re good to go) or as much as providing inputs on how to improve them.  We need your feedback by
noon on Friday November 6th.

If you’d like to work on creating a new standard, there is still time.  You should collaborate with at least two
others (if they are from your institution, they should have a different job focus).  To access the drafts follow
this link. Note:  After selecting the measure to work on, you may be locked in Google’s Preview mode.  If you
are, click on the icon for “popout”   and you should be able to edit.

It is highly recommended that you FIRST review the existing standards so you will become familiar with the
way they have been done and the tone, tenor, and pitch of the writing.

Standards Table of Contents by Focus Area

Application Development
Defects Found During Development
Defects Found During Testing; Internal, External, or Combined
Incident Management
Category/Type of Incident
Rework Incidents
Product/Service Management
Adoption Rate
Incidents per Service
Program and Project Management
Active Projects
Functionality Delivered (projects), Percentage of not weighted
Functionality Delivered (projects), Percentage of weighted
OnCost Delivery (projects)
OnTime Delivery (projects)
Security Management
Security Related Incidents Electronic
Security Vulnerabilities
Service Desk, Trouble Call Resolution
Voice Call Abandon Rate
Chat Session Abandon Rate
First Contact Resolution
MTTR (by time grouping qualifier)
MTTR minus customer wait time (by time grouping qualifier)
Service Management
Service Availability Service Downtime per SLA
Service Availability Service Downtime Total
Top N Categories/Types of Incidents

Application Development
Defects Found During Development

Description:  Defects found during the effort to develop deliverables
Attributes

1. Development
Dedicated effort for creating deliverables starting from effort initiation to Testing (a dedicated
effort for testing development products before they are deployed to the customer).

2. Defects
Any identifiable item which does not function as intended by the enduser, or is absent.  Any
deviation from the expected or required functionality.  Any specific item requiring correction
before deployment.  NOTE:  this does not include customer initiated changes to requirements.

3. Defects found
Defects identified during development.

Formula
Straight count of Defects

Defects Found During Testing; Internal, External, or Combined

Description:  Defects found during testing but not live deployment
Attributes

1. Testing
Dedicated effort for testing development products before they are deployed to the customer.
a. internal testing you can designate this measure to cover only internal testing (QA)
b. external testing you can designate this measure to cover only external (customer)
testing, including Beta Testing by a select group.
c. Combined you can designate this measure to cover all testing from completion of
development to deployment

2. Defects
Any identifiable item which does not function as intended, or is absent.  Any deviation from the
expected or required functionality.  Any specific item requiring correction before deployment.
NOTE:  this does not include customer initiated changes to requirements.

3. Defects found
Defects identified during the testing.

Formula
Straight count of Defects

Incident Management
Category/Type of Incident

Description: categories or types for grouping and filtering incidents

Attributes

1. Category/Type
A class or division of things regarded as having particular shared characteristics that are clearly
defined in the metric title. (For example: Cause of Incident, Duration to Resolve, etc)
a. Cause Incident cause is a categorization of the determination of why the incident
occurred.
b. Duration to resolve the time it took to resolve the incident, from the customer’s point of
view. Total time from Incident identification to incident resolution. (see Incident
Resolution for more information)
c. Resolution Type the categorization of the determination of how the incident was
resolved.

2. Incident
Client request for assistance with a service as reported to either a central support organization
such as a service desk, or the human resources responsible for a service.
Incident constitutes the failure of a product or service to meet or maintain established
performance standards. When a product or service fails to meet expectations or requirements

Formula
Count of incidents per Cause, Duration to Resolve, or Resolution Type.

Rework Incidents

Description: How many incidents were incorrectly classified as resolved

Attributes

1. Incident
Client request for assistance with a service as reported to either a central support organization
such as a service desk, or the human resources responsible for a service.
Incident constitutes the failure of a product or service to meet or maintain established
performance standards. When a product or service fails to meet expectations or requirements

2. Rework
Each (and every) time any incident requires more effort after it was incorrectly or not fully
resolved but was considered to be resolved. Note: usually presented as a percentage of total
incidents worked in a given timeframe

Formula
1(Number of Rework Incidents/Total Incidents)

Product/Service Management
Adoption Rate

Description: Percentage of users of a service/product based on target populace

Attributes

1. Target Populace
The adoption rate is based on the target populace those who could use the service/product.
The people to whom the product/service is available.

2. Service/Product
An offering provided to a set of customers with an intended level and frequency of use.

3. Use
Use is equal to voluntary use, not mandated by policy. Used at least once per intended
frequency that is applicable.

            4.  Adoption
When the product is used as the primary product for the intended use, the user is considered an
“Adopter”

Formula
Number of “Adopters” divided by the number of people in the target populace

Incidents per Service

Description: The number of incidents per service [product?]

Attributes

1. Incident [see previous work on incidents]
A client requests assistance with a service as reported to either a central support organization
such as a service desk, or the human resources responsible for a service. Incident constitutes
a the failure of a product or service to meet or maintain established performance standards.

2. Service
Service is a predefined system including; software, hardware, and/or human capital for
identifying and responding to a customer need.

3. Product
Product is an application that provides a service.

Formula
Straight count of incidents

Program and Project Management
Active Projects

Description: The number of active projects in any given time period

Attributes

1. Date of measurement
Active projects during a given time reflects any project which was active at anytime during the
time frame.  There needs to be a clear start and stop time to the time frame.  For example
Active Projects for the last Fiscal Year, would require defining the “last Fiscal Year” with the start
, stop and inclusive dates.  Since different institutions use different “fiscal year” definitions, the
specific timeframe should be The time period must be defined using actual dates.

2. Project
A project should have a predefined minimum of estimated hours required to complete. Projects
are unique work efforts not normal maintenance or operations work. IT projects are approved
by a specifically designated authority (at UNCG, we have the Project Review Committee).

3. Active
The project has to be approved by a specifically designated authority. The project is not
deferred, on hold, complete (closed) or canceled.  Also, the project should have people working
toward the deliverables.  The project is complete when a specifically designated authority (which
may differ for each project) signs off on it’s completion. Note, in many cases this authority is the
customer.

Formula
Straight count of Active Projects with a defined timeframe.

Functionality Delivered (projects), Percentage of not weighted

Description: For each project, a measure of how much functionality delivered as a deviation
delivered compared to the planned amount

Attributes

Date of Delivery (see previous definition)

2. Project
See previous definition

3. Delivered
See previous definition Live and in use

4. Planned Functionality
the customer agreed upon expected functionality as documented in the scope

5. Actual Functionality
the delivered functionality

Formula
Percentage of what you delivered: actual functionality divided by planned functionality

Functionality Delivered (projects), Percentage of weighted

Description: For each project, a measure of how much functionality delivered as a deviation
delivered compared to the planned amount

Attributes

Date of Delivery (see previous definition)

2. Project
See previous definition

3. Delivered
See previous definition  Live and in use

4. Planned Functionality weighted
the customer agreed upon expected functionality as documented in the scope with a weight
determined by the customer.  The customer has to weigh each item in the scope so that the
sum of all equals 100%.

5. Actual Functionality
the delivered functionality with the weights used in the planned functionality.

Formula
Percentage of what you delivered: actual functionality times it’s weight  divided by planned
functionality (at a total of 100%)

OnCost Delivery (projects)

Description: For each project, a measure of how close to the Budgeted Cost the project was
delivered at

Attributes

Date the project Closed.  Closed is when no more work is due to be done.  When all effort is
completed.  The customer has to have signed off on the project completion.

2. Project
See Other Definition minus the first sentence.

3. Delivered
Deployed to production and in use for business practices.

4. Budgeted Cost
The documented planned funding expectations for the project.  only actual monies to spend (not
including staff/employee hours)

5. Actual Cost
The organization’s actual spend on the project not including staff/employee hours)

Formula
Actual Cost divided by Budgeted Cost

OnTime Delivery (projects)

Description: Percentage of projects delivered “OnTime” each month and cumulative by
specified timeframe

Attributes

Projects delivered during a given time period with a clear start and stop date (mm/dd/yyyy)

2. Project
A project should have a predefined minimum of estimated hours required to complete.  Projects
are unique work efforts not normal maintenance or operations work. IT projects are approved
by a specifically designated authority.

3. Delivered
Customer acceptance of all defined project deliverables given the scope as specified at the end
of the execution phase of the project.

4. OnTime
For projects with a planned duration of less than one year, there should be less than 5%
variance (planned duration + 5% of planned duration) between the baseline (planned delivery)
date and the actual date of delivery. For projects with a planned duration of over a year, there
should be a maximum of 10% variance between the dates.  Any project delivered (all
deliverables completed) earlier or equal to the baseline delivery date (plus the variance allowed)
is considered ontime.

Formula
(Total # of projects delivered / projects that close on time)*100 = % projects delivered on time

Security Management
Security Related Incidents Electronic

Description: This is based on the closing disposition of incident and call records. Benchmark this
over time. This will depend on the level of control as well as other factors such as software
quality

Attributes

1. Security Incident Electronic
An identifiable event where established security protocols have been bypassed

2. Security Protocol
Sign in requirement, password, biometric, two factor authentication, or accessing any nonpublic
data without proper authorization. For example, where unauthorized use/access of information
occurs.

Formula
Simple count of Security Related incidents electronic, can be for any defined time period (don’t
use shorthand CY, FY, Year instead use actual dates)

Security Vulnerabilities

Description: This is based on the closing disposition of incident and call records. Benchmark this
over time.  This will depend on the level of control as well as other factors such as software
quality

Attributes

1. Security Vulnerability
Any scenario in which potential unauthorized access to data may occur.  Or where an
opportunity for unauthorized disclosure or access exists.
a. this involves potential, not realized security related incidents.
b. One data instance may contain multiple vulnerabilities vulnerabilities are defined by
the method of access not the actual data being accessed.

Formula
Simple count of vulnerabilities identified.  Can be categorized or grouped by type or ways of
accessing data.

Service Desk, Trouble Call Resolution
Voice Call Abandon Rate

Description: The number of calls where users abandon the call divided by the total number of
eligible calls

Attributes

1. Voice Call
Any customer initiated contact that lasts more than 30 Seconds. the 30 second onhold window
is used to cover organizations that use automated response software.

2. Abandoned Call
Any Call that the customer terminates after 30 seconds and before successful contact is
established

3. Defined Period
Use a Start and Stop date for defining the period in question (do not use terms like FY, or CY as
they may differ from institution to institution).

Formula
The number of abandoned calls divided by the number of calls for any defined period.

Chat Session Abandon Rate

Description: The number of Chat sessions where users abandon the chat session divided by the
total number of eligible chat sessions

Attributes

1. Chat Session
Any customer initiated chat session that lasts more than 30 Seconds.  The 30 second onhold
window is used to cover organizations that use automated response software.

2. Abandoned Chat
Any Call that the customer terminates after 30 seconds and before successful contact is
established

3. Defined Period
Use a Start and Stop date for defining the period in question (do not use terms like FY, or CY as
they may differ from institution to institution).

Formula
The number of abandoned chat sessions divided by the number of chat sessions for any
defined period.

First Contact Resolution

Description: Percentage of customer’s needs met on first contact

Attributes

1. First Contact
“Any contact where a customer has reached out for IT support.”  First contact is defined as
staying with the same team and not being handed off to another person or team. First contacts
can include email, FAX, phone call, text, chat, video chat, in person, sticky notes, social media,
memos, web request, autologger, etc.  First Contact is the initial communication.   If a contact
results in further contact or escalated contact, it does not qualify as a “first contact”

2. Resolution
The customer does not reopen contact for the same issue the customer’s need is satisfied.
This includes a confirmation by the customer that their issue is solved.  No further follow through
is necessary.

Formula
The percentage of contacts that DO NOT require ANY further contacts to address the
customer’s issue.  Contact resolved upon initial contact divided by total contacts.

MTTR (by time grouping qualifier)

Description: This measures the major part of total call time up to the resolution when the user
is satisfied with the resolution.  Please note, it only measures the time an incident is in various
states, and excludes the time ‘waiting for user’ and ‘closed’

Attributes

1. Resolution
Resolution of an incident occurs when the ticket is closed to the customer’s acceptance in so far
as the customer’s expectations fall within the SLA.  To report a resolution to the user, the
service must be operating at or above the level defined in the service’s SLA and a support agent
must verify correct operation of the service.

● Service (or all dependent services) restored for end users
● Failed hardware repaired or replaced
● System operating at full expected capacity?
● System operating at a capacity to meet current needs?
● Ticket closed to customer’s satisfaction
● Support personnel validates that the issue has been resolved.

2. Start Time of Resolution
The Start Time of resolution is the time of first report of failure to support personnel.

3. End Time of Resolution
The End time of Resolution is the time that the support agent notified the customer after
verifying that the service is operating at or above the level defined in the service’s SLA.

5. Mean Time To Resolve
Mean Time to Resolve is the average time for incident resolution across all resolved incidents.

Formulas

MTTR  = End Time of Resolution Start Time of Resolution
Total number of resolved incidents

MTTR minus customer wait time (by time grouping qualifier)

Description: This measures the major part of total call time up to the resolution when the user
is satisfied with the resolution.  Please note, it only measures the time an incident is in various
states, and excludes the time ‘waiting for user’ and ‘closed’

Attributes

1. Resolution
Resolution of an incident occurs when the ticket is closed to the customer’s acceptance in so far
as the customer’s expectations fall within the SLA.  To report a resolution to the user, the
service must be operating at or above the level defined in the service’s SLA and a support agent
must verify correct operation of the service.
● Service (or all dependent services) restored for end users
● Failed hardware repaired or replaced
● System operating at full expected capacity?
● System operating at a capacity to meet current needs?
● Ticket closed to customer’s satisfaction
● Support personnel validates that the issue has been resolved.

2. Start Time of Resolution
The Start Time of resolution is the time of first report of failure to support personnel.

3. End Time of Resolution
The End time of Resolution is the time that the support agent notified the customer after
verifying that the service is operating at or above the level defined in the service’s SLA.

4. Waiting on Customer Status
Waiting on Customer Status is the status (or statuses) that incident tickets are placed in while
the support agent is waiting on a response from the customer.

5. Mean Time To Resolve
Mean Time to Resolve is the average time for incident resolution across all resolved incidents.

Formulas
MTTR minus customer wait time  = (End Time of Resolution Start Time of Resolution minutes
Waiting on Customer) / Total number of resolved incidents

Service Management
Service Availability Service Downtime per SLA

Description: Availability of the service per Service Level Agreement

Attributes

1. Service/Product
Service = a predefined system including; software, hardware, and/or human capital for
identifying and responding to a customer need

2. Downtime
Anytime the service is not available as defined inside of the service level agreement. If no time
is explicitly stated, then 24/7/365.

3. Total time available
The amount of time measured in minutes the service is supposed to be running per the SLA

4. Service Level Agreement
A customer approved agreement describing the parameters for expected service availability.

Formula
1 (downtime / total time available per SLA) In minutes

Service Availability Service Downtime Total

Description: Overall Availability based on 24/7/365 clock/calendar

Attributes

1. Service/Product
Service = a predefined system including; software, hardware, and/or human capital for
identifying and responding to a customer need

2. Downtime Total
Anytime the service is not available, measured in minutes

3. Total time available
The amount of time measured in minutes, based on a 24/7/365 clock/calendar.

Formula
1 (downtime / total time available)   measured in minutes

Top N Categories/Types of Incidents

Description: The top N (a number) of incidents by Category/Type.

Attributes

1. Incident: see Category/Type of Incident

2. Category/Type: see Category/Type of Incident

3. Top N: Looking at the incidents by Category or Type, in order of most incidents to
fewest. Select the Top N.

Formula
Count in descending order, including only those that make it into the top N. Note: if you
aggregate obscure (or low number) categories/types into an “other” category/type; and that
“other” category is within your top N, break out the grouping so that you don’t have “other” in the
selected group.

ITPerformanceMeasurementStandards

More Related Content

What's hot (6)

Viewers also liked (20)

Similar to ITPerformanceMeasurementStandards (20)

ITPerformanceMeasurementStandards