PGCS 2019 Master Class Integrating SE with PPM

The projects are managed as if they were merely complicated ‒
when in fact, they were complex.
They are planned as if everything is known or knowable at the start
‒ when in fact, they involve high levels of reducible (Epistemic) and
irreducible (Aleatory) uncertainty and resulting risk.
Combining Systems Engineering and Project Management is a
critical success factor in reducing these uncertainties, resulting in
increased probability of program success. [93]
Glen B. Alleman
Thursday 22 August 2019, 9:00 AM ‒ 5:00 PM
Master Class
Integrating
Systems Engineering (SE)
and
Project Performance Management (PPM)
The Match Made in Heaven Which Increases the
Probability of Project Success
PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia1

2 PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia

Workshop Contents
3
PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
Foundation Topics Advanced Topics
Introduction

From Key‒Note Talks
Here’s The Framework for the
Workshop4
PGCS 2019 Key Note, Canberra Australia
Increasing
Probability of
Program
Success
Delivering Needed
Capabilities for
Program Success
Measures of Effectiveness
Measures of Performance
Key Performance Parameters
RiskManagement
Arrive at Needed
Time for Needed
Cost for Program
Success
Technical Performance
Measures
Schedule Performance
Measures
Cost Performance Measures
Introduction

Seeing the Three Phases of Project as a
Whole is the Foundation of Systems
Thinking [148], [205]
5
Concept
Design and
Delivery
Operation and
Support
Introduction

During all 3 Phases There is Shared Data
Supported with the Risk Management
Process
 Concept of Operations
(ConOps)
 System Architecture
 Measures of Effectiveness
(MOE)
 Measures of Performance
(MOP)
 Key System Attributes (KSA)
 Key Performance Parameters
(KPP)
 Programmatic Architecture
(IMP/IMS)
 Technical Performance
Measures (TPM)
 Quantifiable Backup Data
(QBD)
 Cost Performance (CPI)
 Schedule Performance (SPI)
 Physical Percent Complete
(P%C)
 Estimate To Complete (ETC)
 Estimate at Completion (EAC)
6
Systems Engineering Program Performance Management
 Risk Burndown Performance
 Cost Margin Burndown Performance
 Schedule Margin Burndown Performance
Introduction

Workshop Framework
Knowing where to
aim is the
foundation of all
project success.
Aiming starts with
the Systems
Engineer’s
Description of Done
of the needed
Capabilities for
Mission or Business
Success.
Managing the
performance of work
needed to Get to
Done, is the role of
Project Performance
Management.
Introduction

The Integration of SE and PPM
provides …
8
… in Units of Measure Meaningful to the
Decision Makers.
Introduction

Introduction

Transformation Context of SE + PPM[163]
10
 Mechanistic
 Differentiated
 Work the people
 Top-Down, Managed
 In Parallel
 Efficiency Oriented
 Systemic
 Integrated team based
 Work the work
 Outside-In, Lead
 Each Other, for Each Other
 Complexity, Robust
Introduction

The Starting Point for Our Workshop
11
Introduction

Why Project Performance Management
rather than Project Management?
 Project Management methods are everywhere,
from PMI, to guides for each country and industry,
to internal governance documents and all the
tools that implement those Guides.
 The Egyptians had project management methods to
build the pyramids [158]
 What’s missing is a single, concise set of
Principles and Processes for Increasing the
Probability of Project Success. This starts with
Five Immutable Principles:
1. What Does Done Look Like?
2. What’s our Plan to Reach Done?
3. What resources are needed to reach Done?
4. What implements impacting our ability to reach
Done?
12
Introduction

What Does It Mean When We Say
Project Success?
 On Budget
 On Schedule
 Delivered
Requirements arrive
for the planned cost
at planned time
 Capabilities to
accomplish the
mission or meet
Business Goals
 Tangible benefits to
organization
 Tangle benefits to
stakeholder
 Enable future benefits
to organization &
stakeholder
13
Project Management
Paradigm
Systems Engineering
Paradigm
Project Efficiency Project Efficacy
Introduction

Critical Success Factor for All Projects
14
Both Systems Engineering and Project Management
have foundations in Risk Management
For Success we must seamlessly integrate Risk Management into Both
Introduction

The Core Question to be Answered by Integrating SE & PPM
What Data and Processes need to be Integrated?
Why, When, and What are the Tangible Benefits to the Project and
the Business from this Integration?
Introduction

Roles of PPM and SE [98]
 Focused on the Business
Requirements
 When are these needed?
 How much will they cost?
 Responsible for designing
and operating the control
system that manages the
work associated with the
solution that meets:
 Technical Performance
Measures
 Quantifiable Backup Data
 Focused on the Business
Solutions that deliver
Capabilities
 What are they?
 How are they assembled?
 Responsible for defining,
designing, and delivering
this solution that meets:
 Measures of Effectiveness
and Performance [105]
 Measures of Performance
Key Performance
Parameters
 Key System Attributes
16
Project ManagerSystems Engineer
Introduction

Responsibility of PPM and SE [136]
The project managers
are responsible for
meeting all project
targets, especially
providing the product on
time and within the
determined budget,
while systems
engineers lead the
technical efforts
necessary to developing
the system.
A systems engineer is
the project’s supreme
technical arm.
System engineer must
understand the
accepted product
development
processes, “tailor” them
to each specific project,
implement them in the
development process,
and pass them on to
production.
17
Project ManagerSystems Engineer
Introduction

We Need to End the Tug of War Between
Competing Paradigms …
18
… and
Start Integrating the Processes and Data
Needed to Increase the Probability of Project
Success (PoPS)
Introduction

Project Performance Manager
 Focused on Business
Requirements
 How Much and When
 Responsible for designing and
operating the project control
system to manage work that
produces the system
Systems Engineer
 Focused on Business
Requirements
 How Much and When
 Focused on Business Solution
 What and How
 Responsible for Defining,
Designing, and Delivering the
Solution
Introduction

Integrating these Two Perspectives [98]
20
WHEN
Project
Schedule
(IMP/IMS)
HOW MUCH
Cost
Breakdown
Structure
(CBS)
WHY
Vision
Business
Case
Risk Register
WHO
WHAT
Product
Breakdown
Structure (PBS)
How
Work
Breakdown
Structure
(WBS)
Introduction

Start with the end in
mind ‒ Stephen Covey
…
What knowledge, skills,
and experiences will
we leave this workshop
with?
Even though these
may be restating the
obvious, we need to be
on the same page to
successfully integrate
Systems Engineering
and Project
Management.
Terminal
Learning
Objectives
(TLO’s)
TLO’s

TLO’s for the Workshop
We will Learn …
1. The Immutable Principles of Program
Performance Management (PPM) and Systems
Engineering (SE).
2. How integrating the principles of SE and PPM
creates an Integrated Project Performance
Management System (IPPMS)
3. The Practices and Processes of the IPPMS
4. How the IPPMS can measurably contribute to
increasing the Probability of Program Success
(PoPS)
22
TLO’TLO’s

The purpose of all
processes, their
application and
continuous
improvement is to
Increase the Probability
of Project Success
Setting the
Stage for
WHY we
Need an
Integrated
Project
Performance
Management
TLO’s

Projects are
“One-Off”
Events.
You’ve got one
chance to get it right.
What’s your
Probability
Of
Success?

Capabilities Requirements Plans Execution  Continuous Risk Management
† A Concept of Operations (ConOps) describes the characteristics of a proposed system from the viewpoint of an individual who will use
that system. It is used to communicate the quantitative and qualitative system characteristics to all stakeholders.
The 4  1 Elements
Needed To Increase
Project’s Probability of
Success

How
Where
When
Who
Why
What
Identify Needed
Capabilities
Establish a
Performance
Measurement
Baseline (PMB)
Execute the
Performance
Measurement
Baseline
Capabilities
Based Plan
Operational
Needs
Earned Value
Performance
0% /100%
Technical
Performance
Measures
System Value
Stream Technical
Requirements
Identify
Requirements
Baseline
1
2
3
4
Technical
Performance
Measures
PMB
Continuous Risk Management Process
Changes to
business strategy
Changes to
requirements
Changes to
project plan

But First, Some History of the
SE / PPM Integration Problem
 Systems Engineering, Project Management, and
Program Management evolved from similar roots
during World War II. [89], [90]
 Program and Project Management and Systems
Engineering are Different
 Program Management and Project Management not
universally defined
 Systems Engineering, is also not universally defined
 Lack of an Integrated Planning system is a primary
source of disconnect between SE and PPM
 Integration of PPM an SE is difficult, but not
Impossible [1]
 Integrating roles and cultures creates other issues …
27
TLO’TLO’s

The Hypothesis
A Hypothesis of the Workshop
 When PPM and SE organizations are separate but equal,
using separate processes, stove pipes are created [88]
 It is difficult to close the gaps between these stove pipes by
changing the Cultural, another approach is needed.
28
Change Process ‒ Remove Stove Pipes ‒ Integrate SE
and PM
TLO’TLO’s
 The stove pipe gaps between Systems Engineering and
Project Management can be closed with a shared Integrated
Project Performance Management Process (IPPMS).

The Core Failures Resulting from the
Separate but Equal Paradigm [22], [23], [26], [27],
[28]
29
 Capabilities as
Scenarios in ConOps
 Analysis of
Alternatives (AoA)
 Established MOE’s
MOP’s, KPP’s and
KSA’s
 Assessment
Technical and
Operational Needs,
Cost, and Risks to
needed Capabilities
 Cost, Schedule,
Performance (CSP)
measures needed to
manage delivery of
Requirements
 Risk Management of
C,S, and P
 Physical % Complete
 EAC, ETC, TCPI
 CSP Margin
management
TLO’TLO’s

Components Integrated into a
Integrated Project Performance Management
System
 Technical and Programmatic management of work
processes
 Defining Technical and Programmatic components
of the Integrated system
 Defining interfaces and interactions between the
Technical and Programmatic components
 Implementing deliverables built from the
components
 Integrating the deliverables into a System to
deliver needed Capabilities
 Confirming the Capabilities meet the MOE’s,
MOP’s, TPM’s, KPP’s and KSA’s
30
TLO’TLO’s

A Process‒Centric Solution to the
Organizational Problem [22], [23], [26], [27], [28]
31
PM Processes
 Cost
 Schedule
 Technical Performance
 Risk Management
SE Processes
 Mission and Vision
 Effectiveness
 Capabilities
 KPP’s / KSA’s
TLO’s

Overview of the Integration [28]
32
TLO’s

Integrating
Project Performance Management &
Systems Engineering [26]
 The Integrated PPM and SE delivers an …
 Increased likelihood of efficiency and
effectiveness of the project’s resources and
funding
 Improved transparency between SE and PPM
efforts, common understanding, and adaptability
to change
 Without this integration, PPM and SE
disciplines are not well aligned in their
objectives and incentives leading to …
 Lack of efficiency
 Duplication of effort
 Working at cross purposes
33
TLO’s

Practices of an
System
34
Practice Systems Engineering Project Management IPPMS
Define needed
Capabilities of the
System to accomplish
the mission or deliver
Business Value
Increasing maturity of
Deliverables defined in
the Concept of
Operations
Develop the Integrated
Master Plan, showing
the Program Events,
Significant
Accomplishment, and
Accomplish Criteria for
each Capability
Connect the IMP with
the Concept of
Operations where the
Capabilities are defined
Identify the Measures of
Effectiveness (MOE)
and Measures of
Performance (MOP)
The operationally
relevant and measurable
MOE’s and the MOP’s
required to satisfy the
MOEs from WBS for
each deliverable
Integrated Master Plan,
with Significant
Accomplishment and
Accomplishment Criteria
define the MOPs and
MOEs
Connect the MOEs and
MOPs with the TPMs,
KPPs and KSA’s in a
vertical and horizontal
trace from top to bottom
Identify reducible and
irreducible uncertainties
that create risk
Identify risks to
accomplishing MOEs
and MOPs
Identify risks to
accomplishing Technical
Performance Measures
Fully integrated risk
management and
margin management
activities vertically and
horizontally
TLO’s

Processes of an
System
35
Process Systems Engineering Project Management IPPMS
Managing work
processes
INCOSE Vee
Integrated Master Plan
Integrated Master
Schedule
Program Events,
Significant
Accomplishment, and
connected to steps in
Vee
Defining components of
the system
ConOps [89], Capabilities
Based Planning
Work Breakdown
Structure for
Deliverables
WBS elements
connected to delivered
Capabilities
Defining interfaces
between the
components
Design Structure Matrix (DSM) showing dependencies between deliverables
Implementing
deliverables from the
components
Capabilities Breakdown
Structure in sysML
Work Packages and
Tasks in the Integrated
Master Schedule
Physical Percent
Complete at SA and PE
level showing
compliance with MOE
and MOP maturity
assessments
TLO’s

The Integration of Systems Engineering and
Project Performance Management Share
Technical Measures [71]
36
TLO’s

With the outcomes of the SE + PPM
Workshop, we will Understand the Combined
Solution based on …
 Define Framing
Assumptions for
connecting SE and PPM [95]
 Define units of measure
found in SE and PPM
supporting each framework
 Identify methods for
integrating each framework
to enhance Probability of
Program Success
37
 Apply Principles and
Practices to a System of
Systems Unmanned Aerial
Vehicle
 Identify artifacts of
individual Processes
 Define the practical
measures needed to
increase the Probability of
Program Success (PoPS)
 Define the Essential Views
of Integrated Project
Performance Management
System (IPPMS)
Principles Practices
+
TLO’s

Motivation for Integrating SE and PPM
starts with 4 Known Root Causes of Project
Failure
38
Unrealistic Performance
Expectations, with missing Measures
of Effectiveness (MOE) and Measures
of Performance (MOP).
Unrealistic Cost and Schedule
estimates, based on inadequate risk
adjusted growth models.
Inadequate assessment of risk and
unmitigated exposure to these risks
without proper handling plans.
Unanticipated technical issues
without alternative plans and solutions
to maintain effectiveness of the
product or service.
Unanticipate
d Growth of
Cost and
Schedule
“Borrowed” with permission from
Mr. Gary Bliss, Director Performance
Assessment and Root Cause
Analyses, Office of Assistant
Secretary of Defense for Acquisition,
Technology, and Logistics.
TLO’s

Core Project Failure Root Causes in the
Systems Engineering Paradigm
 Inadequate understanding of the requirements
 Lack of systems engineering, discipline, and
authority
 Lack of technical planning and oversight
 Stovepipe developments with late integration
 Lack of subject matter expertise at the
integration level
 Lack of availability of systems integration
facilities
39
TLO’s

Core Project Failure Root Causes in the
Project Management Paradigm (Continued)
 Incomplete, obsolete, or inflexible
architectures
 Low visibility to risk
 Over Estimates of Technology maturity
 Failure to measure Physical Percent Complete
 Failure to identify reducible and irreducible
uncertainty and missing risk management
processes to handle resulting risk
40
TLO’s

Common Goals ‒ Different
Perspectives
 Capabilities Based Planning
 Concept of Operations
(ConOps)
 From the ConOps, produce
an Integrated Master Plan
(IMP) with:
 Measures of Effectiveness
(MOE)
 Measures of Performance
(MOP)
 Key Performance Parameters
(KPP)
 Integrated Master Schedule
(IMS) with:
 Technical Performance
Measures (TPM)
 Quantifiable Backup Data
(QBD)
 Work (WBS), Cost (CER), Risk
(RBS), and Organizational
Breakdown Structure (OBS)
 Risk Management
 Cost, Schedule, Risk
Management integrated in a
single Performance
Measurement Baseline
41
System Engineering Project Management
Both SE and PPM seek successful outcomes ‒ completing the project
to produce a new or enhanced system, for the needed cost, on the
needed date.
TLO’s

Proper System Requirements
Attributes
42
Attribute Definition
Correct
Every requirement represents something that is required for
the system to be built.
Unambiguous
Every requirement has only one interpretation and includes
only one requirement (unique).
Complete
Requirements possesses these qualities:
1. Everything it is supposed to do is included.
2. Definitions of the responses of software to all situations are
included.
3. All pages are numbered.
4. No sections are marked “To be determined.”
5. Is necessary
Verifiable Every requirement is verifiable.
Consistent
1. No requirement that conflicts with other preceding
documents,
2. No subset of requirements are in conflict.
TLO’s

Proper System Requirements Attributes
(Continued)
43
Understandabl
e by Customer
There exists a complete unambiguous mapping between
the formal and informal representations of the
requirements.
Achievable
The designer should have the expertise to assess the
achievability of the requirements, including subcontractors,
manufacturing, and customers/users within the constraints
of the cost and schedule life cycle.
Design
Independent
The requirements does not imply a specific architecture or
algorithm.
Concise
Given two requirements for the same system, each
exhibiting identical level of all previously mentioned
attributes—shorter is better.
Modifiable
The structure and style are such that any necessary
changes to the requirement can be made easily,
completely, and consistently.
TLO’s

Proper System Requirements Attributes
(Continued)
44
Traced
Origin of each requirement is clear and traceable to a
document, design, or regulation.
Traceable
Requirements are written in a manner that facilitates the
referencing of each individual requirement stated therein.
Annotated
There is guidance to the development organization such
as relative necessity (ranked) and relative stability.
Organized Requirements contained are easy to locate.
TLO’s

The World of Engineered Systems
45
TLO’s

The World of Project Management [151]
46
TLO’s

These Individual Processes must be Assembled
into an
System47
TLO’s

Let’s Sort Out the Complexity of the
Integration of Systems Engineering and
Project Management Starting with General
Management [119]
TLO’s

Work
Breakdown
Structure
defines
deliverables
Needed
Capabilities
Requirements
Derived from
Capabilities
Work Packages
to produce
deliverables in
the WBS terminal
nodes
Credible Work
Packages
sequences to
produce
delivered value
needed by the
business
Risk mitigation
or retirement
shown in the
schedule
Formal risk
management in
RM Tool
All
programmatic
data under
change control
TLO’s

Keeping track of Live
Stock is a traditional
role for horse‒back
staff and their dogs.
This is expensive, risky,
error prone, and
requires supporting
facilities for the horses
and dogs as well as the
Cow Boys.
A Sneak
Peek
at our
Case Study
A Livestock
Counting System
using an
Unmanned Aerial
Vehicle50 PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia

The Notional Project
51
 We work for an
Unmanned Aerial Vehicle
(UAV) firm entering the
ranching and farming
market place.
 We know how to build
complex equipment,
including flying machines,
electronics, and training
systems for government
agencies.
 We now want to do the
Notional
Project

Our Case Study Project Will Be …
 A System of Systems, to provide live stock
counting capabilities on Colorado ranch
 Four components are:
 Command and Control ‒ what we want the UAV
to do an when we want it to do it
 Airborne Sensors ‒ what sensors needed in
needed spectrum, for different times of the year
 Ground sensors ‒ for collection of data at entry
and exit points of the pastures
 Animal Sensors ‒ to augment the airborne and
ground sensors
 We’ll focus on the airborne system for this
52
Notional
Project

As to methods there
may be a million and
then some, but
principles are few. The
man who grasps
principles can
successfully select his
own methods. The man
who tries methods,
ignoring principles, is
sure to have trouble.
— Harrington
Emerson, August 2,
1853 to September
2, 1931[91]
Immutable
Principles
Principles

Max Wideman Suggests a Principle of
Project Success …
 Express a general or fundamental truth [or] a basic
concept
 Makes for a high probability of project success. The
corollary is that the absence of the condition will
render project success on a majority of the key criteria
as being highly improbable.
 Provides the basis for establishing logical processes
and supporting practices that can be proven through
research, analysis, and practical testing.
 Be universal to all areas of project management
application.
 Be capable of straightforward expression in one or
two sentences.
 Be self-evident to experienced project management
personnel.
54
Immutable
Principles

As to methods there
then some, but
man who grasps
principles can
who tries methods,
— Harrington
Emerson, August 2,
1853 to September
2, 1931[92]
Principles of
General
Management
GM Principles

Action Outcome
Implement Strategy
Ensure Capabilities
Prioritize Problems And Solutions
Identify Redundancies
Deliver Solutions
† “Capabilities‒Based Planning: A Methodology for Deciphering
Commander’s Intent,” Peter Kossakowski, 10th ICCRTS, Track 12
GM Principles

General Management and the
Cost and Schedule Framework
57
Program
Manager
 Establish initial
program
baseline
 Develop cost
control process
(EVM)
Execution
Management
 Execute processes
 Maintain baselines
 Use CAIV and SAIV
in Trade Studies
and Decision
making
Monitor
 Take
performance
measures
 Conduct cost
review
Control
 Analyze and Assess
 Corrective and Preventive
actions
 Architecture and PPM decisions
based on study results
 Anticipate future trends
Status
Lessons
Learned
Processes
Baselines
Actions
GM
Principles

General Management is a Risk
and Opportunity Framework [71], [73,], 75], [89], [144]
58
Risk and
Opportunity
Management
Program
Management
Systems
Engineering
Management
Customer
Management
Operations
Management
Subcontract
Managemen
t
Risk and
Opportunity
Management
GM
Principles

A Simple Question ‒ What is a
System? [41]
59
An aggregation of System
Elements and Enabling System
Elements to achieve a given
purpose or top provide a
capability.
Provide the means for
putting a needed
Capability into service,
keeping it in service, or
ending its service.
Configuration items,
subsystems, segments,
assemblies, or parts
that make up the
system.
GM
Principles

General Management is a
Systems Engineering Framework [71], [73,], [75],
[89], [144]
60
Systems
Engineering
System
Behavior
Mission
Environment
Scope of
Effort
Scale of
Effort
Stakeholder
Involvement
Desired
Outcomes
GM
Principles

General Management Principles
A Summary
 General Management is a participant in
Systems Engineering
 Technical ‒ Capabilities Elicitation from Customer
 Programmatic ‒ Contracting for delivering
Capabilities
 Financial ‒ Budgeting and funding work
 Customer ‒ product and customer facing
management
61
GM
Principles

As to methods there
then some, but
man who grasps
principles can
who tries methods,
— Harrington
Emerson, August 2,
1853 to September 2,
1931[92]
Principles of
Systems
Engineering
SE Principles

Systems Engineering is [156] …
Led by Systems Engineers, where:
 All functions play a role through an Integrated
Process and Product Development (IPPD);
 With the Functions rigorously applied across the
program;
 To provide the technical glue allowing separate
design disciplines and subsystems function
together to provide an integrated system
performing a specific job.
63
SE
Principles

The Systems Viewpoint
 Design Engineers ‒ view the system from the inside
 Concerned with other system elements only as they affect
their own design task; but not necessarily how theirs may
affect others
 Systems Engineers ‒ view the system from the
outside
 Concerned with the effect of all system elements as they
affect overall system design / performance / cost /
schedule
 Project Performance Management ‒ views the
system as
 Project planning of the objectives, roles and responsibilities
for delivering those objectives
64
SE
Principles

System Engineering Connects the Dots
Between all the Project Information [178]
65
 Stakeholder Needs
 Use Cases
 Operational Scenarios
 Stakeholder
Requirements
 System Requirement
 Interfaces
 System Architectures
 Verification Objectives
 Test Cases
SE
Principles

Horizontal Integrator(s) for each subsystem and its
integration
A Systems Engineering Organization for our
Cow Counting Project
66
Chief
Systems
Engineer
System
Design and
Integration
Lead
Payload
Systems
Lead
System
Requireme
nt Lead
Mission
Operations
Lead
System
Verification
and
Validation
Lead
Risk
Manager
System
Design and
Integration
Lead
System
Design and
Integration
Lead
System
Design and
Integration
Lead
System
Design and
Integration
Lead
System
Design and
Integration
Lead
SE
Principles
…

Systems Engineering Leads the Technical
Execution of the Project [140]
67
Project Management Systems Engineering
Management
Planning
 Project Management Plan (PMP),
Integrated Master Plan (IMP), and
Integrated Master Schedule (IMS)
 Systems Engineering
Management Plan, technical
elements of the IMP/IMS,
technical processes
Organizing
 Organizational Breakdown
Structure (OBS)
 Work Breakdown Structure (WBS)
 Systems Engineering
Organization Chart
 Working Groups
 Review
 Risk Management
Staffing
 Project Manpower Plan, Roll-
on/Roll-off, Project Office Staff
 SE recruiting, training, team
building
Controlling
 Earned Value Management,
project reviews, Monthly
Management Reviews
 EVMS, Engineering Change
Board, Technical Metrics, Baseline
Control, System Design Meetings
 Policies, Procedures, Training,  Requirements development,
SE
Principles

The Hypothesis of Systems
Engineering [108]
 If a solution exists for a specific context, then
there exists at least one ideal Systems
Engineering solution for that specific context.
 System complexity greater than or equal to the
ideal system complexity is necessary for fulfill
all system outputs.
 Key Stakeholder preferences can be
represented mathematically.
 The real physical system is the perfect model
of the system.
68
SE
Principles

Systems Engineering Principles [108]
Systems Engineering …
1. Integrates the systems and the disciples considering
the budget and schedule constraints
2. Confirms Complex Systems build Complex Systems
3. During the development phase, is focused on
progressively deeper understanding of the interaction,
sensitivities, and behaviors of the system
4. Has a critical role through the entire system life‒cycle
5. Based on a middle range set of theories
6. Maps and manages the disciple interactions present in
the decision-making process
69
SE
Principles

Systems Engineering Principles [108]
Systems Engineering …
7. Assures decision quality depends on coverage of
system knowledge present in the decision-making
process
8. Requires Policy and Law are properly understood for
all system functions and interactions in the operational
environment
9. Assures decision made under uncertainty account for
risk
10. Verifies the system’s value is demonstrated for the
stakeholders
11. Validates the system’s value is demonstrated for the
stakeholder
70
SE
Principles

A Taxonomy of ISO System Engineering
Standards
71
SE
Principles

The Composite Elements of
Systems Engineering
72
SE
Principles

73
Process
Inputs
 Customer
Needs
 Technology
base
 Program
Decision
Requiremen
ts
 Requiremen
ts applied
through
Specificatio
ns and
Standards
Trade Studies
 Effectiveness
 Risk
 Interfaces
 Data
 Configuration
Process Outputs
 Decisions
 System
Configurations
 Specifications and
Baseline
† SMC Systems Engineering Handbook, Version 3, FigurePGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
Technical
Requirements
Loop
System Design Loop
System Verification Loop
Project
Control
Loop
SE Principles

74 SMC Systems Engineering Handbook, VersioPGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
SE Principles

These Five Immutable
Principles, and their
Five Processes, and
Ten Practices (that
we’ll apply in the case
study) provide
actionable information
to the decision makers
to increase the
Probability of Project
Success. [59]
Five
Immutable
Principles of
Project
Success

1. What Does DONE Look
Like?
2. How Do We Get to
DONE?
3. Is There Enough Time,
Money, and Resources,
To Get to DONE?
4. What Impediments Will
Be Encountered Along
The Way to DONE?
5. What Units of Measure
are used to confirm
Progress To Plan Toward
All Successful Projects Require Credible Answers
To These 5 Immutable Principles [59] …
PPPM Principles

16 Elements of Program Management Used
to Implement the Five Immutable Principles[113]
77
Business Enablers
Program Enablers
Program Process Capabilities
15: PPM Process
Management
16: PPM
Development and
Succession
11:
Organization/IPD
12: Customer
Partnership
13: Program
Review Process
14: Configuration/
Data Management
2: Program Planning
3: Performance
Management
4: Sub-Contract
Management
5: Follow-On
Business
Development
9: Financial
Management
10: Risk
Management
7: Requirements
Management
8: Schedule
Management
6: Earned Value
Management
1: Program Management
Involvement in Proposal
PPM
Principles

The 4+1 Questions
Every Successful
Project Must Answer
Capabilities Requirements Plans Execution
2. What technical and operational requirements are
needed to deliver these capabilities?
1. What capabilities are needed to fulfill the Concept of Operations†,
the Mission and Vision, or the Business System Requirements?
3. What schedule delivers the product or
services on time to meet the requirements?
4. What periodic measures of
physical percent complete
assure progress to plan?
What impediments to success, their mitigations, retirement plans,
or “buy downs are in place to increase the probability of success?”
+ Continuous Risk Management
† A Concept of Operations (ConOps) describes the characteristics of a system from the point of view of an individual who will use
that system. It is used to communicate the quantitative and qualitative system characteristics to all stakeholders.





Performance–Based Project Management®, Copyright © Glen B. Alleman, 2012 - 2018
PPPM Principles

79
Identify Needed
Capabilities
Establish a
Performance
Measurement
Baseline
Execute the
Performance
Measurement
Baseline
Capabilitie
s Based
Plan
Operational
Needs
Earned
Value
Performance
0% /100%
Technical
Performanc
e Measures
System Value
Stream
Technical
Requiremen
ts
Identify
Requirements
Baseline Technical
Performanc
e Measures
PMB
Changes to
Needed Capabilities
Changes to
Requirements Baseline
Changes to
Performance Baseline





Performance–Based Project Management®, Copyright © Glen B. Alleman, 2012 - 2018PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia79
PPM Principles

80
Define the set of capabilities needed to achieve the project objectives or the particular
end state for a specific scenario. Using the Concept of Operations (ConOps), define
the details of who, where, and how this capability is to be accomplished, employed,
and executed.

Define the technical and operational requirements for the system capabilities to be
fulfilled. First, define these requirements in terms isolated from any implementation
details. Only then bind the requirements with technology.

Build a time–phased network of work activities describing the work to be performed,
the budgeted cost for this work, the organizational elements that produce the
deliverables, and the performance measures showing this work is proceeding
according to plan.

Execute work activities, while assuring all performance assessment represent 100%
completion before proceeding. This means – No rework, no forward transfer of
activities to the future. Assure all requirements are traceable to work & all work is
traceable to requirements.

Apply the processes of Continuous Risk Management for each Performance–
Based Project Management® process area to: Identify, Analyze, Plan, Track,
Control, and Communicate programmatic and technical risk.

Performance–Based Project Management®, Copyright © Glen B. Alleman, 2012 - 2018PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia80
PPM Principles

Identify Needed System Capabilities
81
What Should We Do?
Where Are We Now?
Abstracted from:
“Capabilities‒Based Planning – How It Is
Intended To Work And Challenges To Its
Successful Implementation,” Col. Stephen K.
Walker, United States Army, U. S. Army War
College, March 2005
PPM
Principles

Identify System Requirements†
82
† Systems Requirements Practices, Jeffery O. Grady, McGraw Hill, 1993
PPPM
Principles

Establish Three Elements of the
Performance Measurement Baseline
83
Cost Baseline
Schedule Baseline
Technical Baseline
Determine
Scope and
Approach
Develop
Technical
Logic
Develop
Technical
Baseline
Develop
WBS
Define
Activities
Estimate
Time
Durations
Sequence
Activities
Finalize
Schedule
Identify
Apportioned
Milestones
Determine
Resource
Requireme
nt
Prepare
Cost
Estimate
Resource
Load
Schedule
Finalize
Apportioned
Milestones
Determine
Funding
Constraint
s
Approve
PMB
Perform
Functional
Analysis
PPPM
Principles

Execute Performance Measurement
Baseline
84
PPPM
Principles

Perform Continuous Risk Management
85
Analyze
Plan
Track
Control
Identify Risks, Issues, and
Concerns
Evaluate, classify, and
prioritize risks
Decide what should be done
about each risk
Monitor risk metrics and
verify/validate mitigations
Make risk decisions
Subproject and partner
data/constraints, hazard
analysis, FMEA, FTA, etc.
Risk data: test data,
expert opinion, hazard
analysis, FMEA, FTA,
lessons learned,
technical analysis
Resources
Replan Mitigation
Program/project
data
(metrics
information)
Statement of Risk
Risk classification, Likelihood
Consequence, Timeframe
Risk prioritization
Research, Watch (tracking
requirements)
Acceptance Rationale,
Mitigation Plans
Risk status reports on:
Risks
Risk Mitigation Plans
Close or Accept Risks
Invoke contingency plans
Continue to track
PPM
Principles

Measures Showing Progress from
Mission Capabilities to Project Done
86
“Coming to Grips with Measures of
Effectiveness,” N. Sproles, Systems
Engineering, Volume 3, Number 1, pp. 50–58
MoE
KPP
MoP TPMMission Need
Acquirer Defines the Needs and Capabilities
in terms of Operational Scenarios
Supplier Defines Physical Solutions that
meet the needs of the Stakeholders
Operational
measures of success
related to the
achievement of the
mission or
operational
objective being
evaluated.
Measures that
characterize
physical or
functional attributes
relating to the
system operation.
Measures used to
assess design
progress,
compliance to
performance
requirements, and
technical risks.
PPM
Principles

Tangible Benefits of This Approach
87
Project Performance
Management
Benefits to the Customer
Program, Planning, and Controls
Rapid creation of the risk adjusted
Performance Measurement Baseline.
Earned Value Management
ANSI‒748D compliant processes, tools, and
training.
Programmatic and Technical Risk
Management
Credible integrated risk management process
guided by DoD, DOE, AACE, and PMI
standards.
Management Process Improvement
Value focused organizational change
management.
Program Performance Assessment
Unbiased External Independent Reviews
(EIR).
Proposal support – Management
Volume
IMP/IMS, Basis of Estimate (BoE), and Risk
sections.
PPM
Principles

Performance Target(s)
 Cost
 Schedule
 MOP, MOE, TPM,
KPP
Project Performance Management Actions
 Schedule Management
 Cost Management
 Specialty Engineering Management
 Contractual Deliverables Management
Performance
Variances
 Cost
 Schedule
 MOP, MOE, TPM,
KPP
 Cost
 Schedule
 MOP, MOE, TPM,
KPP
Deliverabl
es
Each process and measure operates in the presence
of uncertainty, requiring estimates of all control
parameters
Project Performance Control
Loop
PPM Principles

Relate time phased
budgets to specific
contract tasks
Enable statistical
estimation of
completion cost
Track and
monitor discrete
project metrics
Communicate
project status
Provide
quantitative data
for decision making
Provide a
documented project
performance trace
Alert project managers
to potential schedule
and cost risk impacts
Provide managers
information at a practical
level of summarization
PPM Principles

Two distinct paradigms
can be integrated if the
processes they are
based on can be
connected in a logically
consistent manner.
‒ Plato’s Dualism in
Phaedo
Basis for
Integrating
Systems
Engineering
with
Project
Management

Seven Key Principles of Program and
Project Success [125]
1. Establish a clear and compelling vision.
2. Secure sustained support “from the top”.
3. Exercise strong leadership and management.
4. Facilitate wide open communication.
5. Develop a strong organization.
6. Manage risk.
7. Implement effective systems engineering and
integration.
91
Integrating
SE&PPM

Some Simple Definitions of PPM and
SE
 Plan and coordinate
work activities needed to
deliver a satisfying
product.
 Monitoring
accomplishments of
project objectives.
 Control and implement
corrective and
preventative action that
are impediments to
project progress to plan.
 The art and science of
developing operable
systems capable of
meeting requirements
within opposed
constraints.
 Seek a safe and
balanced design in the
face of opposing
interested and multiple,
sometimes conflicting
constraints.
92
Project Performance
Management
Systems Engineering
Integrating
SE&PPM

Using the Vee to Integrate SE and
PPM [98]
93
Organizational Drivers Strategic Drivers
Project Controls Project Delivery
Integrating
SE&PPM

Separate but Equal Components of an Integrated
Project Performance Management System
(IPPMS)
94
+
+
Performance Target(s)
§ Cost
§ Schedule
§ MOP, MOE, TPM, KPP
Project Performance Management Actions
§ Schedule Management
§ Cost Management
§ Specialty Engineering Management
§ Contractual DeliverablesManagement
Performance Variances
§ Cost
§ Schedule
§ Cost
§ Schedule
Deliverables
Each processand measure operatesin the presence of
uncertainty, requiring estimatesof all control parameters
Control Loop
ProcessInputs
§ Customer
Needs
§ Technology
base
§ Program
Decision
Requirements
§ Requirements
applied
through
Specifications
and
Standards
Trade Studies
§ Effectiveness
§ Risk
§ Interfaces
§ Data
§ Configuration
ProcessOutputs
§ Decisions
§ System
Configurations
§ Specificationsand
Baseline
RequirementsLoop
Design Loop
Verification Loop
Control Loop
Integrating
SE&PPM

Separating What from How is a Critical
Success Factor for the IPPMS [88]
 Understanding the Problem (What) is
independent of the managing the development
of the solution (How)
 For any given problem (What) there are many
possible solutions (How)
 The separation of concerns is the basis of
good Systems Engineering
 Seldom are systems built from scratch
 Most systems are implemented using new
technology or integrating previous systems into
the new system
95
Integrating
SE&PPM

What is an Integrated Project Performance
Management System (IPPMS)?
96
Integrating
SE&PPM

Success Starts with Capabilities‒Based
Planning
 What capabilities do we need to posses to
accomplish our mission?
 What are the Measures of Effectiveness and
Measures of Performance for these
Capabilities?
 What Technical Performance Measures are
needed for each deliverable that fulfills the
Measures of Effectiveness and Measures of
Performance?
97
Integrating
SE&PPM

Units of Measure in the Systems
Engineering Domain [96]
 Measures of Effectiveness (MOE)
 Measures of Performance (MOP)
 Key Performance Parameters (KPP)
 Key Systems Attributes (KSA)
 Technical Performance Parameters (TPM)
98
Integrating
SE&PPM

Measures of Effectiveness (MOE)
99
Measures of Effectiveness …
 Are stated in units meaningful to the buyer,
 Focused on capabilities independent of any
technical implementation, and
 Are connected to the mission success.
“Technical Measurement,” INCOSE–TP–2003–020–01
Integrating
SE&PPM

Measures of Performance (MOP)
100
Measures of Performance are …
 Attributes that assure the system has the
capability and capacity to perform the need
Capabilities,
 An assessment of the system that assures it
meets design requirements to satisfy the MoE.
Integrating
SE&PPM

Key Performance Parameters (KPP)
101
Key Performance Parameters …
 Have a threshold or objective value,
 Characterize the major drivers of performance,
 Are considered Critical to Customer (CTC).
Integrating
SE&PPM

Key System Attributes (KSA)
 Key System Attributes …
 Provide decision makers with an additional level of capability
prioritization below the KPP.
 A KSA does not have to be related to a KPP and there is no
implication that multiple KSA’s equal a KPP.
102
Integrating
SE&PPM

Technical Performance Measures
(TPM)
103
Technical Performance Measures …
 Assess design progress,
 Define compliance to performance
requirements,
 Identify technical risk,
 Are limited to critical thresholds,
 Include projected performance.
Integrating
SE&PPM

Technical Performance Measurement
104
Integrating
SE&PPM
Variance
Planned Value
Planned Profile
Current Estimate
Milestones
Threshold
Upper Limit
Lower Limit
MeanToBetweenFailure
Time = Program Maturity
Achieved to Date
TPM

105
As an Example, Let’s Start with
and
For a New Stylish Coffee Cup
Integrating
SE&PPM

MOE and MOP Starting Points of Success
of a Stylish Coffee Cup [167]
106
Nee
d
Wan
t Specification
N Weight < 120g
W Weight < 100g
N Non-Porous
N Thermal Conductivity <
2.5W/m.K
W < 1.4 W/m.K
W Surface Finish <±0.04mm
N Produce > 5000 items/day
W > 8000 items/day
N Rigid solid @ ~110℃
W Reflective coating
W Volume ~ 280‒350ml
0.4
0.3
0.3
0.3
0.7
0.4
0.3
0.3
0.4
0.3
0.3
0.16
0.12
0.12
0.09
0.21
0.12
0.09
0.09
Measures of PerformanceMeasures of Effectiveness
=
=
=
=
=
=
=
=
Integrating
SE&PPM

These Two ½’s Make a Whole
107
+
=
Integrating
SE&PPM

The Challenge of Integrating SE and PPM is
to Increase the Probability of Project
Success108
Integrating
SE&PPM

Increasing the
Probability of Project Success Start By …
109
… Rounding up Technical and
Programmatic Cats and
Constructing an Integrated
Project Performance
Management System
(IPPMS)
Integrating
SE&PPM

“Complex systems usually come
to grief, when they do, not
because they fail to accomplish
their nominal purpose.
Complex systems typically fail
because of the unintended
consequences of their design …
“I like to think of System
Engineering as being
fundamentally concerned with
minimizing, in a complex artifact,
unintended interactions between
elements desired to be separate.
Essentially, this addresses
Perrow’s [181] concerns about
tightly coupled systems. System
engineering seeks to assure that
elements of a complex artifact
are coupled only as intended.”
– Michael Griffin, NASA
Administrator, Boeing Lecture,
Purdue University, March 28,
2007
A System of
Systems
(SoS)
Framework
SoS

At The Top of the SoS is Where We Define
Needed Capabilities
111
Systemof
Systems

An Actual System of Systems (SoS)
112
Systemof
Systems

A System of Systems Process Model †
113
† Roving UAV IED Interdiction System, MSSE Cohort 311‒093A, Naval Post Graduate School,
March 2001
Systemof
Systems

A Look Ahead to a System of Systems
114
Ground Based Sensors
Command and Data Center
Mobile
Sensors
UAV with Airborne Sensors
Systemof
Systems

Learning’s from Prior SoS UAV
Projects
115
 Challenging requirements downstream raise unavoidable complexity
throughout program lifecycle
 Lack of requirements rationale permit unnecessary requirements
 Requirements volatility creates moving target for designers
Requirements
Complexity
 Engineering trade studies not done: a missed opportunity
 Architectural thinking/review needed at level of systems, hardware, software,
and operational processes
System Level
Design and
Analysis
 Inadequate system architecture and poor implementation
 General lack of design patterns (and architectural patterns)
 Coding guidelines help reduce defects and improve static analysis
 De‒scopes often shift complexity to operations
System
Complexity
 Growth in testing complexity due to Cross‒Cutting functions
 More components and interactions to test
 COTS products a mixed blessing
V&V Complexity
 Shortsighted decisions make operations unnecessarily complex
 Numerous “operational workarounds” raise risk of command errors
Operational
Complexity
Systemof
Systems

Connecting the Dots Between
Systems Engineering and Project Performance
Management
116
SE Processes PPM Processes
5 Immutable
Principles of Project
Success
Systemof
Systems

It has been shown that
we can increase the
Probability of Project
Success (PoPS), by
integrating Systems
Engineering and
Project Performance
Management
Relationship
Between
Systems
Engineering
and
Project
Performance
Management117 PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
SE+PM

Relationships Between PPM and SE[179]
118
Cost Management
Procurement Management
Project Charter
Schedule Management
Scope Management
Quality Management
Stakeholder Management
Resource Management
Communications Management
Life Cycle Cost and Disposal
Requirements Management
Integration Management
Stakeholder Analysis
Personnel Development
Architecture Synthesis
Trade Studies
Test, Verification & Validation
Concept of Operations
Mission and
Logistics Engineering
Requirements
Synthesis and Analysis
Process Management
Pre‒Planned
Product Improvement
Interfaces,
Specifications, and Standards
Analysis of Alternatives
Risk Management
Dennis Van Gemert,
Earned Value Management
Work Breakdown Structure
SE+PM

…ilities are Critical Attributes of
All Systems [32], [180]
119
Repairability Maintainabilit
y Interoperability
Durability
Manufacturability
Safety
Testability
Usability
Sustainability
Quality
Reliability
Scalability
Extensibility
Modularity
Agility
Robustness
Resilience
Adaptability
Flexibility
Evolvability
SE+PM

Fundamentals of Systems Engineering[26]
120
SE+PM

Dimensions of Project Complexity and the
Problems it Creates
121
Project
Dimensio
ns
Uncertainty
Dependency &
Interdependency
ExternalitiesTechnology
Innovation
New to the world
New to the market
Existing Platform
Derivative
Complete Uncertainty
High Uncertainty
Medium Uncertainty
Low Uncertainty
No Understanding
Limited Understanding
Partial Understanding
Full Understanding
No Impact
Limited Impact
Significant Impact
Catastrophic Impact
Low Tech
Medium Tech
High Tech
Inventing New Tech
SE+PM

Risk Management is
Project Management
for Adults
‒ Tim Lister
Risk
Management
of Cost,
Schedule, and
Technical
Performance
Starts
with Systems
Risk

Risk

Managing in the Presence of Reducible and
Irreducible Uncertainty that Created Risk
124
It is moronic to predict without first establishing an error rate for the
prediction and keeping track of one’s past record of accuracy.
— Nassim Nicholas Taleb, Fooled By Randomness
Risk

Core Elements of Project Risk
Management
 The effectiveness of risk management
depends on the people who set up and
coordinate the risk management process.
 On many program’s risk management consists
only of having a policy and oversight.
 If we treat red flags as false alarms rather than
early warnings of danger this incubates the
threats to program success.
 Group think of dominate leaders often inhibits
good thinking about risks.
125
Risk
Risk

All Risk comes from Uncertainty, which
comes in two forms ‒ reducible and
irreducible126
Uncertainty
Irreducible
(Aleatory)
Reducible
(Epistemic)
Natural Variability
Ambiguity about
behavior of
process
UNK UNK
(Ontological
)
Probability of
Event’s
Occurrence
Probabilistic
Impact from Event
Period of
Exposure to Event
Period of
Exposure to
Aleatory process
Risk

 Aleatory Pertaining to stochastic (non‒deterministic) events,
the outcome of which is described using probability.
 From the Latin alea
 For example in a game of chance stochastic variability's are the
natural randomness of the process and are characterized by a
probability density function (PDF) for their range and frequency
 Since these variability's are natural they are therefore irreducible.
 Epistemic (subjective or probabilistic) uncertainties are event
based probabilities, are knowledge‒based, and are reducible
by further gathering of knowledge.
 Pertaining to the degree of knowledge about models and their
parameters.
 From the Greek episteme (knowledge).
Separating these classes helps in design of assessment
calculations and in presentation of results for the integrated
program risk assessment.
127
Aleatory & Epistemic Uncertainty
127
Risk

Technical and Programmatic Risk
128
Technical Risk Management
Tracking and Controlling
Performance Deviations
Deliberating and
recommending a decision
alternative
Risk analysis of decision
alternatives, performing
trade studies and ranking
Proposing and/or
identifying decision
alternatives
Formulation of objectives
Hierarchy and Technical
Identify
Analyze
Plan
Track
Control
Communicate
Risk

Naturally Occurring
Uncertainty in the IMS Creates Risk
 Cost
 Schedule
 Capacity for work
 Productivity
 Quality of results
 Activity correlation
129
Risk

All non-trivial systems
have complexity, are
complex, and may be
complicated.
We need a means to
sort these out and
produce a clear and
concise description of
what the system does,
how we’re going to
develop the system,
and how to measure
our progress toward
DONE
Sorting Out
the
Programmatic
Complexity
Needed to
Address the
Technical
Complexity130 PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
Complex

Sorting Out Complexity Starts with a
Credible Performance Measurement
Baseline (PMB)131
Risk
SOW
Cost
WBS
IMP/IMS
MOE, MOP
TPM, KPP
PMB
❺ Deliverables defined
in the SOW, traced to
the WBS, with
narratives and
Measures of
Performance (MoP)
❹ BCWS at the Work
Package level, rolled to
the Control Accounts
showing cost spreads for
all work in the IMS
❻ Starting with MoP for
each critical deliverable in
the WBS and identified in
each Work Package in the
IMS, used to assess
maturity in the IMP
❶ WBS contains Products
and Processes in a “well
structured” decomposition,
traceable to the deliverables
❷ IMS contains all the
Work Packages,
BCWS, Risk
mitigation plans, with
traces to the IMP
measuring increasing
maturity through
Measures of
Effectiveness (MoE)
and KPPs (JROC and
Program)
❸ Technical and Programmatic
Risks Connected through the
WBS, Risk Register, IMP and
IMS
The PMB is the Document
of Record for the Project
Performance is
Measured through the
PMB
Complex

The WBS is Paramount
 The WBS defines the deliverables and the
supporting processes that produce them
 The WBS Dictionary describes the technical
and operation behaviors that will be assessed
during the development of the deliverables
 The terminal nodes of the WBS define the
deliverables produced by the Work Packages
in the IMS and assessed through the IMP
Accomplishment Criteria (AC)
132
Complex

Integrated Master Plan (IMP)
Integrated Master Schedule (IMS)
 The IMP defines increasing maturity for the
deliverables as the program “moves from left
to right”
 Significant Accomplishments (SA) assessed
with Measures of Effectiveness (MoE)
 Accomplishment Criteria (AC) assessed with
Measures of Performance (MoP)
 Work Packages rolled up the AC’s
 Risks are assigned at all levels of the IMP and
IMS
133
Complex

All Risk Comes from Uncertainty
 Natural occurring (Aleatory) uncertainties
create risk in cost and schedule processes
create risks to completing on time and on
budget
 Event based uncertainties (Epistemic) create
risk that impacts to cost, schedule, and
technical performance
 Epistemic risks are handled through risk
mitigations
 Aleatory risks are handled through in cost,
schedule, and technical performance margin
 To be credible, the PMB must include both
134
Complex

Costs are assigned to Packages of
Work
 Labor and material cost are represented in the
Integrated Master Schedule (IMS) and provide
visibility to the probability of program success
 Variances in labor and material costs are
modeled in the same way as work durations
 Event based risks impact both cost and
schedule and are modeled in the PMB
 Risk retirement cost is allocated for the work
effort in response to Event Based risks
135
Complex

Statement of Work
 Work in the PMB starts with the Statement of
Work and flows through the Work Breakdown
Structure to the Deliverables
 Measures of Effective (MoE) and Measures of
Performance (MoP) defined in the SOW or
WBS Dictionary with Technical Performance
Measures
 Traceability from the IMP to the IMS to all
performance measures in the SOW is the
basis of program performance measurement
 Measures of Physical Percent Complete for
each Deliverable is the Basis of Project
136
Complex

 Key Performance Parameters (KPP) and
Technical Performance Measures (TPM)
define how deliverables comply with the
Statement of Work (WBS) and Concept of
Operations (CONOPs).
 TPMs inform Physical Percent Complete for
cost and schedule measures of delivered
project outcomes.
 TPM, MoE, MoP, and KPPs provide
assessment of the cost and schedule
performance.
137
Complex

The Work Breakdown
Structure is the starting
point for developing all
other elements needed
for the Performance
Measurement
Baseline.
The TSAS WBS is
defined using the
MIL‒STD‒881D
Appendix H for the
UAV.
From this, the details of
the avionics
subsystems will be
used for the
development of the
and Integrated Master
Schedule.
Project
Success
Starts with
the
Work
Breakdown
Structure is
WBS

The WBS is Paramount†
139
Risk & Risk Assessmen
- Level of Detail
- Program Oversight
level of detail
Technical
- TPMs
- Specifications
- Design docs
- Performance char.
Earned Value
Cost and Schedule
Status
† Glen B. Alleman, Institute for Defense Analyses
Gordon Kranz, DOD (OSD) PARCA
Plan and
Schedule
IMP / IMS
WBS

Physical Architecture and the WBS
140
WBS

Systems Architecture Drives
the Work Breakdown Structure
141
WBS

A primary failure mode
of complex programs is
not having a clear,
concise, measurable
definition of Done, in
units of measure
meaningful to the
decision makers
This starts with the
Integrated Master
Plan [112]
The
Description
of DONE
Starts with
the
Integrated
Master Plan
IMP

143
The IMP tells us What Done Looks Like in Units
of Measure Meaningful to the Decision Makers
The Plan describes where we are going, the various paths we can
take to reach our destination, and the progress or performance
assessment points along the way to assure we are on the right path.
These assessment points measures the “maturity” of the product or
service against the planned maturity. This is the only real measure of
progress – not the passage of time or consumption of money.
The Integrated Master Plan (IMP) Is A Strategy For The
Successful Completion Of The Project
IMP

Quick View of Step‒By‒Step IMP
144
❶ Identify Program Events (PE) ‒ Maturity Assessment
❷ Identify Significant Accomplishments (SA) ‒ entry
criteria to the PE’s
❸ Identify Accomplishment Criteria (AC) ‒ exit criteria from
Work Packages
❹ Identify Work Packages needed to complete the
❺ Sequence the Work Packages (WP), Planning
Packages (PP), Summary Level Planning Packages
(SLPP) in a logical network in the IMS.
❻ Adjust the sequence of WPs, PPs, & SLPPs to mitigate
major risks.
IMP

Attributes of the IMS
 Integrated, networked, multi‒layered schedule
of efforts required to achieve each IMP AC
 Detailed tasks and work to be completed
 Calendar schedule shows work completion dates
 Network schedule shows interrelationships and critical
path
 Expanded granularity, frequency, and depth of risk
areas
 Resource loading
 The IMS is vertically traceable to IMP events,
through AC’s and SA’s
145
IMP

The Importance of the IMP
 The IMP is the single most important document to
a program’s success
 It clearly demonstrates the providers understanding of
the program requirements and the soundness of the
approach a represented by the plan
 The IMP/IMS provides:
 Up Front Planning and Commitment for needed
Capabilities from all participants
 A balanced design discipline with risk mitigation
activities
 Integrated requirements including production and
support
 Management with an incremental verification for
informed program decisions
146
IMP

Building the IMP Starts at the RFP with
Systems Engineering Measures
147
Risk Management
IMP

The IMP / IMS Structure
148
IMS
IMP
Describes how program
capabilities will be
delivered and
how these
capabilities will
be recognized
as ready for
ddelivery
Supplemental Schedules (CAM Notebook)
Work Packages and Tasks
Criteria
Accomplishment
Events
or
Milestones
IMP

Vertical and Horizontal traceability of project work
shows increasing maturity as units of
Physical Percent Complete
 Vertical traceability AC  SA  PE
 Horizontal traceability WP  WP  AC
149
Program Events
Define the maturity
of a Capability at a point in time.
Significant Accomplishments
Represent requirements
that enable Capabilities.
Exit Criteria for the Work
Packages that fulfill Requirements
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
package
IMP

The Integrated Master Plan’s role during
Project Execution
150
Program ExecutionPMB for IBRProposal SubmittalDRFP & RFP
Performance Measurement Baseline
Tasks (T)
BOE
% Complete
Statement of Work
Program Deliverables
IMP
Accomplishments (A)
Criteria (C)
EVMS
Events (E)
Budget Spreads by CA & WPCAIV
Capabilities Based Requirements
X BCWS =
Probabilistic Risk Analysis
=
Time keeping and ODC =
Technical Performance Measure
BCWP
ACWP
Cost & Schedule Risk Model
BCWS
D ecreasing techni cal and programmatic ri sk usi ng R i sk Management Methods
IMS
Physical % Complete
Continuity and consistency from DRFP through Program Execution
WBS
IMP

151
 Program Event (PE)
 A PE assess the readiness or completion as a
measure of progress
 First Flight Complete
 Significant Accomplishment (SA)
 The desired result(s) prior to or at completion of an
event demonstrate the level of the program’s progress
 Flight Test Readiness Review Complete
 Accomplishment Criteria (AC)
 Definitive evidence (measures or indicators) that
verify a specific accomplishment has been completed
 SEEK EAGLE Flight Clearance Obtained
F‒22 IMP
Example
IMP

The Integrated Master
Plan (IMP) says what
Done looks like in
measures of increasing
maturity of the
Deliverables, through
assessment of the
Significant
Accomplishments and
Accomplishment
Criteria [111]
5 + 1 Steps
to Building
the IMP/IMS
5 Steps

The IMP Tells Us Where We Are Going
The IMS Tells Us When We Plan To Arrive
Capabilities
June
2004
Contract
Award
Aug 2005
PDR
Jan 2006
CDR
Jan 2005
IBR
Nov 2006
TRR
Mar 2007
1st Flight
5 Steps

IMP and IMS evolves from RFP to Performance
Measurement Baseline, providing traceability from RFP to
delivered products and services
154
IMS
IMP
Describes the
strategy for
successful
program
delivery
Supplemental Schedules
Tasks
Accomplishment
Criteria
Significant
Accomplishments
Program
Events
Program Master Schedule
 Presents the Contract Period of Performance (POP), Program Events (PE), Key
Milestones, Major Program Deliverables, and Reports Progress at a Summary Level
 Includes the First Level WBS and Significant, Measurable Events for Each Level-Two
WBS Element
Integrated Master Plan (IMP)
 Identifies Program Events (PE), Significant Accomplishments (SA), and
Accomplishment Criteria (AC).
 Establishes the Structure, Parameters & Basis For the Integrated Master
Schedule (IMS) Development
Integrated master schedule (IMS)
Logic network schedule of program planned activities keyed to the
IMP’s accomplishment criteria
Basis of performance measurement system; common element
integrating cost, schedule, & performance
Constructed to provide integrated planning down to the work
package task level, provides horizontal & vertical traceability,
summarization of info and critical path identification and analysis
Supplemental schedules
 Created - as needed - to provide lower levels of detail data
within these schedules are summarized in the IMS and are
part of the program’s Performance Measurement Baseline
(PMB)
 Supports control account schedules and the management of
day-to-day operations
5Steps

5 + 1 Steps to a Credible IMP/IMS
1. Identify Program Events (PE)
2. Identify Significant
Accomplishments (SA)
3. Identify Accomplishment
Criteria (AC)
4. Identify work for each
5. Sequence Work Packages
6. Assemble IMP/IMS
155
5Steps

Identify Program Events
156
Actors Processes Outcomes
Systems Engineer
Define the process flow for
product production from
contract award to end of
contract
Confirmation that the Program
Events represent the logical process
flow for program maturity
Program Manager
Confirm customer is willing
to accept the process flows
developed by the IMP
Engagement with contracts and
customer for PE definition
Project Engineer
Identify interdependencies
between program event
work streams
Value Stream components identified
at the PE level before flowing them
down to the SA level
IMP/IMS Architect
Capture Program Event
contents for each IPT or
work stream
Lay the foundation for a structure to
support the description of the
increasing mature as well as the flow
to needed work.
5Steps

Program Events are the Assessment of the
Evolving Maturity of the Program’s Capabilities
 Program Events are maturity
assessment points in the program
 They define what levels of maturity
for the products and services are
needed before proceeding to the
next maturity assessment point
 The entry criteria for each Event
defines the units of measure for the
successful completion of the Event
 The example below is typical of the
purpose of a Program Event
The Critical Design Review (CDR) is a multi-disciplined product and process assessment
to ensure that the system under review can proceed into system fabrication,
demonstration, and test, and can meet the stated performance requirements within cost
(program budget), schedule (program schedule), risk, and other system constraints.
157/238
5Steps
157

Identify Significant Accomplishments (SA)
for Each Program Event (PE)
System Engineer
Identify Integrated Product
Teams (IPT) responsible for the
SA’s
Define the boundaries of these
programmatic interfaces
Technical Lead
Confirm the sequence of SA’s
has the proper dependency
relationships
Define the product development
flow process improves maturity
Project Engineer
Confirm logic of SA’s for project
sequence integrity
Define the program flows
improves maturity
CAM
Validate SA outcomes in support
of PE entry conditions
Confirm budget and resources
adequate for defined work effort
IMP/IMS Architect
Assure the assessment points
provide a logical flow of maturity
at the proper intervals for the
program
Maintain the integrity of the IMP,
WBS, and IMS
158/238
5Steps
158

The SA’s Define The Entry
Criteria for Each Program Event
159/238
Preliminary Design Review Complete
5Steps
159

Identify Accomplishment Criteria (AC) for
SA
160
CAM
Define and sequence the
contents of each Work Package
and select the EV criteria for
each Task needed to roll up the
BCWP measurement
Establish ownership for the
content of each Work Package
and the Exit Criteria – the
Project Engineer
Identify the logical process flow
of the Work Package to assure
the least effort, maximum value
and lowest risk path to the
Program Event
process flow of the product or
service
Technical Lead
Assure all technical processes
are covered in each Work
Package
technical outcome of each Work
Package
IMP/IMS Architect
Confirm the process flow of the
ACs can follow the DID 81650
structuring and Risk Assessment
processes
Guide the development of
outcomes for each Work
Package to assure increasing
maturity of the program
5Steps

AC’s Are Higher Fidelity Models of the
Program’s Increasing Maturity Flow
161/238
5Steps
161

Work Packages Identify Work for Each
Control Account
Manager
Identify or confirm the work
activities in the Work Package
represent the allocated work
Bounded work effort defined
“inside” each Work Package
Technical Lead
Confirm this work covers the
SOW and CDRLs
All work effort for 100%
completion of deliverable visible
in a single location – the Work
Package
IMP/IMS Architect
Assist in the sequencing the
work efforts in a logical manner
Foundation of the maturity flow
starting to emerge from the
contents of the Work Packages
Earned Value
Analyst
Assign initial BCWS from BOE
to Work Package
Confirmation of work effort
against BOEs
162/238
5Steps
162

Work is Done in “Packages” that Produce
Outcomes Measured with TPM’s
163/238
5Steps
163

Sequence Work Packages (ACs) for each
Significant Accomplishment (SA)
CAM
Define the order of the Work
Packages needed to meet the
Significant Accomplishments for
each Program Event
Define the process flow of work
and the resulting
accomplishments to assure value
is being produced at each SA
and the AC’s that drive them
IMP/IMS Architect
Assure that the sequence of
Work Packages adheres to the
guidance provided by DCMA
and the EVMS System
description
Begin the structuring of the IMS
for compliance and loading into
the cost system
Program Controls
Baseline the sequence of Work
Packages using Earned Value
Techniques (EVT) with
measures of Physical Percent
Complete
Direct insight to progress to plan
in measures of physical progress
164/238
5Steps
164

Sequence Work Packages (AC’s)
into an IMS for each Program Event
165/238
5Steps
165

Assemble Final IMP/IMS
IMP/IMS Architect
Starting with the AC’s under
each SA’s connect Work
Packages in the proper order for
each Program Event to increase
the maturity of each deliverable
Establish the Performance
Measurement Baseline
framework with EAC and their
measure
Program Manager
Confirm the work efforts
represent the committed
activities for the contract
Review and approval of the IMS
– ready for baseline
Project Engineer
Assess the product development
flow for optimizations
Systems Engineer
Confirm the work process flows
result in the proper products
being built in the right order
166/238
5Steps
166

 The IMP is the “Outer
Mold Line”, the
Framework, the “Going
Forward” Strategy for the
Program.
 The IMP describes the
path to increasing maturity
and the Events measuring
that maturity.
 The IMP tells us “How” the
program will flow with the
least risk, the maximum
value, and the clearest
visibility to progress.
 The IMS tells us what
work is needed to produce
the product or service at
The Plan Tells Us “How” We
are Going to Proceed Toward
Done
The Schedule Tells Us “What”
Work is Needed to Make
167/238
The 6 Steps Result In An IMP/IMS Showing
What Done Looks Like
5Steps
167

168/238
Sequencing of Significant Accomplishments
for a Program Event
5Steps
168

Events
Define the maturity
of a Capability at a point in
time.
Accomplishments
Represent requirements
that enable Capabilities.
Criteria
Exit Criteria for the Work
Packages that fulfill Requirements.
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
package
 Deliverables Based Planningsm describes of the increasing maturing of a product or service
through Events or Milestones, Accomplishments, Criteria, and Work Packages.
 Each Event or Milestone represents the availability of one or more capabilities.
 The presence of these capabilities is measured by the Accomplishments and their Criteria.
 Accomplishments are the pre–conditions for the maturity assessment of the product or service at
each Event or Milestone.
 This hierarchy decomposes the System Capabilities into Requirements, Work Packages, and the
activities the produce the deliverables. This hierarchy also describes increasing program maturity
resulting from the activities contained in the Work Packages.
 Performance of the work activities, Work Packages, Criteria, Accomplishments, and Events or
Milestones is measured in units of “physical percent complete” by connecting Earned Value with
Technical Performance Measures.
The structure of a
Deliverables Based
Plan
169/238
5 Steps
PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia169

Now that we’ve visited
Systems Engineering
and Project
Management principles
and practices, let’s
connect all the dots
and start on the hands
on section of the
Workshop.
Connecting
all the Dots
Between SE
and PPM to
form an
IPPMS
Fielded

Technology Implementation
Required
Capabilities
Analysis
Architectural
Analysis
Required
Testing
Analysis
Integration and
Test
Required
Capabilities
Analysis
technology
Configuration
Analysis
Required
Testing
Analysis
Integration and
Test
Required
Capabilities
Analysis
Technology
Design
Required
Testing
Analysis
Integration and
Test
System in Service
Required
Capabilities
Analysis
Architectural
Analysis
Required
Testing
Analysis
Integration and
Test
Specification Issues
Completed System
Tested
Primitive Elements
Tested System
Sub‒System Elements
Tested System
Elements
System Element
Specification
System Sub-System
Element Specification
Primitive Element
Specification
Implementation
Specification
Primitive System Elements
Fielded System
WhatDoes
DoneLook
Like?
PlantoReach
DoneAs
needed?
Resources
neededto
ReachDone
Impedimentsto
ReachingDone
Physical
Percent
Complete
✔️ ✔️
✔️ ✔️ ✔️
✔️ ✔️ ✔️
✔️ ✔️ ✔️ ✔️ ✔️
Fielded

Key Integration Points Between Project
Management and Systems Engineering [205]
172
Clarity of
Responsibility
Common Language
Mutual
Understanding
Fielded

Enough Of Principles Let’s …
… and start INTEGRATING Systems
Engineering and Project Management
on a system, starting a Cow Counting
System
Fielded

Starting with a Work
Breakdown Structure
(WBS), Define the
MOP’s, MOP’s, TPM’s
and KPP’s from our
own program’s WBS
Then let’s assemble
this information into an
Integrated Master
Schedule.
With the IMP and its
MOE’s and MOP’s, the
other activities on our
program are straight
forward
With These
Processes,
Let’s
Develop
SE/PM
Artifacts for
Our Cow
Counting174 PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
Cow

A Hands On Example
175
Count Cows without having to go
count cows
Cow

How Many Cows, of What Breed, Do We
Have in the Pastures?
176
Belted Galloway’s
In the Front Pasture
Herford’s
In the Back Pasture
This is an actual picture from our back yard
Not our cows, but we see them every day
Cow

An Actual Problem in the Field
Cow

Benefits of UAVs in Cattle
Management
 Collect more data with less labor
 Labor shortage addressed with UAV
 Aerial maps improve land management
practices
 Rangeland monitoring
 Monitor livestock, fences, and water resources
 Drought stress monitoring
 Collect data on individual animals
 Smart ear tags
 Biometrics and animal behavior
178
Cow

Our Project Starts with Documents
179
 ConOps
 SOW
 MOE
 MOP
 TPM
 WBS
 Risk / Opportunities
 Deliverable Outcomes
Cow

Business Needs Analysis
 Out of the many applications for UAV
technology, cattle tracking is one where
technology can be applied off the shelf.
 Ranchers in the Western United States have a
difficult time tracking down their cattle and rely
heavily on personal recognizance of their farm
ground to do cattle counts.
 With the availability of low cost unmanned
aerial vehicle systems, the rancher can now
know where their livestock located may be,
using a UAV system to replace the role of the
manual livestock surveyor.
180
Cow

Concept of Operations
 Our UAV is a suite of sensor based aerial and
stationary ground platforms that provide real-
time Reconnaissance, Surveillance, and Asset
information to a managers and staff of a
commercial cattle ranch.
 It supports this staff as they plan, coordinate,
and execute operations through increased
situational awareness (SA) by integrating
intelligence, surveillance, and reconnaissance
(ISR) into a single platform about their assets
– cows on the open range.
181
Cow

Concept of Operations (Continued)
 The user sends a UAV to an
area of interest (within a 5-
mile radius) from a pickup
truck mounted launch
system.
 The user pre-programs the
gas-powered autonomous
UAV for the designated 77
square mile area.
 The UAV must be able to
launch and be recovered on
a rough surface less than
200 meters in length, be
able to reach the
designated area within 10
minutes hour, loiter as high
as 1,000 feet above ground
 The UAV and payload
sensors are to provide 4-
hour persistent coverage of
77 square mile area four
times in a 24 hour loiter time
and provide imagery
between three and ten
seconds of data capture.
 UAV will contain EO/IR to
provide the user with initial
asset situation.
 Based on this preliminary
information, the user may
direct the UAVs to fly at
lower levels to capture Full
Motion Videos (FMVs) of
the herd.
182
Cow

CONOPS (Concluded)
 Because the system must be packaged and
transported with limited cargo capacity of
ranch vehicles, the UAV suite must conform to
limits on power, weight and size.
 We will be using weight as the TPM, as
derived from the Business Case and an
Energy Key Performance Parameters (KPP),
to demonstrate better cost and schedule
performance.
183
Cow

 Range of flight
 Resolution of images
 Ability to return home safely when
commanded to do so, or with loss of
communication or any other disruptive event
 Reliability and maintainability in a ranching
environment
184
Cow

 Speed during transit and during loiter
 Loiter time, once on station
 All the …ilities
 Weight of aircraft and payload
 Fuel consumption, to and from station and
while on station
 Accuracy of location once on station
 Sensor performance across the spectrum ‒
Visual and Infrared
185
Cow

(TPM)
 Weight limits for each major subsystem
 These needed to define and maintain center of
gravity of other flight dynamics parameters
 Full Motion Video resolution and frame rate
 Electroptical / Infrared sensitivity
 Fuel consumption
186
Cow

Connecting MOE’s, MOP’s and TPM’s in the
187
Cow

Approach and Structure for SE Processes
Applied to the UAV
188
Project Management
Processes
Agreement
Processes
Technical
Processes
Support Processes
Technical Management of
Inputs and Outputs
Cow

Reminder of How Systems Engineering
Contributes to These Programmatic
Elements189
Risk
SOW
Cost
WBS
IMP/IMS
TPM
PMB
❺ Deliverables defined
in the SOW, traced to
the WBS, with
narratives and
Measures of
Performance (MoP)
❹ Budget at the Work
Package level, rolled to
the Control Accounts
showing cost spreads for
all work in the IMS
❻ Measures of
Performance (MoP) for
each critical deliverable in
the WBS and identified in
each Work Package in the
IMS, used to assess
maturity in the IMP
❶ The Products and
Processes in a “well
structured” decomposition,
traceable to the deliverables
❷ IMS contains all the
Work Packages,
BCWS, Risk mitigation
plans, with traces to
the IMP measuring
increasing maturity
through Measures of
Effectiveness (MoE)
and KPPs (JROC and
Program)
❸ Technical and Programmatic
Risks Connected through the
WBS, Risk Register, IMP and IMS
Cow

A Systems Engineering Process Model for
Our UAV Project [182]
190
PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia † Roving UAV IED Interdiction System, MSSE Cohort 311-093A, Naval Post Graduate School, Mar
Cow

Applying the “Vee” to our Cow Counting
UAV
With Some Actual Numbers
 Define the Needed Capabilities
 MOE, MOP, KPP, KSA
 Start with an Integrated Master Plan
 KPP for deliverables
 Show the increasing maturity of the products
using the Vee
 Conduct Program Reviews to confirm we have
the right artifacts – from the SE view – for the
program
191
Cow

INCOSE Vee, the IMP/IMS, and
Definition of Done
192
Cow

Event Based Planning Defined in the
 SRR (Systems Requirements Review)
 SFR (System Functional Review)
 PDR (Preliminary Design Review)
 CDR (Critical Design Review)
 ASFUT/GSFUT (Air System/Ground System
Functional Unit Test)
 TRR (Test Readiness Review)
 SVR (System Validation Review)
 PRR (Production Readiness Review)
193
Cow

Cow Counting UAV
Systems Engineering Processes
 User Needs
 Functional Specifications
 Performance Specifications
 Product Specifications
 Build a System
 Verify Individual Components Work
 Verify Performance to Specifications
 Verify System Functionality
 Demonstrate System
194
Cow

Cow Counting UAV
Project Management Processes
195
 Integrated Master
Plan and Integrated
Master Schedule
vertically and
horizontally
traceable.
 The left side of the
Vee and the right
side are connected
by the IMP program
events.
Cow

Cow Counting UAV MOE’s, MOP’s and
TPM’s
196
MOE
1.
KPP
1.
MOP
1.
TPM
1.
Technical and Programmatic Insight ‒ Risk Adjusted Performance to Plan
Technical Insight ‒ Risk Adjusted Performance to Plan
Cow

Topology of a SE Based Organization
Provides basis for cost and schedule performance
management
Defines single schedule to design, build, and verify the
product
Key development, integration, and operational methods to assure
product’s performance measures and parameters are measurable and all
work to produce the deliverables defined in the Integrated Master
Schedule.
For each deliverable, risks are defined and mitigated.
Measures of Effectiveness used to assure mission needs are fulfilled
within the cost parameters to allow tradeoffs to be made.
Defines the single authorities program plan for technical and
programmatic activities needed to provide mission
capabilities
Defines the design that meets specification requirements
Defines product-based organization that parallel the WBS
Defines the work activities needed to meet the project
objectives
Defines the product structure and support processes
Defines product function, performance and verification
requirements, program and product objectives
Cow

Knowledge is of two kinds.
We know a subject
ourselves, or we know
where we can find
information upon it. When
we enquire into any subject,
the first thing we have to do
is to know what books have
treated of it. This leads us
to look at catalogues, and at
the backs of books in
libraries.
‒ Samuel Johnson
18 April 1775

 The documents below are a collection of materials
I’ve applied over my career working System of
Systems
 If you were to read only one text on this topic it
should be:
 Systems Engineering Guide: Collected Works from
MITRE’s Systems Engineering Experts,
https://www.mitre.org/publications/systems-
engineering-guide/about-the-seg
 The second book I depend on is
 Systems Engineering Principles and Practice, Second
Edition, Alexander Kossiakoff, et al, John Wiley &

System Engineering and
Project Performance Management
Resources
1. Integrating Program Management and Systems Engineering: Methods, Tools, and Organizational
Systems for Improving Performance, Eric Rebentisch, Editor‒in‒Chief, John Wiley and Sons,
2017.
2. Systems Thinking: Building Maps for World Systems, John Boardman and Brian Sauser, John
Wiley & Sons, 2013.
3. Systems Thinking: Coping with 21st Century Problems, John Boardman and Brian Sauser, CRC
Press, 2008.
4. The Art of Systems Architecting, Second Edition, Mark W. Maier and Eberhardt Rechtin, CRC
Press, 2000.
5. Engineering Complex Systems with Models and Objects, David W. Oliver, McGraw Hill, 1997.
6. Decision Making in Systems Engineering and Management, 2nd Edition, Gregory Parnell, Patrick
Driscoll, and Dale Henderson, editors, John Wiley & Sons, 2010.
7. Systems Requirements Analysis, Jeffrey O. Grady, Academic Press, 2006.
8. Systems Engineering: Coping with Complexity, Richard Stevens, Peter Brook, Ken Jackson, and
Stuart Arnold, Prentice Hall, 1998.
9. Systematics: How Systems Work and Especially How They Fail, John Gall, Quadrangle, 1975.
10. The Systems Bible: The Beginners Guide to Systems Large and Small, Third Edition, John Gall,
General Systemantics Press, 1975.
11. Thinking in Systems: A Primer, Donella H. Meadows, Chelsea Green Publishing, 2008.
12. The Requirements Engineering Handbook, Ralph R. Young, Artech House, 2004.
13. Requirements Engineering: A Good Practice Guide, Ian Sommerville and Pete Sawyer, John
Wiley & Sons, 1997.PGCS 2019 Master Workshop, 21‒22 August, Canberra Australia
References
Bibliograph
y
201

Resources
14. Enterprise Risk and Opportunity Management: Concepts and Step‒by‒Step Examples for
Pioneering Scientific and Technical Organizations, Allan S. Benjamin, John Wiley & Sons, 2017.
15. Effective Risk Management: Some Keys to Success, Second Edition, Edmund H. Conrow, AIAA
Press, 2003.
16. Design Structure Matrix Methods and Applications, Steven D. Eppinger and Tyson R. Browning,
The MIT Press, 2012.
17. “Engineering Complex Systems: Implications for Research in Systems Engineering,” IEEE
Transactions on Systems, Man, and Cybernetics ‒ Part C: Applications and Reviews, Vol. 33,
No. 2m May 2003.
18. ”A Framework for Understanding Uncertainty and its Mitigation and Exploitation in Complex
Systems,” Daniel Hastings and Hugh McManus, 2004 Engineering Systems Symposium.
19. “System Engineering Plan and Systems Engineering Management Plan Alignment,” Chet
Bracuto and Bob Scheurer, NDIA 11th Annual Systems Engineering Conference, October 21,
2008.
20. “The Extension of Systems Architecting to the Architecting of Organizations,” Eberhardt
Rechtin, ICSE ’99
21. “A Project‒Product Lifecycle Management Approach for Improved Systems Engineering
Practices,” Amira Sharon and Dov Dori, INCOSE International Symposium, Session 8, Track 2,
June 2008.
22. “Toward a New Mindset: Bridging the Gap Between Program Management and Systems
Engineering,” Mark Langley, Samantha Robitaille, and John Thomas, PMI Global Congress, 22
202
References
Bibliograph
y

Resources
24. Systems Engineering Handbook: A Guide for System Lifecycle Processes and Activities, Fourth
Edition, 2015.
25. NASA Systems Engineering Handbook, NASA‒SP‒2016‒6105 Rev 2, December 2016
26. “Integrating Project Management and Systems Engineering,” Ann Hodges, Sandia National
Laboratories, National Nuclear Security Agency
27. “Integrating Systems Engineering with Project Management: a Current Challenge!,” Rui Xue,
Claude Baron, and Daniel Prun, 24th Annual INCOSE International Symposium, June 2014.
28. “Survey Report: Improving Integration of Program Management and System Engineering:
Results of a Joint Survey of PMI and INCOSE,” Edivandro Conforto, Monica Rossi, Eric
Rebentisch, Josef Oehmen and Maria Pacenza, 23rd INCOSE Annual International Symposium,
June 2013.
29. “Systems Engineering the Project,” Van Gemert, PMI Global Congress, 2013.
30. Engineering Complex Systems with Models and Objects, David W. Oliver, Timothy P. Kelliher,
and James G. Keegan, McGaw Hill, 1997.
31. Modelling Complex Projects, Terry Williams, John Wiley & Sons, 2002.
32. “Investigating relationships and semantic sets amongst system lifecycle properties (ilities).” de
Weck, O. L., Rhodes, D. H., & Ross, A. M. (2012, June). 3rd International Engineering Systems
Symposium, TU Delft, the Netherlands.
33. “The Relationship of System Engineering to the Project Cycle,” Kevin Forsberg and Harold
Mooz, National Council On Systems Engineering (INCOSE) and American Society for
203
References
Bibliograph
y

Resources
34. “The Case for Systems Management,” Hal Mooz and Kevin Forsberg, Center for Systems
Management, 2002.
35. The Essentials of Project and Systems Management 2nd Edition, Howard Eisner, John Wiley &
Sons, 2001.
36. Systems Engineering Body of Knowledge 1.9.1, “Systems Engineering and Project
Management,” pp. 823‒846.
37. Seven Key Principles of Program and Project Success: A Best Practices Survey.
NASA/TM‒2008‒214692.
38. NASA Space Flight Program and Project Management Handbook, NASA/SP‒2014‒3705
39. The Handbook of Program Management, Second Edition, James T. Brown, McGraw Hill, 2014.
40. Performance‒Based Project Management, Glen B. Alleman, American Management
Association, 2014.
41. “Chapter 3: Systems Engineering,” of the Defense Acquisition Guidebook, Defense Acquisition
University, 9820 Belvoir Road, Fort Belvoir, VA 22060.
42. A Project Manager Lessons Learned,
https://www.nasa.gov/pdf/2383253main_62682main_jerry_madden_forum7.pdf
43. Integrating Program Management and Systems Engineering: Methods, Tools, and
Organizational Systems for Improving Performance, Eric Rebentisch, Editor‒in‒Chief, John
Wiley, 2017.
44. The Handbook of Program Management: How to Facilitate Project Success with Optimal
Program Management, Second Edition, James T. Brown, McGraw‒Hill, 2014.
45. Integrated Project Performance Management and Control: First Comes the Theory then the
204
References
Bibliograph
y

Resources
47. Essential of Project and Systems Engineering Management, Second Edition, Howard Eisner,
John Wiley & Sons, 2002.
48. Defence Systems Engineering Handbook, Defence Engineering Group, University College,
London, 2002.
49. Systems Engineering ‒ Application and Management of the Systems Engineering Process, IEEE
Std 1220‒2005.
50. “Systems Engineering ‒ Systems Life Cycle Processes,” ISO/IEC 15288.
51. “Evaluation and Synthesis of Methods for Measuring Systems Engineering Efficacy within a
Project and Organization,” Timothy Daniel Flynn, Massachusetts Institute of Technology,
February 2007.
52. “SMC Systems Engineering Primer & Handbook: Concepts, Processes, and Techniques,” Space
& Missile Systems Center, U. S. Air Force, 29 April 2005.
53. “SMC Systems Engineering Specialty Engineering Disciplines: Frameworks and Descriptions,”
Space & Missile Systems Center, U. S. Air Force, 3 October 2011.
54. “Investigating Relationships and Semantic Sets Amongst System Lifecycle Properties (ilities),” 3rd
International Engineering Systems Symposium, TU Delft, the Netherlands.
55. “Framework for Managing Systems‒of‒Systems Ilities,” Kang Shian Chin and Sim Kok Wah,
DSTA Horizons, 2013/14.
56. “Engineering Enterprise Using Complex‒System Engineering (CSE),” Brian E. White, 1st Annual
Systems of Systems (SoS) Engineering Conference, 13‒14 June 2005.
57. Design Software‒Intensive Systems: Methods and Principles, Pierre F. Tiako, Information
Science Reference, 2009.
58. “Systems Engineering and Program Management,” David E. Stem, Michael Boito, and Obaid
Younossi, RAND MG413.
205
References
Bibliograph
y

Resources
59. Performance‒Based Project Management, Glen B. Alleman, American Management
Association, 2014.
60. Integrated Project Performance Management sand Control: First Comes the Theory, then the
Practice, Mario Vanhoucke, Springer, 2014.
61. Integrating Program Management and Systems Engineering: Methods, Tools, and
Organizational Systems for Improving Performance, Eric Rebentisch, Editor‒in‒Chief, John
Wiley, 2017.
62. “Clarifying the Relationships between Systems Engineering, Project Management, Engineering
and Problem Solving,” Joseph Kasser and Deek Hitchins, Asia‒Pacific Council on Systems
Engineering Conference, Yokohama, 2013.
63. “Systems Engineering for Project Managers,” Mike Wilkinson, INCOSE/APM Joint Workshop,
15 January 2013.
64. “The Art and Science of Systems Engineering,” Michael Ryschkewitsch, Dawn Schaible, and
Wiley Larson, Systems Research Forum, Vol. 3, No. 2 pp. 81‒1000, 2009.
65. “Twelve Roles and Three Types of Systems Engineering.” Sara A. Sheard, Software
Productively Consortium, February 11, 2003.
66. “Systems Engineering, Program Management conjoined Disciplines over the Project Life
Cycle,” William Lyders, NDIA 8th Annual Systems Engineering Conference, October 2005.
67. Handbook of Systems Engineering and Management, Andrew P. Sage and William B. Rouse,
John Wiley and Sons, 1999.
68. Systems Engineering Management, Fifth Edition, Benjamin S. Blanchard and John E. Blyler,
John Wiley & Sons, 2016.
206
References
Bibliograph
y

Resources
70. “A Project-Product Lifecycle Management Approach for Improved Systems Engineering
Practices,” Amira Sharon, Valeria Perelman, and Dov Dori, June 2011.
71. “Systems Engineering Measurement Primer A Basic Introduction to Measurement Concepts
and Use for Systems Engineering SEBOK Measurement,”
https://www.sebokwiki.org/wiki/Measurement
72. “Visualizing Project Management and System Engineering as an Integrated Process,” Harold
Mooz and Kevin Forsberg, Center for Systems Management, 1997.
73. Space and Missile Systems Center Risk Management Process Guide, Version 2.0 ‒ 5
September 2014.
74. “Guide to the Systems Engineering Body of Knowledge (SEBoK), IEEE Computer Society,
https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_Body_of_Knowledge_(SE
BoK)
75. ”Relationships between Systems Engineering and Project Management,” Guide to the Systems
Engineering Body of Knowledge,
https://www.sebokwiki.org/wiki/Relationships_between_Systems_Engineering_and_Project_Ma
nagement
76. “Systems Engineering Guide for System of Systems,” Systems and Software Engineering
Deputy Under Secretary of Defense (Acquisition and Technology), Version 1.0, August 2008.
77. Primer for Model‒Based Systems Engineering, Vitech, 2011.
78. “The Art and Science of Systems Architecting” Mark Maier, Aerospace Corporation.
79. “Best Practices for Using Systems Engineering Standards (ISO/IEC/IEEE 15288, IEEE
15288.1, and IEEE 15288.2),” April 2017.
207
References
Bibliograph
y

Resources
80. System of Systems Engineering: Innovations for the 21st Century, Mo Jamshidi, Editor, John
Wiley & Sons, 2009.
81. Systems Engineering: A 21st Century Systems Methodology, Derek Hitchins, John Wiley &
Sons, 2007.
82. Systems of Systems Engineering Principles and Applications, Mo Jamshidi, CRC Press, 2009.
83. “Reducing Unwelcome Surprises in Project Management,” Tyson Browning and Ranga
Ramasesh, MIT Sloan Management Review, Spring 2015.
84. “Understanding and Managing Project Complexity,” Syed Waqar Azim, The University of
Manchester, 2010.
85. “Complexity Management for Project, Programmes, and Portfolios: An Engineering Systems
Perspective,” Josef Oehmen, Christian Thuesen, Pedro Parraguez Ruiz, and Joana Geraldi,
PMI White Paper, 2015.
86. System of Systems Engineering: Innovations for the 21st Century, Edited by Mo Jamshidi, John
Wiley & Sons, 2009.
87. System Engineering Analysis, Design, and Development: Concepts, Principles, and Practices
(Wiley Series in Systems Engineering and Management) 2nd Edition, Charles S. Wasson, John
Wiley & Sons, 2015.
88. Process for Systems Architecture and Requirements Engineering, 1st Edition, Derek Hatley,
Peter Hruschka, and Imtiaz A. Pirbhai, Dorset House, 2000.
89. “Toward Project Management 2.0, Raymond E. Levitt, Engineering Project Organization
Journal, 1(3), pp. 197‒210, September 2011.
90. “Systems Engineering and Project Management: Complementary Disciples, or Competing
Paradigms,” T. Cooke-Davies, INCOSE Symposium, July 2012.
91. An Empirical Analysis of the Relationship Between Project Planning and Project Success,” Dov
208
References
Bibliograph
y

Resources
86. “Systems Engineering and Project Management for Product Development: Optimizing Their
Working Interfaces,” Mariela I. Santiago, Naval Postgraduate School, September 2013.
87. “How Do Committees Invent?” Melvin E. Conway, Datamation, 14(5), pp. 28‒31, April 1968.
88. “The Value of Systems Engineering in Product Management,” Oliver Hoehne and Steven
Turner, INCOSE International Symposium, July 2016, Edinburgh, Scotland, UK
89. IEEE Guide for Information Technology ‒ System Definition ‒ Concept of Operation (ConOps)
Document, (IEEE STD 1362-1998), in “Systems Engineering Guide, MITRE Corporation,
https://www.mitre.org/publications/systems-engineering-guide/se-lifecycle-building-
blocks/concept-development/concept-of-operations
90. Chapter 28, “Capabilities-Based Planning,” TOGAF® Standard, Version 9.2
91. “The Twelve Principles of Efficiency,” Harrington Emerson, The Engineering Magazine, New
Work, https://babel.hathitrust.org/cgi/pt?id=bc.ark:/13960/t2g746111;view=1up;seq=13
92. Processes for System Architecture and Requirements Engineering, Derek Hatley, Peter
Hruschka, and Imtiaz Pirbhai, Dorset House, 2000.
93. Architecting Systems. Concepts, Principles and Practice Concepts, Principles and Practice,
Hillary Sillitto, College Publications, 2014.
94. “Managing development projects: The partnership between project managers and systems
engineers,” Sigal Kordova, Eyal Katz, and Moti Frank, Systems Engineering, Volume 22, Issue
3.
209
References
Bibliograph
y

Resources
95. “Identifying Acquisition Framing Assumptions Through Structured Deliberations,” Mark V. Arena
and Lauren A. Mayer, Rand Corporation, RAND TL 135
96. “INCOSE Systems Engineering Measurement Primer v2.0,” INCOSE-TP-2010-005-02, 5
November 2010.
97. Megaprojects and Risk: An Anatomy of Ambition, Bent Flyvbjerg, N. Bruzelius, and W.
Rothengatter, Cambridge, Cambridge University Press, 2003.
98. “Integrating Systems Engineering and Project Management: Cutting the Cost and Boosting the
Success of Complex Projects,” Andrew Wright and Arty Tailor, APM/INCOSE UK, Joint Working
Group of SE/PM Integration, APM Systems Thinking SIG.
99. “System Testing in the Avionics Domain,” von Aliki Ott, zur Erlangung des Grades einer
Doktorin der Ingenieurwissenschaften, Vorgelegt im Fachbereich 3 (Mathematik & Informatik)
der Universit¨at Bremen im Oktober 2007.
100. ”A Systems Engineering Approach to Unmanned Aerial Vehicle Design,“ M. Sadracy, 10th AIAA
Aviation Technology, Integration, and Operations (ATIO) Conference, 13—15 September 2010.
101. “Autonomous Vehicles in Support of Naval Operations,” Committee on Autonomous Vehicle in
Support of Naval Operations, Naval Studies Board, National Research Council, 2005.
102. “Unmanned Aerial Vehicles Landscape Analysis: Application in the Development Context,”
USAID Global Health Supply Chain Program: Procurement and Supply Management, 2017.
103. “Unmanned Aerial Vehicles: A Survey on Civil Applications and Key Research Challenges,”
Hazim Shakhatreh, Ahmad Sawalmeh, Ala Al-Fuqaha, Zuochao Dou, Eyad Almaita, Issa Khalil,
Noor Shamsiah Othman, Abdallah Khreishah, Mohsen Guizani, IEEE Access, Volume 7, 2019.
210
References
Bibliograph
y

Resources
104. “Challenges of Developing UAV Applications: A Project Management View,” Ahmed Idries,
Nader Mohamed, Imad Jawhar, Farhan Mohamed, Jameela Al-Jaroodi, Proceedings of the
2015 International Conference on Industrial Engineering and Operations Management Dubai,
United Arab Emirates (UAE), March 3 – 5, 2015.
105. “Theory of Effectiveness Measurement,” Richard K. Bullock, Major, USAF, Air Force Institute of
Technology, September 2006.
106. “Systems Engineering Guide for System of Systems,” Version 1.0, August 2008.
107. “Recommended Practices: System of Systems Considerations in the Engineering of Systems,”
TTCP technical Report, TR-JSA/TP4-1-2104, August 2014.
108. “System Engineering Principles, Reference Section 3.2,” in Engineering Elegant Systems:
Theory of Systems Engineering, 4th Draft, Michael D. Watson, NASA Marshall Space Flight
Center, 9 February 2018.
109. “Uses of Drone Applications to Monitor Productivity,” Piero Anticona Tello, PPM World Journal,
Vol. V111, Issue IV – May 2019.
110. “NASA’s Systems Engineering Competencies,” Academy of Program/Project & Engineering
Leadership
111. “Better Schedule Performance Assessments Derived from Integrated Master Plan-References
Schedule Metrics,” David C. Bachman.
112. “Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide,” Version
0.9, October 21, 2005
113. “Program Management Framework,” from Booz Allen training for national Nuclear Security
Agency (NNSA), US Department Energy.
211
References
Bibliograph
y

Resources
114. “Leading Complex Projects,” Steven R. Meier, PMI Loudon County Chapter, 13 May 2013.
115. A Handbook of Software and Systems Engineering: Empirical Observations, Laws and
Theories, Albert Endres and Dieter Rombach, Pearson ‒ Addison Wesley, 2003.
116. “Aligning Systems Engineering and Project Management Standards to Improve the
Management of Processes,” Rui Xue, Claude Baron, Phillippe Esteban, and Abd-El-Kader
Sahraoui, Springer-Verlag, Berlin 2011.
117. “Systems Engineering Management and the Relationship of Systems Engineering to Project
Management and Software Engineering,” Ray Madachy, Naval Postgraduate School, July 19,
2011.
118. “Exploring the Relationship between Systems Engineering and Software Engineering,” Art
Pyster, Rick Adcock, Mark Ardis, Rob Cloutier, Devanandham Henry, Linda Laird, Harold ‘Bud’
Lawson, Michael Pennotti, Kevin Sullivan and Jon Wade, 2015 Conference on Systems
Engineering Research, Procedia Computer Science 44, pp. 7-8-717, 2015.”
119. “A Complexity Primer for Systems Engineers,” INCOSE, July 2015,
https://www.incose.org/docs/default-source/ProductsPublications/a-complexity-primer-for-
systems-engineers.pdf?sfvrsn=0&sfvrsn=0
120. “Systems Engineering to Improve the Governance in Complex Project Environments,” Giorgio
Locatelli, Mauro Mancini, and Erika Romano, International Journal of Project Management, pp.
1395-1410, November 2014.
121. “Toward a Typological Theory of Project Management," Aaron Shenhar and Dov
Dvir, Research Policy, 25(4), pp. 607-632, 1996.
122. “Clarifying the Relationships between Systems Engineering, Project Management, Engineering
and Problem Solving,” Joseph Kasser and Derek Hitchins, Asia-Pacific Council on Systems
Engineering Conference, 2013.
212
References
Bibliograph
y

Resources
123. “Identifying Relationships between Systems Engineering Processes and Project Success in
NASA Complex Projects and Other Organizations,” Kathryne Angela Schomburg, Iowa State
University, 2015.
124. ”Project Management: Systems Engineering & Project Control Processes and Requirements,”
Johnson Space Center, JPR 7120.3, March 2004.
125. “Seven Key Principles of Program and Project Success ‒ A Best Practices Survey,” Vincent J.
Bilardo, et. al, NASA/TM-2008-214692, April 2008.
126. “A Conceptual Framework for Defense Acquisition Decision Makers: Giving the Schedule Its
Due,” Chad Dacus and Col Stephen Hagel USAF (Ret.), Defense ARJ, Vol. 21, No. 1, pp. 486-
504, 2014.
127. “Engineering Elegant Systems: Theory of System Engineering,”
128. “A Model-Based Approach To System-Of-Systems Engineering Via The Systems Modeling
Language,” Kevin Hughes Bonanne, Purdue University August 2014.
129. A Primer for Model-Based Systems Engineering, 2nd Edition, David Long and Zane Scott,
Vitech, 2011.
130. Modeling and Analysis of Dynamic Systems, Second Edition, Ramin S. Esfandiari and Bie Lu,
CRC Press, 2014.
131. Systems Analysis and Design, 5th Edition, Alan Dennis, Barbara Haley Wixom, and Roberta M.
Roth, John Wiley & Sons, 2012.
132. “Agile Systems, Agile Systems Engineering, and Self-Organizing Agile Systems Security,” Rick
Dove, ILTAM/INCOSE IL, Tel Aviv, 8 Jan 2014.
133. “Project Management vs. Systems Engineering Management: A Practitioners’ View on
213
References
Bibliograph
y

Resources
134. “Improving Project-Product Lifecycle Management with Model-Based Design Structure Matrix: A
Joint Project Management and Systems Engineering Approach,” Amira Sharon, Olivier L. de
Weck, and Dov Dori, Systems Engineer, Vol. 16, No. 4, 2013.
135. “Integrating Systems Engineering with Program and Project Management,” Randy Ade,
INCOSE INSIGHT, 23 June 2015.
136. “Managing development projects—The partnership between project managers and systems
engineers,” Sigal Kordova, Eyal Katz, and Moto Frank, Systems Engineering, Volume 22, Issue
3, May 2019.
137. “Best project management and systems engineering practices in the preacquisition phase for
federal intelligence and defense agencies,” Steven R. Meier, Systems Engineering, Volume 39,
Issue 1, March 2008.
138. “Key Concepts in Integration,” Eric S. Rebentisch, in Integrating Program Management and
Systems Engineering, John Wiley & Sons, 2017.
139. “Behavioral Competencies of Highly Regarded Systems Engineers at NASA,” Mary Ellen Derro
and Christine R. Williams, IEEEAC Paper #1198, Version 1, December 18, 2008.
140. Systems Engineering: Roles and Responsibilities,” Dr. Steve Jolly, NASA PI-Forum, July 27,
2011
141. Guide to the Project Management Body of Knowledge, Fifth Edition, Project Management
Institute.
142. “A Framework for Systems Engineering Development of Complex Systems,” Dr. Karl L.
Brunson, Dr. Jeffrey Beach, Dr. Thomas A. Mazzuchi, and Dr. Shahram Sarkani, CrossTalk,
July/August 2009.
214
References
Bibliograph
y

Resources
144. Systems Engineering Guide: Collected Wisdom from MITRE’s Systems Engineering Experts,
https://www.mitre.org/publications/systems-engineering-guide/about-the-seg
145. Project Management: A Systems Approach to Planning, Scheduling, and Controlling, 8th
Edition, Harold Kerzner, John Wiley & Sons, 2003.
146. Systems Thinking, Third Edition: Managing Chaos and Complexity: A Platform for Designing
Business Architecture, Jamshid Gharajedaghi, Morgan Kaufmann, 2011.
147. Enterprise Systems Engineering: Advances in the Theory and Practice (Complex and
Enterprise Systems Engineering), edited by George Rebovich and Brian White, CRC Press
Taylor & Francis Group, 2011.
148. “Project Management and Systems Engineering ‒ Z11,” Issue 1.1, Jan 2018, INCOSEUK and
Association for Project Management.
149. “Perspectives on the Use of Unmanned Aerial Systems (UAS) to Monitor Cattle,” Jayme Garcia
Arnal Barbedo and Luciano Vieira Koenigkan, Outlook in Agriculture, 47(3), pp. 214-222, 2018.
150. “Comparison of Three Techniques to Identify and Count Individual Animals in Aerial Imagery,”
Pat A. Terletzky and Robert Douglas Ramsey, Journal of Signal and Information Processing, 7,
pp. 213-135, 2016.
151. “How do you find the green sheep? A critical review of the use of remotely sensed imagery to
detect and count animals,” Tracey Hollings, Mark Burgman, Mary van Andel, Marius Gilbert,
Timothy Robinson, and Andrew Robinson, Methods in Ecology and Evolution. 19 January 2008.
152. “Best Practices for Using Systems Engineering Standards, (ISO/IEC/IEEE 15288, IEEE
15288.1, and IEEE 15288.2) on Contracts for Department of Defense Acquisition Programs,”
April 2017.
215
References
Bibliograph
y

Resources
153. “The Measurement Handbook: A Practical Guide for Developing and Analyzing the Capabilities-
based Assessment, pre-Materiel Development Decision Analysis, and Analysis of Alternatives,”
Office of Aerospace Studies, Air Force Materiel Command, Kirkland AFB, 6 August 2014.
154. “Technical Measurement Guide,” Garry Roedler and Cheryl Jones, INCOSE-TP-2003-020-01,
December 2005.
155. “EIA-632, Processes for Engineering a System”
156. “Systems Engineering Revitalization: An Industry Perspective,” DSP Conference, Dr. John B.
Noblin, LM IS&S.
157. “If it can’t be expressed in figures ‒ project performance management,” Simona Bonghez and
Adina Grigoroiu, PMI Global Congress 2013‒EMEA,
https://www.pmi.org/learning/library/project-performance-management-project-life-cycle-5818
158. ”Project Management as Perceived from Ancient Egyptian Projects, Abdel-Fattah El-Marshly,
pp. 275-290, in Dimensions of Project Management: Fundamental, Techniques, Organizations,
Applications, Springer-Verlag, 1990.
159. “Will Cattle Producers Be Willing To Adopt Electronic Cattle Monitoring Systems?,” Lori D.
Allmon, Masters Thesis, Oklahoma State University, Stillwater Oklahoma, 2013.
160. “Drones Count Wildlife More Accurately and Precisely Than Humans,” Jarrod C. Hodgson, et.
al. Methods in Ecology and Evolution, 4 January2018.
161. “Strategic Cattle Roundup using Multiple Quadrotor UAVs,” Sunghun Jung and Kartik B. Ariyur,
International Journal of Aeronautical & Space Science. 18(2), pp. 315-326, 2017
216
References
Bibliograph
y

Resources
162. “Systems Engineering: The Art of Gluing Pieces,” Bernardo A. Delicado, INCOSE Institute for
Technical Leadership, Madrid, Spain, 22 May 2017.
163. Organize for Complexity: How to Get Life Back Into Work to Build the High Performance
Organization”, Niels Pflaeging, BetaCodex Publishing, 2014.
164. The Engineering Design of Systems: Models and Methods, 2nd Edition, Dennis M. Buede, John
Wiley & Sons, 2009.
165. “On Evidence-Based Risk Management in Requirements Engineering,” Daniel Mendez
Fernandez, Michaela TieBler, Marcos Kalinowski, Michael, Felderer, and Marco Kuhrmann,
arXiv:1707.00144v2, 31 July 2017.
166. Unmanned Aerial Vehicle, Wikipedia page,
https://en.wikipedia.org/wiki/Unmanned_aerial_vehicle
167. “Design Methods Fact Sheet: Measures of Effectiveness (MOE) and Measures of Performance
(MOP),” The University of Queensland, MECH4551 ‒ System Design Projects, Semester 2
2001.
168. “Distinguishing Two Dimensions of Uncertainty,” Craig R. Fox and Gulden Ülkümen, Chapter
1m Perspectives on Thinking, Judging, and Decision Making, Oslo Norway.
169. “Planning, Tracking, and Reducing a Complex Project’s Value at Risk,” Tyson R. Browning,
Project Management Journal, Vol. 50(1), pp. 71-85, 2019.
170. “Can Project Management Learn Anything From Studies of Failure in Complex Systems?,”
Chris Ivory and Neil Alderman, Project Management Journal, Vol 36, Issues 3, 2005.
171. Normal Accidents: Living with High Risk Technologies, Basic Books, 1984.
172. “Review of Research into the Nature of Engineering and Development Rework: Need for a
Systems Engineering Framework for Enabling Rapid Prototyping and Rapid Fielding,” Shawn
Dullena, Dr. Dinesh Verma, Dr. Mark Blackburn, 17th Annual Conference on Systems
217
References
Bibliograph
y

Resources
173. ISO/PAS 19450:2015 ‒ Automated Systems and Integration ‒ Object‒Process Methodology.
174. Model-Based Systems Engineering with OPM and SysML, Dov Dori, Springer, 2016.
175. Systems Thinking: Coping with 21st Century Problems, John Boardman and Brian Sauser, CRC
Press, 2008.
176. Systems Engineering: Coping with Complexity, Richard Stevens, Peter Brook, Ken Jackson,
and Stuart Arnold, Prentice Hall, 1998.
177. “Summary Research Report on Critical Success Factors in Government Program
Management,” James H. Dobbins and Richard G. Donnelly, Acquisition Quarterly – Winter
1998.
178. “Model-Based Systems Engineering (MBSE), Connecting the Dots Process.” Loyd Baker,
EnergyTech 2015.
179. “A framework to improve performance measurement in engineering projects,” Li Zheng, Claude
Baron, Philippe Esteban, Rui Xue, Qiang Zhang, INSIGHT - International Council on Systems
Engineering(INCOSE), Wiley, 2017, 20 (4), pp.40-43.
180. “Situation-Oriented Requirements Engineering,” Nimanthi Atukorala, Iowa State University,
2019.
181. Normal Accidents: Living with High-Risk Technologies, Charles Perrow, Princeton University
Press, 1999.
182. The Fifth Discipline: The Art and Practice of the Learning Organization, Peter Senge, March
2010.
183. “Model-Based Design of Project Systems, Modes, and States,” Ali Shafaat and C. Robert
Kenley, Systems Engineering, Vol 22, Issue 4, July 2019.
218
References
Bibliograph
y

Resources
186. “Systems Engineering Leading Indicators for Assessing Program and Technical Effectiveness,”
Donna H. Rhodes, , Ricardo Valerdi, , Garry J. Roedler, Systems Engineering, Volume 12,
Issue 1, Spring 2009.
187. System-of-Systems Symposium: Report on a Summer Conversation, Popper, S., Bankes, S.,
Callaway, R., and DeLaurentis, D., July 21–22, 2004, Potomac Institute for Policy Studies,
Arlington, VA.
188. “Systems Engineering Guide for Systems of Systems,” Version 1.0, August 2008, Director,
Systems and Software Engineering Deputy Under Secretary of Defense (Acquisition and
Technology), Office of the Under Secretary of Defense, (Acquisition, Technology and Logistics).
189. “Systems Engineering Guide for Systems of Systems Essentials,” Director of Systems
Engineering, Office of the Director, Defense Research and Engineering, Washington, D.C.,
December 2010.
190. “System of Systems Engineering and Family of Systems Engineering from a Standards, V-
Model, Dual V-Model, and DoD Perspective,” John O. Clark, Systems and Software Technology
Conference, April 2010.
191. “Business Requirements Analysis, Dennis Van Gemert, in “Systems Engineering the Project,”,
in 2013.
192. “Framework For Managing System-of-systems Ilities,” KANG Shian Chin, PEE Eng Yau, SIM
Kok Wah, PANG Chung Khiang, DSTA HORIZONS | 2013/14.
193. “Complexity and Project Management: A General Overview,” José R. San Cristóbal, Luis Carral,
Emma Diaz,3 José A. Fraguela, and Gregorio Iglesias, Complexity Volume 2018, Article ID
4891286, 2018.
219
References
Bibliograph
y

Resources
195. “Design Consideration: UAV System Architecture for Smart Applications,” Ender Ayhan
Rencüzoǧullari, Ali Kiliç, and Sadettin Kapücu, Conference: International Eurasian Conference
on Science, Engineering and Technology (EurasianSciEnTech 2018),Ankara, Turkey.
196. “A Model for Assessing UAV System Architectures,” Andrew Renault, Procedia Computer
Science 61:160-167, December 2015.
197. “Designing Unmanned Systems with Greater Autonomy: Using Federated, Partially Open
Systems Architecture Approach,” Daniel Gonzales and Sarah Harting, RAND National Defense
Institute, 2014.
198. “A Unified Product and Project Lifecycle Model for Systems Engineering,” Amira Sharon,
Technion - Israel Institute of Technology, March 2010.
199. “Enhancing knowledge management into systems engineering through new models in SysML,”
Julien Gardan and Nada Matta, 27th CIRP Design 2017.
200. “Global Hawk Systems Engineering Case Study,” Bill Kinzig, MacAulay- Brown, Inc. Air Force
Center for Systems Engineering, Wright-Patterson Air Force Base, Dayton, Ohio.
201. “The Measures Handbook: A Practical Guide for Developing and Analyzing Measures in the
Capabilities-Based Assessment, pre-Materiel Development Decision Analysis, and Analysis of
Alternatives,” 6 August 2014, Office of Aerospace Studies, Air Force Materiel Command,
Kirkland Air Force Base.
202. “Model-Based Project-Product Lifecycle Management and Gantt Chart Models: A Comparative
Study,” Amira Sharon and Dov Dori, Systems Engineering, Vol. 20, No. 5, 2017.
203. “Program Success ‒ A New Way to Predict,” John Higbee, Defense Acquisition University, 25
August 2003.
220
References
Bibliograph
y

Resources
204. “Suitable Metrics for Measuring the Effectiveness of the Systems Engineering Process,”
Benjamin Q. Luong and Lauren L. Nguyen, Boeing, 2008.
205. “Z-11 Project Management and Systems Engineering,” INCOSEUK, Issue 1.1 Jan 2018.
206. “Unifying Systems Engineering: Seven Principles for systems engineered solutions,” Joseph
Kasser and Derek K. Hitchins, The 20th International Symposium of the INCOSE, Denver 2011.
221
References
Bibliograph
y

Systems Engineering and Project
Performance Management Websites
1. https://segoldmine.ppi-int.com/
2. https://www.incose.org
3. https://www.nasa.gov/offices/oce/functions/systems/index.html
222

PGCS 2019 Master Class Integrating SE with PPM

More Related Content

What's hot (20)

Similar to PGCS 2019 Master Class Integrating SE with PPM (20)

More from Glen Alleman (20)

Recently uploaded (20)

PGCS 2019 Master Class Integrating SE with PPM