Integration of technical development within complex project environment
1 like293 views
This document discusses the integration of design and safety functions within complex technical system development, highlighting existing deficiencies in project management that lead to unanticipated changes resulting in cost overruns and unsatisfactory outcomes, particularly in safety-critical engineering projects. The author aims to develop a framework for better planning, monitoring, and control of technical activities in such environments by integrating existing concepts into a complexity management framework. The study emphasizes the importance of identifying critical success factors and managing complexity to improve project outcomes and reduce risks.
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 1 of 70
1. Introduction
1.1 Overview
The characteristics of complex systems include a difficulty in determining outcomes from the inputs [1] and
there being present a ‘degree of disorder’ [2]. The relationships and dependencies between processes,
organisations and other external systems describe perfectly that of any complex system. What is more, the
general trend is for development complexity to increase over time. This is while simultaneously catering for
advances in technology and the downward pressures on development timescales and cost. The result is
greater prevalence of the wicked problem, that is a technical system development that is ‘highly resistant to
resolution’ [3]. This can be thought of as Technical Development Complexity.
This dissertation builds on the research undertaken within the previous literature survey [4]. This work
described the high failure rate amongst projects, a summary of the main reasons for this and areas of further
investigation. Also a variety of definitions were given for complexity and compelling reasons that this should
be the focus of attention when planning for the development of a system. The dissertation develops the
literature survey’s conclusions, describing a framework for the treatment of complex system development
within a wider project environment.
As the network of individual activities and their interactions increase it is reasonable to expect that the
mechanisms that are used to achieve a successful outcome will be greater in terms of their magnitude and
resources to undertake them. Therefore, complexity inevitably influences the effort required to plan, monitor
and control development activities. These mechanisms and conditions can be specified within development
processes but it would appear reasonable that these also need to be tailored to suit the particular
development characteristics and especially those relating to its complexity. These are known as Critical
Success Factors (CSF) and describe ‘essential areas of activity that must be performed well if you are to
achieve the mission, objectives or project’ [5]. However, it should be noted that outside describing their form
and importance, literature provide no satisfactory methods or process for determining suitable CSFs.
1.2 Objectives
It will be proposed that complexity comprises a number of aspects and that, for the purpose of this
dissertation, these can be comprised of themes and criteria. These aspects will not be homogeneous across
all the development activities nor the development lifecycle. This variation across a project can be seen as a
complexity profile that will evolve throughout the development lifecycle. Outwardly similar developments,
even within the same organisation, may not exhibit the same complexity characteristics. This can often be
attributed to the presence (or lack) of constraints outside the development team’s control and the result of
influential management decisions. Consideration of complexity will have a number of goals:
Identify the most influential Critical Success Factors (CSF) to put in place environmental conditions
required for a successful outcome;
Identify methods to put the responsibility for interventions where it is best placed through early
identification the establishment of environmental factors is often beyond the direct influence of the
development team or its managers [6][7];
Recognise how complexity evolves throughout the development and how some aspects increase as
others diminish. As such the methods to plan, monitor and control development activities will need
to evolve accordingly throughout the development lifecycle;
Describe how interventions can influence complexity and displace its effects elsewhere;
Recognise and anticipate development risks identified through the consideration of complexity early
in the development lifecycle and during detailed planning activities;
Develop a framework for assigning interventions to defined risks more closely, at the appropriate
level within the Work Breakdown Structure (WBS). The identification of interventions will be initiated
by the recognition of CSFs and risk planning;](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-5-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 2 of 70
Propose methods to guide where the most effort should be applied in planning, monitoring and
controlling activities through the identification of areas of maximum complexity. This is where the
greatest number of residual risks is likely to be. This is most important where there exists a constraint
of available resource which inevitably means effort must be prioritised. This is a factor of varying
significance in all but the most exceptional of technical developments;
Ensure the framework is scalable with regards to the size of the development and to the level within
the WBS Structure to which it can be applied. This is closely related to the preceding item;
Provide a framework to supplement planning techniques to better capture aspects of complexity,
such as coupling behaviours between activities. This will also allow analysis to identify individual
activities or clusters of planned activities that have the potential to influence the overall
development outcome disproportionately and as such deserve enhanced management effort;
Guide the selection and implementation of development status measures. These should be
determined on the basis of the impact of particular areas of technical development and the key
activities within these areas. These should be selected primarily for their use in controlling the
development and initiating avoiding action and not merely for purpose of reporting status.
An underlying theme of the framework will be that the activities it directly influences (planning and
monitoring), and indirectly influences (controlling), are chosen to balance risk and benefit with effort
involved. It should complement current techniques, and absolutely not conflict with them. It is the aim that
this will allow their use with a minimum of additional resource overhead. It is recognised that the framework
will not be implemented in isolation of existing project management and system engineering tools and
techniques. A secondary objective is to allow project and process complexity to be understood through
practical usage of the techniques. This can be achieved through the feedback of learning back into the process
to improve it and develop better responses for future system development.
There are often latent issues within system development that are only manifested through delays, cost
overruns or quality issues. This is problematic for several reasons. The intervention required is generally
greater the later it is left. Also as the development tempo intensifies, the effort required to change the
process and organisation will inevitably draw on the very same resources as those developing the system
itself. The disruption and additional uncertainty of making these changes will exacerbate the impact of the
original latent issue. In fact, the scope of such a change can be seen as a project in its own right [8] and is as
undesirable as it is unnecessary. Following on, there is evidently great benefit in balancing the effort between
undertaking risk management as opposed to that of issue management. This balance is often forgotten and
especially where there is insufficient substantiating data to support the mitigation of risks or to support
change. The focus instead is on constant management of issues arising from bad risk planning, commonly
known in project management as ‘firefighting’ [9], leading to no long-term sustained improvements in
performance.
A goal of the framework is to provide strong enough indicators to prompt this early management action.
Many of the principles in the management of system related risks are equally applicable to project risks. This
can be illustrated by an annotated Bow Tie diagram [10][11] as shown in Figure 1. The role of complexity
management is both to identify the threats on the left hand side that are brought about by complexity and
to determine effective methods for controlling them.
Selecting CSFs should be followed by an understanding of residual risks and ways of providing an early
warning of their realisation. A semi-formal method for identification of CSFs within a framework will
encourage their wider use and raises the possibility of their use at development and project review points,
such as design reviews and stage gates. An objective should be the early identification and best possible
assignment of responsibility for the of the mitigation of development risks, not only across the breadth of
the project but to a sufficiently senior level of management [12].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-6-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 3 of 70
Heuristic techniques will be chosen and developed as a means of guiding decision making and providing
inputs into traditional project management and system engineering processes. These will anticipate areas
within system development that require the most effort and where effort will bring about the greatest
impact. This will provide multiple Plan- Do-Check-Act (PDCA) cycles, an iterative four-step management
method used to control and allow continual improvement of processes and product development [13]. Also
known as the Demming Cycle, it is equally applicable within the development process. In this application it
will be used to influence the development environment, processes and planning for complex system
development. The techniques described will use a mix of decomposition and reductionist methods, along
with a heuristic approach, to optimise resource usage required both of the framework and by any additional
development effort that results from its application.
Heuristic techniques are well suited to the problems associated with planning and executing a complex
system development and work well alongside iterative development techniques generally advocated for
complex system development. There are many variables in play within complex technical development.
While optimal solutions are achievable for a particular aspect, optimisation across all these often competing
aspects is very difficult within the current methodologies.
An important concept is that of feeding back learning, into the on-going development or subsequent
developments, to improve future applications of the framework. The determination of CSFs or risks should
include the intended or optimal outcome which can be compared against actual outcomes during the
development lifecycle for the efficacy of the CSFs and their implementation to be assessed. This allows
improvements to be made in a timely fashion for the benefit of the development. It also avoids the
unsatisfactory analysis of lessons learned, often at the very end of the project when personnel drift away and
the motivation to hold a review wanes [14].
A glossary of terms and acronyms used throughout this dissertation can be found in Appendix A.
1.3 An understanding of complexity
Complexity must be understood before it can be modelled. It has previously been described and decomposed
in a large number of ways, both within and outside the domain of engineering and many other overlapping
fields of research. Examples include the Strategic Highway Research Program [15] and Helmsman Institute
[16]. For the purpose of this framework it is important for complexity to be represented as simply as possible,
while covering all the necessary aspects as effectively as possible. This may appear counterintuitive but is
vital if the principles of the framework are to be applied in practice. This section will summarise research on
the nature of complexity and will develop it for use in a complexity management framework. A more detailed
explanation of the properties of complexity can be found within the literature survey [4].
Figure 1. Annotated Bow Tie diagram [11].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-7-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 4 of 70
Definitions of complexity highlight is highly structured nature [17][18] which encompasses many interactions
[19] and a large array of possible configurations [20][21]. Other characteristics include there being no
relationship between the behaviour of the overall structure and that of its individual parts and the ‘response’
of system elements to changes to other interrelated system elements [20][21][22]. Together these aspects
make the overall structure difficult to understand [23]. It is noted that uncertainty is often omitted from the
definition of complexity though it is intertwined with several of the aforementioned characteristics [24], such
as configuration variation. This project proposes that ambiguity be considered [25] as a distinct and separate
characteristic, alongside Uncertainty. Ambiguity is a common property of system development, caused by
the absence or unavailability of reliable information, and resulting in the formation of assumptions.
Ambiguity is a dominant factor in the early stages before requirements have been agreed and also during
development phases before coupling of large numbers of activities are fully understood. Assumptions cannot
be completely validated until later in the development and only after ambiguity has been addressed.
An understanding of how the individual characteristics of complexity interact with each other will be pivotal
to the determining of CSFs. Further dimensions of complexity will be elaborated upon in Section 2.
1.4 The importance of considering the organisation
The main two components of a system
development, apart from the actual system being
developed, are the development process and the
development organisation. These can be
considered to be within the direct control of the
management team responsible for system
development. Additionally, there will be a number
of external interfaces and constraints that will be
outside direct control of the system development
management team. These may be from within the
actual project itself, such as overall project budget
or resource availability, or entirely outside of the project.
Examples of the latter may be stakeholders, regulators and
existing adjacent operating systems.
An understanding of these boundaries will be important in
assigning responsibility for interventions. Figure 2 shows the
British Computer Society complexity model with tiered levels
of external interface. This is a simplified representation and
does not show how these tiers interact but nevertheless is
useful in categorising project influences. Specifically, the
model includes the ‘macro-environment’, which
encompasses interfaces furthest from the control of the
development team such market condition, legislation and
regulatory frameworks. ‘Micro-environment’ relates to the
customer and stakeholders and how requirements are
captured and traded-off [26]. The ‘organisational
environment’ and the ‘project environment’ relate to the
system development and the resource and process
constraints imposed on it. The organisational relationship
Figure 2. The British Computer Society’s
complexity model [26].
Figure 3. Type of relationship between
engineering and project management teams
(adapted from SEBoK [27]).](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-8-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
1
The ‘Iron Triangle’ describes The three related constraints of scope, budget, and schedule [32].
Page 5 of 70
between system engineering and the project environment is also an important consideration and something
which this model omits. Figure 3 shows three distinct project organisational types from the viewpoint of the
technical development and is adapted from the Systems Engineering Book of Knowledge (SEBoK) [27].
These organisational types strongly influence what is and what is not within the direct control of the system
engineering team. The definition of the inputs and outputs between project and system development
organisations will allow a better understanding of where the responsibility for putting in place and
implementing CSFs and risk mitigations resides. These interfaces should be defined within such documents
as the Systems Engineering Management Plan [28] and the Project Management Plan [29].
In the first diagram organisational behaviour is typified by transactional type relationships. Engineering may
be contracted out or undertaken by a different organisation, possibly located elsewhere. There may be more
than one engineering organisation, with complexity increasing according to the number of interfaces. The
project management organisation is responsible for managing the outputs only and there is a strong
emphasis on functional organisational structures [30]. In the second diagram the overlap in responsibilities
and accountabilities can also result in a form of complexity.
There is a lesser emphasis on transactional relationship and the project management team have a greater
influence over the management of engineering activities. The degree of overlap will affect this relationship.
The third diagram could be a matrix type structure. The project management team has overall control and
the engineering management team has the least scope to determine how the development will be managed.
The type of relationship will thus affect the scope and the method of managing complexity. In the first
example system engineering is entirely self-sufficient in terms of contractual complexity. In the last example
this is far less likely. Thus the treatment should be amended accordingly.
1.5 Existing management frameworks
There has been a large volume of literature on the subject of managing projects and technical development
activities based on a blend of theory and practical experience. The framework presented in this project will
take the existing management frameworks and models, many discussed in the literature survey [4], and will
attempt to reconcile inconsistencies in approach and content. This section will both consolidate the
discussion in the literature survey [4] and introduce concepts that have been found subsequently.
The predominant systems engineering model is the ‘V-model’ as shown in Figure 4. It does not specifically
address complexity but has some concepts that will be applied within the developed framework. Specifically,
its definition of the development lifecycle and requirements-centric view is useful. The satisfaction of
requirements is a convenient way of looking at complexity and the monitoring and control of requirements
should form an integral part of the way that a complex system is managed. The V-model represents a typical
idealised development lifecycle, so the phase descriptions within this model will be used to profile
complexity.
Of the system engineering and project
management methodologies that were
previously discussed in the literature survey [4]
the Strategic Highway Research Program’s
5DPM was of particular interest. This
methodology is described in some detail in both
their ‘Guide to Project Management Strategies
for Complex Projects’ [31] and also in ‘Managing
Mega-Project Complexity in Five Dimensions’
[31]. Rather than considering the usual three
project dimensions, i.e. the iron triangle1
[32], it Figure 4. The V-model [26].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-9-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 6 of 69
introduces the concepts of ‘financing’ and ‘context’ as shown in Figure 5. 5DPM also considers CSFs from the
perspective of complexity. Furthermore, this is done within several feedback cycles. Together these
represent considerable synergy with many of the concepts that the author previously felt were worthy of
consideration.
Finance becomes of utmost consideration as both the overall magnitude of the development and risk
increases. This is exemplified by the difficulties experienced in securing investment for the UK’s nuclear new-
build programme [32]. For the purposes of technical development, it is an imposed constraint by virtue of
being outside the direct control of the management and will be treated as such in the model. The fifth
constraint of the 5DPM model is that of context, describing the project environment, including potential
constraints and interfaces. The process can be decomposed as
follows:
Review project factors in each of the five areas;
Identify and prioritise complexity factors;
Develop 5DPM complexity map;
Define CSFs with sub-process steps such as assemble
project team, select project arrangements and prepare
early cost model and finance plan;
Develop project action plan to address resource issues;
Re-evaluate complexity map on commencement of
project.
5DPM describes reviews at defined intervals with an iteration of the above steps. This would most naturally
align with a Stage Gates type governance structure [29], but it could be envisaged that interim reviews would
be undertaken on a periodic basis if stage gates were deemed too far apart for effective control. Overall the
model does not try to model complexity itself but rather attempts to model the component parts (within the
five dimensions) and manage them to better manage a complex project environment. By focussing on these
five dimensions the resulting assessments may provide a superficial view of project management orientated
aspects only. Despite this there is considerable merit to the high-level approach and synergy with the
principles of the PDCA Cycle [4].
The Helmsman Institute complexity assessment [16] is advocated by the International Centre for Complex
Project Management [34]. This again has five areas of general consideration consisting of:
Context Complexity;
People Complexity;
Ambiguity;
Technical Challenge;
Project Management Complexity.
Within these categories there are specific factors. Those solely relating to technical development are:
Integration Complexity;
System Development Complexity;
Impact on Infrastructure.
Each of the categories is in turn considered against a large number of factors relating to such concepts as
stakeholder numbers and alignment, uncertainty and abstraction. Importantly this framework introduces the
concept of ambiguity as does Pich et al [25] and McGowana et al [35]. In each ambiguity is seen as a distinctly
separate, though closely coupled, property to that of uncertainty. The consideration of complexity is of
interest within the wider domain of business in general. The concept of VUCA (volatility, uncertainty,
complexity and ambiguity) [36], as shown in Figure 6, considers many of the previously discussed themes.
Volatility can be viewed as the result of one or the product of several complexity characteristics. Traits
attributable to volatility include a general lack of understanding of the likely impacts of an event
Figure 5. Five dimensional
project management model [15].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-10-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 7 of 70
along with the impacts being
‘unexpected’ and ‘unstable’.
Here volatility is described as a factor in
its own right and complexity is described
alongside the others rather than being
composed of them. Nevertheless, the
presentation of an unpredictable
business environment in terms of a
limited number of characteristics is
useful. Much of the literature references
complexity without sufficiently breaking
it down into its component parts.
Classical definitions of the components
of complexity were described detail
within the literature survey and are as
follows [4][37]:
The ‘Kolmogorov-Chaitin
complexity’, also known as
Program-size Complexity [38];
‘Nonlinearity’;
‘Emergent Complexity’.
Essentially the first component is the number of interfaces relating somewhat to the development magnitude
but also its nature. A large technical development may contain duplicate tasks and low interdependency,
thus limiting its Program-size Complexity, while a comparably smaller technical development may conversely
have many interdependencies and highly intricate activities. The latter two components collectively can be
closely aligned with the previously described volatility within the VUCA framework. A relatively small change
results in far-reaching or a disproportionately large impact. Expanding the definition of emergence for the
benefit of its use in an assessment we may say that it has a number of properties [39]:
Radical novelty – new and unanticipated properties emerge from each interaction;
Coherence – though unexpected the new properties are consistent with rules and behaviours;
Wholeness – emergence causes the new system that is greater than the sum of its parts;
Dynamic – changes from emergence will continue to evolve as long as there is emergence;
Downward causation — the system as a whole dictates the behaviour of its parts as well as the
system being influenced by the interaction of its parts.
This may be caused by two common dynamics, either separately or in conjunction:
No one is in charge – thinking in terms of a technical development organisation this could be
characterised by a network-centric organisation with a decentralised structure;
Simple rules engender complex behaviour – using the example of an organisation this may be a large
hierarchical organisation with many functions.
This section describes the decomposition and treatment of complexity from varying perspectives. The
development of the 5DPM process supports the view that simple consideration of the ‘iron triangle’ is
insufficient. It also reinforces the importance of complexity and the validity of an iterative approach.
However, its treatment of complexity is overly simplistic. The complexity framework developed in this project
will adapt the broad methodology of the 5DPM process, while proposing a more precise definition for
development complexity using a combination of prevailing theory and practical application. Specifically,
definitions of complexity outlined either in the literature survey or in this section will be re-categorised to fit
within the model, maintaining consistency.
Figure 6. The VUCA framework [36].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-11-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
2
The objectives described in the summary of the literature survey [4] will be adopted with a single change. The
development of Object Based Modelling (OBM) will be omitted. DSM will be used alone as a flexible technique that can
be used to model the development process. This of course does not preclude the use of techniques, such as OBM, in
parallel with DSM. Further information on the use of OBM can be found in the paper by Warboys and Keane [41].
Page 8 of 69
Complexity will be decomposed by applying it against the relevant themes within the technical development.
It must also be considered that complexity is in the eye of the beholder. The observer will have experience
and a perspective that will influence subjective analysis based on their ‘bounded rationality’ [40]. The
assessment criteria will ultimately require to be qualified as much as possible to guide interpretation. A more
detailed literature review of this area of research can be found in Appendix B.
1.6 Devising the Complexity Management Framework
This dissertation will define complexity in terms of technical development and combine a method of assessing
complexity with a number of techniques to allow the planning, monitoring and controlling of complex
technical developments. To summarise this dissertation will2
:
a. Identify a method of quantifying Technical Development Complexity;
b. Provide a semi-formal method for the identifications of the pre-requisite pre-planning environmental
factors, or so called critical success factors (CSFs). The outcomes from this method will be influenced
by an individual project’s characteristics, primarily in terms of its complexity;
c. Propose methods for planning technical development and modelling interdependencies using the
method of Design Structure Matrix (DSM);
d. Provide Performance Measures that relate to the technical development and, where possible, to the
assigned CSFs. It is preferable that these are totally objective but it is accepted that there will be
considerable subjectivity involved which will need to be constrained to enhance consistency by
suitable guidance. These performance measures will be both leading and lagging and provide
coverage of the development process biased towards the parts that are particularly significant for
reasons either of criticality or of risk of deviation in terms of criteria such as cost, schedule, scope
and quality. Coverage of the remaining parts of the development process will be at a lower degree
of decomposition commensurate with criticality and risk;
e. Propose methods of identifying risks and how this influences CSFs and the Performance Measures.
f. Demonstrate how all these (a. to e.) can be used together in a Complexity Management Framework.
Together items a. and b. can be thought of as pre-planning activities and together are a strategy for putting
in place an environment conducive for the project. Items a. to e. are developed in more detail throughout
this project, culminating in Section 7, which includes a framework map integrating all these items together.
2. Identifying and quantifying project complexity
2.1. The complexity assessment matrix
The complexity concepts described above will be encompassed within a Complexity Assessment Matrix. The
matrix will form the most important part of the Complexity Assessment which in turn forms a part of the
overall framework. The outputs from the assessment are as follows:
1. Undertake assessment of system development at a high-level within the WBS;
2. Identify of ‘hotspots’ both within the WBS and within the individual themes and direct additional
assessments where they are required. It should be noted that focussed assessments will not be
possible in very early development stages due to the low level of definition and decomposition of
the WBS;
3. Identify development risks and propose CSFs to counter significant levels of complexity. Record
additional information such as:
a. Rationale;
b. Residual risks that remain once CSFs have been put in place;
c. Proposed outcomes that are to be brought about by the CSFs.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-12-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 9 of 69
4. Reassess the project complexity profile with the CSFs in mind to ensure that no new significant
unmitigated hotspots have been created. Revise the outputs of step 3 as appropriate;
5. Identify risks for inclusion within the risk register.
Any consideration of complexity needs to be complete with an adequate explanation of categorisations. The
process will be applicable across the entire development lifecycle and should be scalable. It is to be expected
that there will be overlap between definitions due to a combination of minor shortfalls in the model and
observer perceptions. It is for this reason that the reasoning behind decisions should be recorded to afford
the best chances of consistency in applying the process in future applications.
As discussed complexity will be viewed from the perspective of satisfying requirements, whether technical
or project management orientated, and will form a part of the framework. There will be interactions between
complexity types and it is highly likely that the nature of the complexity will evolve over the development
lifecycle. We shall call this the Complexity Profile. Complexity assessment will have the potential to become
more detailed as the requirements are defined.
It can be reasoned that a pre-requirement, capturing the ‘business mission’ [28], can only be assessed at a
very high level while detailed system requirements can be decomposed to much greater detail. It is important
that the treatment of complexity can be targeted where it will represent the greatest value, as ambiguity
reduces between the derivation of stakeholder requirements and the final validation of the system [28]. This
can be achieved by considering assessment against either the development’s WBS or Product Breakdown
Structure (PBS), depending upon which is used. Lastly there should be a comparison between actual and
intended outcomes and feedback of learning to improve decision making in both the current and future
development processes. It should be borne in mind that the objective of the assessment of complexity is to
inform future decision making in order to enhance the likelihood of success. Section 3 details the use of CSFs
as an established technique to do this by reducing either the probability or impact of complexity related
events. Both complexity assessment and the assigning of CSFs can be viewed as pre-planning activities and
should be reviewed periodically for applicability and appropriateness.
2.1.1. Complexity themes
Using a combination of the methodologies described within the literature survey and Section 1.4, a typical
technical development can be considered to comprise a number of themes. These are captured under the
three broad headings of internal interfaces, external interfaces and system development. The themes are
designed to provide breadth of coverage over all areas pertinent to system development. Some themes, such
as stakeholders and regulatory interfaces could have been combined but it was felt that the identification of
independent themes would prompt more detailed analysis in areas that are particularly influential. This also
led to the apportioning of a third of the themes to the system development itself, an area underrepresented
in other methodologies such as those from 5DPM and the Helmsman Institute [32]. This ensures that the
assessment is more balanced overall and specifically in areas such as satisfying requirements and completing
scope as a part of technical development.
An important consideration is the project environment within which the development is undertaken. There
will be constraints imposed upon the technical development that will, to varying degrees, be out with the
control of the development management. The organisation type, as discussed in Section 1.4, will be a
significant factor but others include the traditional business-level requirements of cost, schedule and scope
[32]. Organisational culture and business-level governance, such Stage Gate type processes [29], will also be
influential in shaping the technical development. The investment constraints and funding profiles, as
elevated to one of the 5DPM dimensions [31], must also be considered. The remaining themes used in the
framework generally follow the commonly used themes of people, processes and technology [42]. These are
used in business architecture and can be decomposed further to better suit the purposes of the complexity
assessment.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-13-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 10 of 69
People can be decomposed into the internal organisation, i.e. the development team and those external to
it. External elements include contracting organisations, regulatory interfaces and all other remaining
stakeholder groups that contribute to the development. The properties of the development process are
important in terms of detail and number of interfaces. Lastly the technological aspect can be thought of as
the nature of technology that the system will comprise, as well as the integration of the technology (internal
interfaces) and external interfaces, i.e. how the system integrates with other systems. The nine Complexity
Themes used in the framework are listed below. These include high-level considerations that may impact the
analysis.
1. Internal factors
a. (Project) environmental constraints – external factors (inputs and outputs only), schedule
(such as pace and phased handover requirements), the imposed funding profile;
b. Development process – how prescriptive and many interfaces;
c. Internal organisation – ranging from few disciplines within same organization to many and
organisation type;
2. External factors
a. Contractual management – ranging from a few simple relationships to many relationships
with differing contractual terms;
b. Stakeholders - relating to system definition and validation;
c. Regulatory interfaces – how many and how influential they are on the development;
3. System development
a. External (system) interfaces – the degree to which system impacts or depends upon
external systems (from none to part of system of systems);
b. Technology – system requirements definition and verification relating to technology type
(low to very high tech and low to highly novel);
c. Internal (system) interfaces – the level of internal integration required between sub-
systems ranging from few simple relationships to many complicated relationships.
2.1.2. Complexity Criteria
Complexity Criteria can be assessed against each of these themes. The twin themes of uncertainty and
ambiguity will dominate the early phases of any project. Though they are often classically excluded from the
definition of system complexity they have such a profound effect on complexity management and will be
elevated in importance. Rather than be seen as sub-sets of complexity they will sit alongside the other
criteria. Uncertainty and ambiguity will naturally diminish under normal conditions as requirements are
defined, verified and validated.
Conversely emergence and nonlinearity will tend to increase over time as requirements are progressively
defined, implemented, verified and validated. They will reach their zenith as the system approaches
handover when even small changes will have a significant impact. The final criterion is that of Program-size
Complexity. This is essentially the quantity of information that represents the system development.
Techniques to manage this include expenditure of resource and effort (personnel and/or computer-tooling)
and the use of modelling techniques. The system development Complexity Criteria are:
1. Uncertainty – likelihood of unexpected events occurring, e.g. stakeholder requirements change
significantly during system early development or assumptions are found to be incorrect;
2. Ambiguity – incompleteness of knowledge about functional variables [25], i.e., a lack of information
upon which to make decisions. An example may be that stakeholder requirements are undefined in
a particular area of system development leading to assumptions being made or to delays in the
development schedule while the stakeholder requirements are defined;
3. Emergence – how change to system configuration leads to unexpected behaviours and interactions
and re-evaluation of derived system requirements. High Emergence is defined as unexpected](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-14-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 11 of 69
behaviours and interactions in many other components or sub-systems;
4. Non-linearity – the effect of a single small change to system requirements on other system
requirements. A non-linear relationship is one where a change results in a disproportionate change
in affected components and sub-systems. High Non-linearity is where a small change results in a large
impact elsewhere;
5. Program-size Complexity – relates to the minimum amount of information required to describe a
process, system or organisational requirements. Manifests itself in shortfalls in modelling and a
general low level of understanding in the system or its development even when uncertainty and
ambiguity is low. High Program-size Complexity requires a high level of effort to model and to
maintain the model. Examples where this may be manifested are requirements management,
configuration management and scheduling.
In summary, uncertainty and ambiguity dominate early in the development lifecycle when their major effect
is lots of low impact changes. Later in the development lifecycle there are less, high impact changes generally
brought about by emergent and non-linear behaviours. Program-line complexity impacts the ability to model,
understand and control these changes, with higher levels of this criteria making schedules and requirements
progressively more difficult to manage.
2.1.3. Complexity Matrix
The Complexity Themes and Complexity Criteria can be presented in matrix form. A scoring system offering
five choices, from very low to very high, has been chosen. Areas to be considered (considerations) for each
of the themes has been chosen from the literature survey [4] and Section 1.5. These looked at sources such
as the Helmsman Institute [16] and NTCP Framework [43]. Considerations provide guidance during
application of the assessment and aid in the assessment of appropriate areas of the WBS against each theme.
Risk has been omitted as a consideration from any of the themes as it is best understood only after complexity
assessment has been undertaken. In this way the assessment will differ from its peers who tend to consider
risk as an input into an assessment rather than an activity that is informed by an assessment’s results.
Considerations can be seen against the themes within the Complexity Matrix as shown in Figure 7.
The most obvious considerations for Project Environmental Constraints are those of imposed cost, schedule
and scope. The funding profile that may constrain the development schedule, as well as overall cost, and
introduce additional assumptions in lieu due to sub-optimal activity sequencing. Components of schedule
include pace, where the development schedule may need to be accelerated to meet external milestones, and
phased handover requirements. There may be additional activities and schedule implications due to project
governance. Other constraints include those relating to procurement and contract usage which may impose
constraints on who contracts are let to and how contracted activities are managed. There may be stipulations
with regards to the technology that is used, for example technologies developed in-house. Lastly,
organisational culture and politics can have a dramatic effect in areas such as communication and decision
making.
Tailoring of the development process to achieve the correct balance of governance and rigour against agility
and flexibility is an important consideration as is facilitation of integration between the technical disciplines.
The use of technology in areas such as requirements management, configuration control and safety case
management will influence not only the development process but also the organisation. The organisation
will strongly influence lines of communication. It is assumed that the fundamental type of organisation
structure is decided early in a project lifecycle. It is generally a difficult undertaking to change it in its entirety
as it is often dictated by the parent organisation undertaking the project. However, it is expected that it may
be influenced by the findings of the Complexity Assessment. Traditional hierarchical structures allow ease of
governance while decentralised network-type structures offer better flexibility [30]. Other aspects include
the number of individual disciplines and their particular experience with regards the development type, the
technology and integration challenges.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-15-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 13 of 69
The internal organisation will interact with contracted development organisations and will be required to
manage the transfer of information (development inputs and outputs) and the interfaces between areas of
scope to ensure there are no omissions or inconsistencies. The development organisation will be shaped by
its place in the wider organisation and particularly how it interfaces with the project organisation. This will
dictate whether the responsibilities for areas such as resource allocation, contract procurement, cost
estimation and scheduling are within the remit of the development organisation and the level of autonomy
it holds.
Other factors include geographical location of disciplines and other interfacing organisations. Contractual
management will of course depend on how much of the development is actually contracted out versus
activities that remain within the internal organisation. The reasons for contracting out of parts of the
technical development will vary, for reasons such as low internal resource or capability or where particularly
specialist or technology specific support is required. The nature of the relationship between the development
organisation and contract organisations will be dependent on the composition of the project organisation.
Responsibilities for actual contract management, including processing of contract variations and scope may
be outside the scope of the development organisation in that they may manage technical interfaces only.
Conversely this may all be within the remit of the development organisation. Again the division between
scope being undertaken by internal resource and that contracted out should be considered against the
outputs of the Complexity Assessment. The identification of significant areas of concern may initiate a
reconsideration of contracting strategy or else prompt the establishment of measures to better manage it
such as rigorous pre-qualification of suppliers. This is an example of a CSF that was introduced as a concept
in Section 1.1 and is subject to more discussion in Section 3.
Stakeholders can be thought of as the direct source of all high-level requirements. They inevitably have a
strong influence on requirements as they are derived in greater detail throughout the development lifecycle.
Factors that will affect stakeholder management include the number and alignment of stakeholders. Non-
alignment of stakeholders’ needs leads to trade-off of requirements to gain the appropriate approvals. Large
numbers of conflicting requirements will require careful management and introduces the risk for conflict and
delays. Other factors include location and availability of access to stakeholders. Incomplete stakeholder
identification may precipitate later changes to requirements and scope with potentially significant impact.
Long development schedules can increase the likelihood of changes to important stakeholders which can
lead to changing requirements also.
Regulatory bodies are important stakeholders which can have a tremendous impact on system requirements.
Technical developments may have weak interfaces, with only one or two regulators, or there may be strong
interfaces, with many regulators. Changes in legislation and regulations may occur at any time due to events
outside the control of the organisation and regulatory regimes may vary greatly between geographical areas.
The development of a system for use in several countries can introduce much complexity surrounding the
management of potentially conflicting regulatory requirements.
External interfaces are systems that interface with the system of interest. There may be few interactions of
a purely transactional nature with other systems or many interfaces. A system being developed as a part of
a system of systems [28] will invoke the greatest management effort. Interfaces may be not properly
understood or may evolve over time beyond the control of the development organisation. Technology is
simply how ‘novel’ and ‘high-tech’ [43] the technology that has been chosen and dictated by the
requirements is. Novelty and high-technology levels, as described by Frank et al [43], generally require highly
iterative development methodologies and high levels of expertise. Short development schedules using such
technology require greater effort, specialised systems engineering techniques and incur disproportionate
level of risk. System integration represents the ‘synthesis of a set of system elements into a realised system
that satisfies system requirements, architecture and design’ [28]. There are likely to be many interfaces that
need to be managed through integration, verification and validation activities. There are techniques to](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-17-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 14 of 69
manage this complexity, including early integration planning. Examples of other factors include the
integration of system elements from other contracted organisations and the careful management of
stakeholders to minimise ‘scoop creep’ [44].
Examples of Complexity Matrices, completed for the case studies, can be found in Appendices O and Q.
2.2. How, where and when to assess
Projects invariably utilise a WBS [45] to organise activities prior to cost estimation and scheduling activities.
The assessment considers each Complexity Theme against the five Complexity Criteria at each node of the
WBS on a particular level. Scores are assigned (very low, low, medium, high and very high) at each
intersection of the matrix. As the WBS increases in detail so does the effort required to undertake the
assessment. It may be justifiable to limit the assessment to relevant portions of the WBS or choose a high-
level within the WBS to restrict the nodes to be considered.
The complexity assessments should be undertaken against the portion of the WBS that describes the
technical development aspects. The WBS, including that describing technical development, will be subject to
refinement over time. As such any early assessment will be at a high-level on the WBS and necessarily at a
corresponding low-level of overall detail. There will be business-level requirements at the offset and other
factors, such as generic organisational processes and the existing regulatory landscape. Additionally, early
planning may make assumptions with regards to the organisation and contracting mechanisms. This pre-
requirements early assessment will identify areas of potential concern to enable early planning through the
identification of high-level CSFs.
As the project and the development progresses the WBS will be defined level by level with early development
phases likely to receive the most detail. The example of a WBS shown in Figure 8 has technical development
activities below several of the high-level nodes. In this example requirements definition would receive more
attention than test and verification activities due to the dependencies between the two and the inherent
difficulty in defining later activities because of this. Complexity assessments would be undertaken as a
minimum at the end of development phases and would serve to inform the planning of subsequent phases
and as an important input into Stage Gate type governance reviews.
Long project phases may demand interim complexity assessments to catch any significant changes during the
phase. Assessments would develop the earlier higher-level reviews to progressively lower levels of the WBS
to identify hotspots of complexity within the development. This technique affords the opportunity to
Figure 8. Sample WBS for an unmanned aerial vehicle (UAV) [46].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-18-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 15 of 69
disregard areas of low complexity and allow expenditure of effort where it needed. For the purposes of this
example WBS ‘control software’ and ‘verification software’ have been identified as having high levels of
complexity. This could prompt the assessment of the next level of the WBS below to determine the particular
node that is of particular concern. This can be repeated as required and, in the most extreme application,
until the lowest level of the WBS has been reached.
2.3. Outputs from complexity assessment
Identification of complexity should be used inform development planning, enabling the establishment of
conducive project environment. An established method is that of CSFs as environmental influences that will
enhance the likelihood of success. They may reduce either the probability or impact of the consequences of
complexity. Both the rationale and the proposed outcome of the putting in place of CSFs should be recorded.
This exercise should also prompt the identification of development risks. Some of these will be residual risks
as a result of putting in place of CSFs.
The putting in place of CSFs will influence the complexity profile. An example of this, developed later within
the case study in Section 8, is that of the development of Boeing’s Dreamliner. Radical changes in Boeing’s
procurement strategy were implemented at the very early stage of the project to improve schedule and costs.
This changed the profile of complexity within the project’s supply chain, introducing ‘’coordination risks’,
with well-publicised results [47]. Using the complexity themes, it may be that complexity within the internal
organisation are merely transferred into contractual management. It is therefore important that the
complexity assessment is repeated in areas where changes are proposed. Indeed, in the example of AREVA’s
design and construction of the Olkiluoto 3 nuclear plant the issues were entirely the reverse of the
Dreamliner and related with AREVA choosing to embark on a first of a kind project alone without the
necessary experience as ‘architect and engineer’,’ without experienced partners’ and without having all the
necessary competences [48]. Repeating the complexity assessment will identify any further areas of
complexity that have resulted from the implementation of CSFs and allow the impact of changes to strategy
to be understood. Additional CSFs can then be put in place or existing CSFs amended as appropriate. The
designation of CSFs is discussed further in Section 3.
2.4. Complexity profile and the interaction of complexity criteria
Complexity is influenced by many factors and is not a static property. It will evolve over the development
lifecycle and its nature may also be transformed by imposed changes. It can be theorised that Complexity
Criteria will interact in specified ways and will follow individual general profiles. Each criterion applies equally
to modelling of the development process through planning and scheduling as it does to the definition of
system elements and their interactions.
Uncertainty relates to the likelihood of
change of development requirements and
it will be the greatest at the beginning
when definition of requirements and scope
is at its lowest. Naturally as definition
increases and the number of assumptions
are reduced, uncertainty is also reduced.
Uncertainty can relate to changes
prompted from sources such as stakeholders, regulators or rework due to iteration or errors. Uncertainty
should reduce as the system is verified, and should diminish to its very lowest level as the system approaches
final validation. Changes, especially those later in the development, will reintroduce uncertainty. Ambiguity
is closely related to uncertainty and relates to the amount of information that is unknown. In the absence of
verified information, it is necessary that assumptions are made to enable aspects of the development to
progress. Without assumptions other activities will be delayed. Ambiguity will reduce as requirements are
Figure 10. The changing nature of the project process [25].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-19-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 16 of 69
defined but as with uncertainty it will
be increased with the introduction of
changes. Early definition activities will
reduce both uncertainty and
ambiguity together.
Two methods of detail with relatively
low levels of uncertainty and
ambiguity include the inclusion of
‘float’ [49] within development
schedules (sometimes known as slack)
and ‘contingency’ [50] within the cost
plan or the inclusion of flexibility within the design to accommodate a range of design definition outcomes
[51]. Emergence, as defined in Section 1.5, will increase throughout the development lifecycle as ambiguity
is reduced. This property is invoked by the interaction of requirements or by the introduction of change in
ways that cannot be anticipated. It specifically relates to the impact and the likelihood of emergent
behaviours arising. The impact of emergence can be managed through the decoupling of development
activities or system elements. Examples of managing emergence are that of removing activities with likely
emergent behaviours from the critical path of a development schedule or controlling the interfaces between
sub-systems through choice of system architecture. Non-linearity also increases as ambiguity is reduced. The
impact of non-linearity can be
managed by identifying such
interfaces and introducing extra
capacity in the affected system
elements to manage potential
future change. This should be
undertaken on the basis of risk and
the extra capacity can be either built
into the changed requirement to
prevent the change being
propagated or those affected by the
non-linear behaviours to absorb the
change with minimal impact. It is
highly desirable to reduce
uncertainty and generally control
the incidences of change as non-
linearity and emergence increase.
Considering this from the point of
view of requirements and scope it
can be reasoned that Program-size
Complexity of the development will
begin and end low. Requirements
will start as those necessary for the
business or operational need and
will be relatively few and at a low
level of detail. Requirement
Program- size complexity will rise as
requirements are defined,
Conceptofoperations
High-levelrequirements
Detailedrequired
High-leveldesign
Detaileddesign
Implementation
Integrationandtesting
Sub-systemverification
Systemverification
Operations&maintenance
Figure 11. Complexity Profiles over the development lifecycle with
no major changes and an incidence of major change during sub-system
verification (bottom).
Point of major change
Figure 9. Level of organisational effort over time [52].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-20-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 17 of 69
increasing in both number and detail. Conversely this property will reduce as requirements are verified until
the business and operational level requirements are finally validated. Scope is similarly represented via the
WBS and activity scheduling. Again these begin at a low-level of detail until the development is fully defined
and planned. As activities (scope) is completed the Program-size Complexity again reduces as the number of
activities to complete and their interactions reduces. This is broadly in line with organisational effort [56] as
can be seen in Figure 9. Changes will again increase Program-size Complexity with the example of previously
verified requirements needing to be reanalysed and adapted to accommodate a late change. Program-size
Complexity is managed through the application of organisational or computing resource and/or modelling
techniques. The magnitude and impact of Program-size Complexity over the design definition phases is
shown in Figure 10 [25]. Pich et al propose that not only does the complexity of information increase over
time but that its relative impact of information content on outcomes diminishes over time. This assertion
chimes with that of INCOSE who propose that approximately 70% of committed lifecycle costs are due to
concept design against 95% [28].
Figure 11 suggest how the profile of a typical technical development may be represented over its lifecycle
with an indicative measure of the magnitude of the particular complexity criteria against time with the
development phase also shown. The first graph shows no major changes while the second graph shows a
major change that is brought about during sub-system verification. This is manifested by an increase in
program-line complexity, uncertainty and ambiguity. Development phasing has been taken from INCOSE as
per Figure 4 on page 5. Such late changes present issues are there are now additional activities to manage
and potentially a large number of system elements, previously verified, need either to be completely re-
evaluated or re-verified against the new requirements. Uncertainty and ambiguity are re-introduced into
system elements that had previously been defined and built or constructed. The impact of non-linearity and
emergence is high due to the timing of the change and the number and nature of system dependencies.
3. Critical Project Success factors and their application in system development
Critical Success Factors were first proposed by D. Ronald Daniel in the 1960’s, after which they were further
developed by John F. Rockart of the Sloan School of Management [5]. Their application was primarily
intended within the areas of business strategy but have also been adopted in project management as success
factors [45][53]. It is the intention that the selection and implementation of CSFs should put in place an
environment more conducive with a successful outcome.
3.1. Selection of success factors from literature
Two mistakes that should be avoided are those of confusing a CSF with a requirement and assigning the CSF
as too high a level. The latter make the CSF too general for useful application. Requirements are distinctly
different from CSF and though their satisfaction is as at least as important, their management is handled
elsewhere. Examples include particularly important stakeholder requirements being associated with
‘Measures of Effectiveness’ (MoE) and system requirements with ‘Key Performance Parameters’ (TPP) of the
system of interest [54]. These will be discussed more fully in a later section but as performance measurement
metrics.
A CSF that is described at too high-level across all activities can be viewed as pre-requisites for any technical
development, project or business endeavour. Examples of high-level CSFs are provided by the APM Body of
Knowledge [45]:
Defining clear goals and objectives;
Maintaining a focus on business value;
Implementing a proper governance structure;
Ensuring senior management commitment;
Providing timely and clear communication.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-21-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 18 of 69
While these obviously worthy objectives, they apply to every project that the author has worked on.
Furthermore, they give no indication of how they might be judged as being satisfied. It is therefore important
that the CSFs do not merely represent best practice but apply to the demands of the particular technical
development as indicated by complexity assessment. As such, the CSFs selected for inclusion in the
framework will need to be supplemented with enough detail for implementation and, if possible,
measurement.
I have selected 141 CSFs for inclusion in the framework from a variety of sources, including APM [7], Pinto
and Slevin [52], Dvir et al [55], Chow and Cai [56], Ragatz et al [57], Koutsikouri and Dainty [58] and Fortune
and White [59. These are in turn divided amongst the Complexity Themes. Some of these CSFs are repeated
in more than one Complexity Theme. Ultimately each will be need to be developed with the appropriate
detail if chosen in practice and adopted for use. For example, ‘support from senior management’ should
include whose support should be in place and when and what this support will achieve. It should be targeted
against particular activities and should recognise the influence that development phasing may have on the
nature of management support that is required. This who, what, how and why approach can be applied to
most items on the list. The full list of CSFs for the framework has been grouped into the applicable Complexity
Themes and can be found in Appendix C.
There are a number of important concepts that should be borne in mind when applying CSFs. Merely stating
development best practice will dilute the effectiveness of the identification of CSFs and instead only
particularly areas of the process should be subject to analysis. As such general process improvement should
be addressed elsewhere. In the spirit of the incorporation of the principles of the Demming cycle, and the
resulting continual improvement, the proposed and the actual effect of the CSF should be recorded. This will
enable the effectiveness of individual CSFs and allow them to be catalogued within a toolbox of CSFs for
future use. To facilitate in the analysis of the effectiveness of CSF they should have performance measure
associated with them where possible.
3.2. Verification of success factors through questionnaire
Potential CSFs identified in the literature were evaluated for their potential utility in the framework via a
questionnaire. The questionnaire used the criterion of perceived impact on the likelihood of project success.
The questionnaire was posted online using the SurveyMonkey application [60] and a total of 122 responses
were achieved by requests via email and Linkedin social media. An overview showing the source of replies
can be seen in Appendix D. Respondents were a varied mixture of engineering and project management
professionals. Respondents were asked to rank the CSFs listed in Appendix C according to their potential
influence on a project into five categories from very low to very high. Any CSFs not considered relevant or
without an impact were removed from the list as being non-applicable. The questionnaire also gave
respondents the opportunity to identify other CSFs that the literature survey and subsequent analysis had
missed. The questionnaire and explanatory text is contained with Appendix E. The unprocessed results, per
respondent, are contained in Appendix R. Processing of the CSFs was undertaken and they are categorised in
Appendix F according to their scores:
4.0 and above - shaded in green (high to very high influence);
3.5 and above - shaded in yellow (upper scoring of medium to high influence);
3.0 and above - shaded in orange (low scoring of medium to high influence);
Below 3.0 - no shading and red type (low to medium influence).
These categories could then be filtered, sorted and compared across age, role description and industry.
Further processing was then undertaken to aggregate relevant age and role group. Data was disregarded
where this was not possible for particular role and industry groups. This ensured that data groups were
statistically significant, allowing the data from the selected groups to be meaningfully compared to the full
dataset. Information relating to the size of the aggregated datasets can be found in Appendix G.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-22-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 23 of 69
personnel had a very different perspective. The following CSFs, were not present in the other role
descriptions. The first two of these were second and third most important while the others descended in
relative influence within the ‘high to very high’ category.
Enhanced planning is applied against areas of criticality and uncertainty;
Fast transfer of information;
Test early, test often philosophy is used during development;
Past experience of management methodologies and tools is available;
Correct choice of management methodologies and tools;
Trouble shooting mechanisms in place.
Organisational - Findings across all role descriptions were closely in agreement with the full dataset with the
exception of the inclusion of’ composition of development team in terms of experience and capability’ within
engineering personnel.
Contractual management, Stakeholder management, External interface management and Regulatory
interface management - Findings across all these themes and across all role descriptions were closely in
agreement with the full dataset
Technology development management - Findings across all age groups were closely in agreement with the
full dataset though engineering personnel rated the CSFs general lower with only ‘test early, test often
philosophy’ achieving the highest ranking.
System integration management - Findings across all age groups were closely in agreement with the full
dataset though engineering personnel rated the ‘modelling and prototyping’ higher than the other role
descriptions in the top three.
3.2.5. By industry
Results filtered by industry can be seen in Appendix J. Sub-datasets were again compared.
Imposed project constraints - The findings were closely in agreement with the full dataset across the role
descriptions with a few notable exceptions within oil and gas. Perhaps as a reflection of an environment
reliant on oil prices and commercial pressures ‘effective change management’ and ‘political stability’ ranked
relatively highly.
Technical development processes - Construction differed significantly with the following CSFs figuring in the
rankings as ‘high to very high’, with responsiveness being the third most important. This reflects the dynamic
nature of a construction environment and requirement to act quickly during the construction phase to
minimise or avoid rework.
Responsive and flexible process to meet client needs;
Strong, appropriately detailed and realistic development plan kept up to date;
Appropriate development planning technique has been chosen;
Correct choice of management methodologies and tools.
Organisational - Energy generation elevated the ‘re-use knowledge and experience from previous projects’
which is a strong component of the nuclear following events such as Three Mile Island and Chernobyl
[66][67]. Oil and gas elevated the ‘collocation of teams’, perhaps due to the international nature of many of
its projects. Professional services included ‘domain-specific know-how’ and ‘appropriate techniques to aid
identification of organisational dependencies and interfaces’. Otherwise there was commonality between
the industries and most influential CSFs.
Contractual management - With some minor differences findings across industries were closely in agreement
with the full dataset.
Stakeholder management - With some minor differences findings across industries were closely in
agreement with the full dataset.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-27-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 24 of 69
External interface management and Regulatory interface management - Findings across all industries were
closely in agreement with the full dataset.
Technology development management - Energy generation attributed some Systems Engineering type CSFs
which reflects the complex nature of many projects in this domain. These included ‘modelling and
prototyping’ and the use of ‘system element maturity’. ‘Continuous improvement process for products’ was
also ranked third, well above any other industry.
System integration management - Findings across all age groups were closely in agreement with the full
dataset though energy generation rated the ‘system element maturity’ highly again.
3.2.6. Conclusion
Analysing the datasets by age, role description and industry does not yield any particular trends nor show
any of the sub-datasets deviate significantly from the findings over the full dataset. The following are very
high level observations:
The age groups ’55 to 64’ and ‘above 64’ show two thirds of all deviations from the overall sub-
dataset;
Particular biases were evident in the Complexity Theme ‘imposed project constraints’ that relate to
the ‘role description’ dataset;
The role description ‘engineering personnel’ shows half of all deviations from the overall sub-dataset;
The industries of ‘energy generation’ and ‘oil and gas’ show two thirds of all deviations from the
overall sub-dataset;
The Complexity Themes of ‘Imposed project constraints’ and ‘Technical development processes’
tended to deviate more across all sub-datasets than any the other themes.
From this is could be seen that particular biases should be guarded against when determining CSFs. A
workshop approach should balance age groups and role descriptions to ensure particular pertinent CSFs are
not overlooked. Furthermore, particular industries may place more emphasis on some CSFs over others. The
formation of a ranked list of CSFs will need to be tailored for its application. It would be advised that adoption
of the CSFs begin from either this list or, for a particular industry, from the nearest sub-data set as filtered by
the closest applicable industry type. Ranking of influence of CSFs could be further refined through subsequent
iterations of the Complexity Management process. Stakeholders could be consulted before and after
application.
4. System development planning techniques
4.1. Overview
An important aspect of the framework is that it will support and enhance planning activities. It has been
recognised that planning related objectives cannot be achieved in isolation of established planning tools and
techniques. Complexity influenced planning will need to be undertaken in parallel with current practice and
indeed maximum benefits can only be realised if they can provide readily used and value-adding inputs into
these established planning activities. This overview discusses the shortfalls and benefits of current practice,
the opportunities in adopting a suitable supplementary technique and the requirements that the technique
must fulfil. A suitable technique called Design Structure Matrix, also commonly known as DSM, will then be
discussed along with ways it can be integrated into current practice.
Gantt charts [63] are the dominant method of project and system development planning. The inherent
advantage over other techniques is their relative simplicity, which allows it to be an effective communication
tool. The use of Gantt charts also underpins project monitoring and reporting techniques such as Earned
Value Management (EVM) [45]53]. This popularity has precipitated the development of advanced tooling,
with packages such as Primavera, providing a way of creating the plan and integrating it with other processes.
Overall they are a good way of managing the overall duration of a project.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-28-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 25 of 69
There are however limitations in the techniques’ use that are increasingly exposed as complexity increases.
Although relationships between activities are represented they are restricted to simple dependencies [64].
These dependencies describe how an activity influences the start (and occasionally the finish) of other
activities including the use of lag. The nature of these dependencies cannot be further explored using this
planning technique alone. Thus the relationship between coupled activities is much simplified and likely to
be misunderstood without further analysis. As complexity and the number of dependencies increase, the
effort required in the production and management of the Gantt chart increases significantly. This leads to
more and more assumptions and simplifications of the nature of dependencies which can often result in the
actual start and duration of activities progressively deviating from the plan as it proceeds. This is obviously a
state of affairs to minimise if not to avoid.
It is unlikely that the dominance of Gantt charts in industry as the primary method of planning will change
anytime soon. Moreover, they have served as an incredibly useful way of coordinating and representing
activities across diverse disciplines and functions. With this in mind, other techniques should supplement or
feed into, rather than replace, Gantt charts, and should provide a way of enhancing the planning where
required.
Planning of all of the development activities across a project is a considerable undertaking which only
increases as complexity increases. It may be surmised that there is a positive relationship between the effort
that is required for effective planning and the effect of the number and potential impact of dependencies.
Other factors that deserve consideration within planning include the varying uncertainty and number of
ambiguity related assumptions. There is also an inverse relationship between the effort expended and the
risk of deviation from the plan. The balance between effort and risk is an important consideration in any
project where there are finite resources available. Such detailed planning has a high overhead and in a
complex project unfocussed planning can diffuse efforts leading to essential dependencies being overlooked
until it is too late. The direction of maximum effort to areas of risks should improve the likelihood of an overall
successful outcome.
Further to the earlier discussions within a survey of literature [4] and within the objectives within Section 1.2,
it would be sensible to use any enhanced planning techniques in areas identified as being either particularly
complex, or moderately complex but with a high potential impact in the event of deviation from the plan.
Using the principles of the Demming cycle, it is also beneficial to understand the impact of the use of
additional planning techniques to allow their use to be developed and improved for later planning of
development activities or future technical development. Lastly there may be opportunities to reuse particular
activity patterns or structures to reduce future planning effort that needs to be expended. To summarise it
is proposed that detailed development activity planning follows these concepts:
Will prioritise enhanced planning in areas identified through complexity assessment;
Determine where and which planning techniques to be used should be influenced by choice of CSF;
Uses enhanced planning to supplement rather than replace traditional project planning techniques;
Keeps a record of where system development planning has influenced the overall planning;
Records and uses learning to assess effectiveness of techniques and improve subsequent planning;
Considers opportunities to develop frameworks findings for re-use in subsequent projects.
4.2. Design Structure Matrix
4.2.1. Introduction
Design Structure Matrix (DSM) has been chosen to supplement traditional planning methods due to its high
functionality and flexibility, allowing it to be used over several domains of planning. DSM is based on a simple
graphical representation of the relationship between activities, called the square N x N matrix [65], and has
its origins in Graph Theory. It is also variously known as the N2
diagram or coupling matrix [28]. It was
developed by several independent parties but most notably by Eppinger and Browning who have refined the](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-29-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 26 of 69
technique, independently and in collaboration, through a number of papers and publications since the 1990s
[66].
The nature of DSM’s notation conventions varies and should be designed to suit the particular application.
For this reason, detailed descriptions will be omitted. For the purpose of this project the DSM as developed
by Eppinger and Browning, will form the basis of future discussions. The IR/FAD convention (as opposed to
the IC/FBD convention) will be adopted. Both variants are broadly similar but the use of rows and columns is
transposed [67]. Full and detailed descriptions of the technique can be found within their recent publication,
‘Design Structure Matrix Methods and Applications’ [67]. They are also associated with DSMweb.org, which
aims to ‘promote and foster’ further development of DSM [68].
DSM is flexible enough to cater for any type of system and can be further adapted for a particular process
and organisation. Furthermore, it is ‘highly compact, easily scalable’ [67] and can be applied where needed
at the level of detail required. The low level of abstraction, intuitive and graphical representation and the
relative density of information that a DSM contains makes it ideal for conveying important information as
both a standalone model or as a precursor for further analysis. It can provide system-level views of areas of
heavy interaction [67] but for the purposes of this project it is also ideally suited for focussed views on
particular aspects of the development.
4.2.2. Design Structure Matrix principles
DSM is a highly flexible technique that can be applied to provide a view of aspect of a technical development
either independently or in combination with another aspect, and can be summarised as follows:
Product architecture, i.e. the interactions of the system of interest itself;
Organisation architecture, i.e. how the development organisation interacts;
Process architecture which describes the development process ranging from high-level to detailed
planning;
Multi-domain Matrix, also known as MDM, which combines several of the above to show interactions
between the elements of the various matrices.
Process can be described as having ‘temporal flow’ architecture due to the inherent time-based interactions
that occur, with product and organisation possessing ‘static architecture’ [67]. MDM will be either temporal
flow or static, depending on its component DSMs. We will primarily be interested in DSMs associated with
process and organisational architecture and process/organisational MDMs as a tool to aid in planning of
activities. This does not however preclude the use of product DSMs or the inclusion of product architecture
in a MDM if the complexity assessment highlights an area of concern. The basic principles of the DSM are
very simple are as follows:
• Matrix elements being considered are represented along the diagonal of the matrix from upper left
to bottom right;
• Element names shown on rows and columns with the ordering of elements kept consistent on both
the rows and columns;
• Inputs are from left and right of the diagonal (rows) and outputs (columns from above and below as
per IR/FAD convention;
• Interaction between elements is shown in the matrix cells with the diagonal cells being blank.
A simple DSM, which merely acknowledges the existence of a relationship between inputs and outputs, is
called a binary DSM. The DSM however can be further developed into a numerical DSM, including attributes
such as importance, number of interactions and impact or strength of interaction. Additional attributes can
be linked to cells if required and elsewhere stored in a database. Whereas binary DSMs are qualitative,
numerical DSMs can be designed to be highly quantitative.
The process for creating and managing a DSM is relatively straightforward and follows a five step process.
Before this is begun all conventions will need to be agreed including supplementary information to be](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-30-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 27 of 69
contained within numerical DSM models. Suggested additional data to be collected includes interaction
strength as a single integer or as a combination of probability and impact. Later changes to notation and the
scope of information to be collected and model will be difficult to reconcile with previously completed DSMs,
which will potentially lead to confusion and duplicated effort. The inherent limitation of the DSM relates to
it describing the activities in terms ‘edges’ rather than ‘nodes’ [67]. For example, a binary process architecture
DSM cannot describe the duration of an activity. This can be overcome through the development of a
numerical DSM that includes information such as duration or resource.
The following sections will outline the essence of the DSM technique. A survey of literature relating to DSM,
describing the technique in more detail, can be found in Appendix K.
4.2.3. Creating and applying the organisational architecture DSM
The steps in the process for creating the Organisational DSM are as follows:
Decompose;
Identify;
Analyse;
Display;
Improve.
Once the DSM is clustered it can be used to inform management decisions with regards to the composition
of teams, geographical location and the types and scope of communication.
The integration of development activities can be extremely challenging. It can be used to shape the formation
of teams or clustering of teams based on concentrations of interactions. The application of collaborative
tools, such as databases, will be more appropriate for large groups while meetings and informal methods are
more effective for smaller groups. Lastly the co-location of particular organisational elements may be
beneficial.
4.2.4. Creating and applying process architecture DSM
Process DSMs are created using the same five steps as are used for organisational DSM. The level of
decomposition will be dictated by the previous complexity assessment and assignment of CSFs. The level
decomposition will also be influenced by availability information during the particular phase of development.
It is advisable that the elements are kept at a consistent level within the respective breakdown structures. If
a model of activities already exists this should be used and used as the planning baseline.
Key activities, pivotal to delivery of the entire process, should be identified. These are chosen based on
dependencies, coupling and risk. As such disproportionate effort should be placed on ensuring that these are
completed as per the plan over less consequential activities. These will be tracked throughout the project
with evidence of slippage or increases in risks above the accepted residual risk prompting additional
interventions.
Analysis of DSMs should enable a better understanding of the interactions between elements and allow the
plan to be amended accordingly. There are several relationship types within DSMs:
Sequential – where activities follow on from each other in a finish to start type dependency
relationship. There may be some overlap may be possible between activities which would be seen as
negative lag on a Gantt chart;
Parallel activities – these activities may rely on same resource but there is actual no dependency.
Resource constraints would normally be considered later using organisational architecture DSM or
on a resource loaded Gantt chart schedule;
Coupled – with iterations between the outputs and inputs between two or more activities. This is the
most difficult relationship to represent in traditional Gantt charts;
Conditional – execution of later activity dependant on decision made in an earlier activity. This may
or may not be sequential and in confined to the transfer of information.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-31-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 28 of 69
The traditional interaction
considered within planning is that
of a finish to start dependency
where one activity needs to be
completed before another can
commence, known as sequential
within a DSM. This can be further
refined by the inclusion of positive
or negative lag to either delay or
advance the commencement of
the following activity by a specified
duration. Other variations exist, including start to start (which suggests parallel activities) or finish to finish
(which suggests a coupling type relationship). It is these relationships that define the effectiveness of a
process or plan and it follows that the greatest impact is achieved by better managing the interfaces between
activities. It will also be worth considering representing iterations as separate activities
Of the relationship types it is coupling that is very often the most troublesome to manage and commonly
causes delays. These relationships most often rely on iterative outputs from one activity into one or more
other activities. It is difficult to predict the number of iterations that may be ultimately required. Other
variables include the duration to review and accept an iterative input into another coupled activity and
difficulties in planning the resource necessary for such interactions. The more activities involved in coupling,
the more closely coupled activities are and the number of anticipated iterative cycles all potentially impact
the overall duration.
There are several types of coupling. Some of these can be eliminated or reduced in terms of their impact
through effective planning and/or management of process. This is an example of a process behaviour that
should be fed back through the process for CSF consideration. Coupling behaviours may be planned or, due
to errors, unidentified interaction type or emergence, be unplanned. Coupling is generally as follows:
1. Inherent coupling – planned coupling behaviour with activities that are structurally interdependent;
2. Poor activity sequencing – information is created too late resulting in the delay of other activities.
Though a planned coupling this type of behaviour can be minimised, though not entirely
eliminated, through effective analysis;
3. Incomplete activities – where activities are unduly delayed with a similar impact as poor activity
sequencing;
4. Poor communication – information or outputs are not passed on completely or in timely fashion
again delaying subsequent activities;
5. Input changes – caused by changes to assumptions;
6. Mistakes – defective inputs created and discovered at a later date.
Evidently coupling types 2 to 6 are to be avoided whether through planning or subsequent controlling of the
plan. Delayed inputs/outputs can result in the formation of assumptions and indeed this is sometimes
adjudged as being a desirable response to a coupling behaviour. Adopting such an action simply exchanges
coupling behaviour behaviours 2, 3 or 4 for the potential for behaviour 5. Inherent coupling suggests
necessary iterations of outputs/inputs between two or more activities. Though necessary, most often even
desirable, convergence to a solution should be encouraged as quickly as is practicable. Additionally, the
reduction in the number of feedback loops is highly desirable also. Indeed, this has the potential for use as a
measure of development status and a measure of complexity in itself.
One goal of the analysis of a process architecture DSM is to optimise sequencing of the activities so that as
many interactions as possible are below the diagonal. Doing so ensures that activities only begin once all
inputs are available and reduces the number of assumptions that need to be made (avoiding the potential
Figure 12. Process architecture DSM [68].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-32-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 29 of 69
for input changes type coupling) or activities being delayed unduly (avoiding poor activity sequencing type
coupling). In the extreme example an output from an activity in the upper right hand corner indicates that
assumptions will need to be made for an early activity to progress. If the assumption is subsequently proved
incorrect there may be the propagation of changes throughout all the activities with substantial impact on
resource usage and schedule completion. There will be conflicts between multiple activities that constrain
the sequencing. The result will be a reordering of the rows and columns.
Another aim of analysis is the removal of coupled relationships through a process call ‘clustering’ and ‘tearing’
[67]. Instead of removing unnecessary assumptions this process introduces assumptions to make the overall
process or plan more efficient. Activities are rearranged into a ‘block’ so that they are grouped around the
diagonal. Assumptions are then applied that remove the ‘inherent coupling’ behaviour. There is however a
risk of the emergence of an ‘input change’ coupling behaviour at a later date which may reintroduce the
coupling. The decision to engage in tearing of activities within the DSM will therefore be based on the risk of
later input changes against the benefit of improving the efficiency of the process by the removal of inherent
coupling.
The discussion of where the process or plan may break down should be undertaken during the collation of
information on activity interactions and particularly for coupling behaviours. The use of Failure Mode Effects
Analysis (FMEA) may be of use to analyse potential points of failure leading to unplanned process iterations.
If significant these should be fed back into the complexity assessment and CSF processes and recorded on
risk register.
Sequencing
Sequencing optimises the ordering of activities. This can be done before the first draft of the DSM, in which
case the effect will be dramatic. There is usually a natural or intuitive ordering of activities so sequencing
performed on an established DSM is not likely to initiate wholesale change. It is a useful process nonetheless.
The process follows a number of steps. The first step, and the easiest, is the identification of activities with
either no inputs or no outputs. Activities with no inputs can be undertaken first and activities with no outputs
undertaken last. The sequence of all remaining activities will be influenced by the number, strength and type
of interaction with other activities depending on whether binary or numerical DSMs are being used.
There are a number of heuristics that can be used to undertake sequencing. The first is minimising the
number of assumptions that are required in the model by reducing the number of interactions above the
diagonal. This reduces the number of activities that need to be completed later in the sequence that feedback
to earlier activities. The second is reducing the distance of the remaining interactions above the diagonal. As
described previously long feedbacks can be the cause of the propagation of change throughout the
development process. These activities are then identified as ‘coupled blocks’. The identification of these
coupled blocks can be used to identify early where additional management effort would be well spent.
Examples include the co-location of those involved, the use of collaboration tools and application of status
and progress measures. Heuristics are of primary interest in this project through constraints of space but also
in fitting with the project’s overall philosophy. Algorithms and commercial applications are available with
potential benefits in terms of both expended effort and accuracy over a large quantity of data.
Clustering and tearing
The identification of blocks of coupled activities can be used for further analysis of the DSM. Decomposition
of coupled blocks of activities may yield a more manageable set of sub-activities. In contract aggregation can
be used to simplify the DSM but this will obscure individual feedbacks and is not generally recommended.
The addition of activities earlier in the sequence can be used to reduce the number of assumptions. The
decision to add activities will usually be made on a cost-benefit basis and should be used to reduce
uncertainty and risk of change. Tearing reduces the interaction between a block of activities by introducing
assumptions in place of iterations. Following the clustering of activities into blocks the process is a follows:](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-33-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 30 of 69
Suggest interaction to be removed, called a ‘tear’. The link or links with the longest feedback loops
back providing the greatest benefit and should be considered first;
Analyse the tear and, with the agreement of the stakeholder within the process, accept or reject the
tear considering the degree of confidence on the assumption that replaces the interaction. This is a
risk-versus-benefit based decision. If the first tear is discounted move onto the next best tear and
repeat the analysis;
The tear removes an interaction which reduces the block of activities in a particular cluster. The DSM
can now be sequenced;
Feedback marks in the DSM are replaced with torn marks to provide a prompt to check activity
outputs affected by the tear against the new assumptions;
Disproved assumptions will result in input change type coupling behaviour and rework of early
activities. If this occurs the tearing exercise was unsuccessful and the intended efficiencies were not
realised.
Single assumptions can apply across multiple tears
Commercial DSM tooling software that incorporates many of the more advanced algorithms and concepts
that have been developed for sequencing and clustering and tearing is available. The most popular and
commonly used of these are referenced on DSMweb.org [68]. None of these packages can directly interface
with the most commonly used scheduling application employed within complex projects is Oracle’s
Primavera [69]. This suggests that the general maturity of the applications has some way to go. Several of
the applications do profess outputs compatible with Microsoft Project, a scheduling application which (in the
author’s experience) tends to be used to plan smaller projects.
4.2.5. Application of the process DSM within a complexity management framework
The ultimate purpose of DSM analysis, in the general case, is to optimise the matrix which will influence the
creation of a time bound schedule. Analysis, as a part of a complexity framework, is no different. There is
however an additional rationale for analysis that is consistent with the principle of focusing on areas of
concern against which management effort can be directed. Analysis can not only optimise the activity
sequence and interventions, but can also identify clusters of activities and single activities that have either
the potential to unduly influence the overall schedule or else are of particular concern. This then affords the
opportunity to address current issues, mitigate risks or put in place measures to provide early issue
identification.
It is important to note that the DSM is less useful in showing attributes such as start dates, durations or the
lagging or leading nature of dependencies. This is one of the most powerful reasons to retain Gantt charts
alongside the use of DSMs. Thus the DSM can be used as an input into the Gantt chart and provide more
information on interaction related risks. DSMs can also be used to drive techniques such as PERT. In this
instance supplementary information relating to likelihood and impact would be contained with the DSM.
Development of a process / organisation architecture MDM can be used to identify required expertise and
resourcing requirements for particular activities and better manage the interface between organisational
and activity driven requirements. It can be used for ‘resource loading’ [70] of the Gantt chart and schedule
to facilitate resource smoothing or levelling of the schedule and implementation of EVM methodologies. It is
proposed that there should be a tiered application of DSM following complexity assessment and assignment
of CSFs. These increase in detail and also effort according to the risks and benefits of application within the
particular area of the technical development. A suggested ranking, descending in terms of effort, is as follows:
Not applied and traditional planning techniques are used;
A high-level application within WBS between elements of the system development;
Application of a binary process DSM at lower level of WBS;
Application of a binary MSM between process and organisation;
Application of a numerical process DSM;](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-34-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 31 of 69
Application of a numerical MSM (process and organisation).
Areas of concern are analysed at a lower level in the WBS with integration of the process and organisation
via the use of a MSM. The use of numerical matrices has the potential to provide extremely rich information
for use within other planning processes. However, this detail inevitably comes at a cost so should be reserved
for areas of the development that are likely to require the greatest degree of planning and control. It is likely
that analysis will highlight additional areas of concern that should be fed back into the complexity assessment
and CSF processes. It is also advised that the use of DSMs at a high-level with the WBS should be undertaken
early in the development lifecycle. The identification of interactions at an early stage may influence the
composition of the WBS and the ordering of WBS elements within the schedule. Since WBS elements are
invariably placed in the Gantt chart in accordance with a WBS numbering scheme it is desirable to place as
many of the WBS elements in the correct sequence as is practicable. This will then aid the logical flowing of
activities from top left to bottom right of a Gantt chart.
It is useful to develop a numerical process architecture DSM convention which incorporates the principles
described within the complexity assessment. This will validate the initial assessment findings and allow for
the identification of any additional CSFs that arise from changes after further analysis. For simplicity the
measurement of uncertainty, ambiguity, emergence and non-linearity will be captured numerically variously
using several convections. Some will be absolute integers while others will be subjective. The additional
information should be determined alongside the inputs, outputs and interactions which are described above.
Some or all of these values should be contained separately within a spread sheet or database to the DSM
matrix itself as to do so would clutter the layout.
Uncertainty is directly related to the probability of change related to a DSM activity. An integer relating to
the likelihood of a change in an activity’s outputs or duration will be adopted. A percentage is often used and
this can be translated into a whole number between 0.1 and 0.9 with 0.1 representing 10%, i.e. very low
likelihood, and 0.9 representing 90%, which is indicates that the outputs are likely to change. No activities
will be assigned 1, i.e. be planned with a 100% chance of change. Change may be due to input changes or
mistakes. Ambiguity impacts the development in terms of the number of assumptions that are required and
delayed start dates if assumptions are not to be used. The number of assumptions can be counted with high
number of assumptions being related to higher levels of uncertainty.
Emergence is where there are unexpected interactions [44], i.e. more activities are affected by a single
change that is originally envisaged. High emergence is where many activities are so affected. Therefore, it is
sensible to assign a value to an activity equivalent to the number of activities that rely on its outputs.
Conversely a value can be assigned to each activity that represents the number of inputs it depends upon.
The number of outputs from a single activity identifies which activities, if subject to change, are particularly
influential on the wider DSM. The number of inputs shows how sensitive an activity is to changes elsewhere.
Non-linearity is seen as a disproportionate impact resulting from a change. This can be represented by a
ranking from 1 to 10 with 1 representing a slight impact on dependant activities and 10 being a substantial
impact. Of course the impact of a change to an activity may not be homogeneous across its outputs to other
activities. For the sake of simplicity an average measure of non-linearity will be proposed for this model.
Figure 13. Assigning additional attributes to process architecture DSM activities.
Activity Uncertainty
(Also known as
likelihood)
Ambiguity
(number of
assumptions)
Emergence Non-linearity
(Also known
as impact)
System requirements – sub-system X 4 5 12 outputs
5 inputs
7
System safety assessment – sub-system x 3 1 5 outputs
5 inputs
9](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-35-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 32 of 69
Program-size Complexity is directly related to the number of activities to be managed. It is not considered
useful to measure this attribute to manage at the level of activities. Measurement of Program-sized
complexity would be more useful at a higher level, for example to measure the size of a work package,
development phase or the project in its entirety. In these instances, suggested measures would relate to the
number of activities or interactions between activities within the chosen area of interest.
Figure 13 shows an example with fictitious data for two sub-systems and examples of data that could be
assigned against the various properties.
Completing a numerical DSM in this way will allow further analysis. Case studies [67] demonstrate that the
assignment of probability and impact (called uncertainty and non-linearity in this text) to an activity can be
used to perform PERT analysis on the schedule. There are however several other ways that these and the
other remaining attributes can be used for analysis. In general, particular thresholds can be filtered and
sorted to identify trends or group activities together for analysis and shading and colouring employed for
ease of reference and communication of activities of particular concern. In each case there is the potential
for tooling to automatically make the necessary calculations based on the inputs into the DSM itself.
Additional management effort and status and progress measures can be introduced to specifically address
areas of concern.
The pairing of poor attributes relating to uncertainty/ambiguity and emergence/non-linearity is certainly not
desirable. Section 2.4 discusses the profile of complexity across the development lifecycle. There is proposed
that uncertainty and ambiguity should naturally reduce over the development lifecycle while emergence and
non-linearity generally increases. High values of three or four of these properties against a group of activities,
especially later in the development lifecycle, show an area of risk. Even a single activity with a similarly poor
complexity profile, if on the schedule’s critical path, deserves closer attention. High uncertainty and
ambiguity later in a development lifecycle may be the result of process or schedule optimisation activities,
such as clustering and tearing, where assumptions are introduced to reduce the schedule duration and
sequential activities are instead undertaken in parallel. This in turn may be as a result of either the original
over-optimistic technical development completion milestone or because of delays demanding a new
schedule.
Another technique is to analyse activity feedback cycles to determine those that are particularly uncertain.
Feedback cycles with a high likelihood of change that coincide with the critical path will be of particular
concern as any delay, due to rework, will directly impact on the completion of technical development. To
calculate the likelihood of change across a three activity cycle the following calculation can be undertaken:
Likelihood of change across a feedback cycle = 1-{(1-a)(1-b)(1-c)}
Where a, b and c are the uncertainties for the three activities in the particular cycle. This essentially calculates
the likelihood of any of the activities changing within the feedback cycle. More activities and high likelihood
of change will produce a low measure.
One method of addressing this is through the introduction of additional activities to reduce the number of
assumptions and thus the potential for input changes. The introduction of other management controls,
described in additional CSFs, should also be considered. Finally, two or more sequential or coupled activities
with large attributes of emergence have the potential for substantial propagation of change. It may be
difficult to reduce the numbers of impacted activities (for example tearing and clustering to reduce
dependencies may instead simply result in input changes) but further reducing uncertainty, even when
already relatively low, is sometimes possible.
Taking the analysis further other activity measures can be derived from its inputs and outputs. Activities
particularly vulnerable to change propagation can be determined by calculating the product of uncertainty
and non-linearity for each input to an activity and taking their sum. Large calculated amounts will highlight
an activity that is at risk of change. For example, activity Y has four inputs. The inputs have products of](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-36-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 33 of 69
uncertainty and non-linearity of 1x8, 4x4, 3x6 and 7x3. The sum of these is 63 (8+16+18+21). This can be
mapped across the DSM using colours or shading to denote activities of low or high vulnerability. Similarly,
activities with high potential to influence change can be determined by calculating the product of uncertainty
and non-linearity for each output and multiplying this by the number of outputs. In addition to allowing effort
to be directed against particular activities these measures will readily allow change impact analysis to be
undertaken and influence decision making. A relevant example of the potential benefit is that of a seemingly
small non-essential change which has the potential for a substantial impact across the schedule. In this
instance the change could safely be argued that it should be disallowed. So in summary the following
measures can be calculated:
Primary measures;
o Uncertainty – subjective measure of likelihood of change, determined in conjunction with
stakeholders during formation of DSM;
o Ambiguity – determined through the number of assumption required for an activity;
o Emergence – determined through the number of inputs and outputs for each activity;
o Non-linearity – determined as the general impact of change of an activity’s outputs on other
activities;
Secondary measures;
o Likelihood of change across a feedback cycle – particularly useful if activities coincide with
the schedule critical path;
o Vulnerability to change propagation – calculated from the uncertainty and non-linearity of
an activities inputs;
o Change propagation Influence - calculated from the uncertainty and non-linearity of the
activity multiplied by the number of outputs it has.
The calculation of mean amounts for the primary measures can give an indication of the complexity profile
for a portion of the overall technical development based on the boundaries of the DSM. Complexity profiles
were discussed in section 2.4. This could be across the entire phase or across the activities currently in
progress and could have one of two purposes. The first would be to provide norms for technical development
that could be used for later developments. Complexity profiles deviating markedly from the norm would
require further investigation. The second would be the identification of trends. This is particularly useful in
identifying when uncertainty and ambiguity remain consistently elevated or are increasing. Either would be
a cause for concern. Coupled with natural increasing emergence and non-linearity, high uncertainty and/or
ambiguity pose a high risk for significant change propagation. The identification of important activities,
activity cycles or clusters of activities should direct the implementation of status and progress measures.
These measures will be discussed in section 5.
4.2.6. Process-organisational MDMs and their application
The creation of a MDM requires both DSMs to be available for the particular area of the development and
for it to be of sufficient maturity. Much of the effort required is expended creating the donor DSMs and the
principle of the MDM is similar to that of
the DSM. Although there are a number
of types of MDMs that can be created it
is a process-architecture MDM that is
particularly applicable to the planning
and control of complex technical
development processes. An MDM can
be either of the binary or numerical type
but unlike the DSM the MDM does not
have a diagonal. It will show the
Figure 14. Process-organisation MDM [68].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-37-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 34 of 69
interaction between particular organisation elements and the overall process which is useful in determining
the impact of a deficit in resource or capability across activities. It will also show the organisational
requirements of a particular activity.
Analysis of the MDM may show a cluster of organisational elements that are particularly influential during a
particular phase. It may be beneficial to temporarily co-locate a number of teams or make additional efforts
to integrate their interactions. The demands of the schedule can be inferred from the MDM allowing resource
and capability requirements to be planned in advance. The MDM can be populated with the estimated level
of effort that is required in terms of man hours or cost. Missing organisational requirements may be identified
by the absence of interactions of MDM rows. The findings of this analysis can be used in cost and resource
estimates and be used to resource-load the Gantt chart to allow the application of EVM.
5. System Performance Measures
5.1. Introduction
The assigning of Performance Measures and the gathering and processing of project data to calculate them
is the ‘check’ activity within Demming’s ‘Plan-Check-Act-Do’ methodology. It is a vital component between
planning and controlling of a project that should inform if and when action should be taken. Effective
Performance Measures should additionally provide information on what type of intervention may be
effective. They may indicate a deviation from the plan but also the emergence of undesirable properties
within the development process. Pertinent examples of such an undesirable property are poor outcomes
after the implementation of a CSF or a large backlog of configuration change requests.
The control and management of on-going development activities is dependent on relevant and accurate
information on both the status and progress of the activities being undertaken. Technical development
processes, as compared to normal business processes, are typified by a number of unique properties which
makes effective process monitoring important [72]:
‘Dynamic, creative and chaotic’;
Contain many feedback mechanisms;
The process in its entirety is largely ‘virtual’ and ‘not always precise’;
The likelihood of change is high due to feedback mechanisms, imperfectly defined requirements and
customer led changes.
Without the appropriate choice and effective implementation of Performance Measures issues may go
undetected until they are manifestly apparent, by which time they may be unrecoverable. Additionally, it is
useful to know where management effort needs to be directed most efficiently. Examples of existing
frameworks used for the formation and collection of measures include House of Quality, known as Quality
Function Deployment, Goal-Question-metric and Balanced Scorecard [73]. The use of metrics within concept
and development activities is not widespread and can lead to a number of ‘coping’ mechanisms being
adopted by personnel in response to them. Simon and Simon list a number of issues that have been identified
during an empirical study [74] and as thus should be avoided. Similarly, Demming describes particular ‘traps’
to be avoided [75]. Conversely Kline et al [76] describe a method for designing performance measures that
is applicable across any type of technical development. The design and implementation of performance
measures should be designed to avoid aspects that reduce effectiveness and embrace those that improve
their chance of having a positive impact.
The measures that are chosen should also suit the particular development by its particular characteristics.
The method recommended by this dissertation is to identify areas of complexity and criticality through the
complexity assessment and to measure the implementation of CSFs. Two ways of looking at the performance
measures is in terms of areas of the WBS and complexity criteria listed and described in Section 2. Both can
be used to tailor the performance measures to balance effort of collection and analysis of data against the
benefits of closer and more accurate monitoring and control. However, the constraints inherent in this](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-38-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 35 of 69
approach are extensive research, undertaken by the previous literature survey [4] and additional research
within this dissertation, revealed the limited number of techniques currently available. This was reiterated
by the findings of questionnaire survey which revealed no additional performance measures from amongst
the 122 respondents. Additional research, undertaken since the literature review, can be found within
Appendix L.
5.2. Desirable properties of Performance Measures
Performance Measures, often call metrics, can be placed in several categories. Mathematical system
performance measures are calculated from fundamental data. These metrics are either derived or combined
and an excellent example is that of the metrics relating to the established method of EVM. In this
methodology uses primary data, relating to the actual costs and physical progress, is compared with cost and
schedule planning to derive measures such as cost and schedule variance and cost and schedule performance
indicators. Practical system metrics are derived from the application of ‘empirically established factual logic’.
Heuristic system metrics are similar to practical system metrics but the scope of the metrics is restricted to
particular issues [73]. Depending on the level of decomposition the metrics described in this dissertation are
most likely to be either practical system or heuristic system metrics. EVM is used extensively within projects
and technical developments and the questionnaire survey will use it as a baseline against which the influence
of other performance is measured.
The properties of the measures are of importance. Foremost the measures should of course have a clear and
obvious purpose and it is also beneficial that the measure is as ‘homomorphic’ as possible with the source
data [73]. The nature of the measures should provide a view of developments objectives ranging from the
short-term to the long-term [73]. Measures should be process orientated as well as schedule and cost
orientated indicators [73]. In accordance with the concept of the balanced scorecard there should be breadth
to the perspective of the measures that are adopted [77]. Measures may be either lagging or leading with
staple measures inevitably relating to cost and schedule. An example of a common system of measurement
is those used within EVM which reconciles planned cost and schedule against actual cost and progress. In this
instance it is also a lagging measure. That is an issue is flagged only once activities fall behind schedule or
estimated costs are exceeded. If the issue is determined early enough action can be taken to converge with
the original plan. However, the severity of the issue is greater the later it is detected and both low sensitivity
of data and the use of monthly reporting cycles may even delay this late detection by a month or two. In this
instance it is much better to predict the occurrence of cost overruns and schedule delays through the use of
leading measures. These measures will target areas of the technical development that will later influence the
actual activity durations and costs.
Other properties that should be considered are their frequency of application and it is important to balance
the effort of application against the benefit that may be derived from the frequency of measurement. A
measure may require a lot of time and effort to determine, for example the findings of an audit. It may
measure the properties of the development that do not change frequently, such as the schedule for the
current phase. In both instances it is appropriate to only determine the measure once per development phase
or perhaps only three or four times per year. Exceeding this frequency of measurement may entail the use
of inaccurate, incomplete or otherwise estimated data or not provide sufficient additional benefit to make it
worthwhile. Conversely the measure may follow a dynamic development property. The status of
development artefacts may be automatically recorded via the database or tooling that is being utilised. This
allows the measures to be taken more frequently or even viewed real-time. The cadence of measure
reporting may also be dictated by time-bound criteria. The frequency of EVM reporting is constrained by the
monthly frequency that actual costs are most often calculated which in turn relates to the submission of
invoices by contractors and consultants. Measures can be quantitative or qualitative. Often it is difficult to
provide quantitative data and only a commentary on the on-going activities is possible. Even when
quantitative is possible it is worth combining with qualitative to provide valuable context to the information.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-39-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 36 of 69
Examples of leading measures include development resource and the quantity development change requests
that have been submitted during a period. They should highlight early concerns either through an absolute
quantity or through a trend over successive reporting periods. Using the first example from above a deviation
below the planned level of development resource may not yet be reflected in a delay in the schedule. Early
identification would allow management action to be taken to elevate resourcing to a suitable level thus avoid
a later schedule deviation. An increasing trend in the number of development change requests may indicate
inadequacies within the existing requirements or stakeholder management issues. Furthermore, Increases in
resourcing requirements and funding may result from an increase in the amount of change. Again early
identification of changes can be used to prevent or minimise an adverse impact on the development
activities. It has been proposed that the selection of measures could be based on process analysis. There are
several barriers to this approach [73]:
There is a high overhead to process modelling which increases processes are already in place;
Analysis and implementation of measures needs to be undertaken early for maximum effect but
process uncertainty is high and change is likely. This can lead to wasted effort and/or incomplete
analysis;
Changes will continue to propagate through the project as contracts are let and strategies evolve;
The measures chosen by such a method tend to be too abstract.
Though some Performance Measures are used in almost each and every project (EVM for example) it is logical
others are chosen on the basis of particular development characteristics. Within this framework the selection
should be influenced by the complexity assessment and the subsequent sub-processes and following some
simple concepts. Specifically, the goals of the measures will be determined with a focus on areas of particular
complexity and risk, including the assessment of areas described in CSFs. The Goal-Question-Metric method
is an exceedingly straightforward way of determining measures. The ‘goal’ of the measure consists of four
components; purpose, issue, object and viewpoint and would be directly related to CSFs and risks. The
‘question’ typically asks for the current status and current trend in this status. The ‘metric’ is perhaps the
most difficult aspect to determine as meaningful measures that are at a low level of abstraction can be
difficult to determine. ‘metrics’ will be very strongly influenced by the range of measures that are currently
available within industry or else are developed during the project itself. An illustration of the Goal-Question-
Metric approach is shown in Figure 15 [78].
The level of decomposition should be considered and the WBS can again be used as the means for this. Low
levels of decomposition provide an overview of development health but will not provide a real indication as
to the source of any issues as the node within the WBS will be influenced by all of those below it. A high-level
of decomposition may suffer from low sensitivity of the data due to lower sample sizes or chosen method of
measurement and will certainly require a good deal more effort to implement. An appropriate level within
the WBS should be chosen for the highest
level of decomposition and it is perfectly
feasible to use metrics following the same
principles at simultaneously at different
levels of decomposition for both
management reporting and to facilitate
targeting response to issues arising.
Methods of data collection, and thus
sensitivity and effort, should be
commensurate with the overall goal of the
Performance Measure.
Performance Measures can be selected to
focus on particular areas of concern within Figure 15. The Goal-Question-Metric method [78].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-40-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 37 of 69
the development. This may be through a higher level of decomposition for a particular phase or cluster of
activities or else the choice of a unique measure. Methods that can direct where measures are to be chosen
include during the complexity assessment and assigning of success factors and during subsequent planning.
In particular DSM can highlight areas of intense coupling and complexity. Such prioritisation of activities may
be due to their properties of uncertainty, emergence or non-linearity and their proximity to the schedule’s
critical path. The use of focused measures must be in conjunction with other less detailed measures to ensure
coverage of the entire schedule of activities. It is also beneficial to overlap metrics, where possible, to provide
a degree of confirmation of the findings [73]. Other factors to consider include:
Metrics should be complete, correct, consistent and clear; [73];
As they will be communicated and reported metrics should be both ‘simplistic’ [73] and be chosen at
an appropriate level of abstraction so they are readily understandable;
The collection of meta-data during and as a part of the individual activities, as opposed to periodic
and dedicated data collection, will reduce the management burden required. An example would be
progress measurement during the production of development artefacts, at the end of each individual
iteration. This will introduce a level of ‘automation’ [79] into the process;
Consideration should be given to the collection of metrics by project assurance functions rather than
the development team [58] to ensure impartiality.
The reliability of some Performance Measures is necessarily subjective due to the scarcity of formal methods
of collecting data in many areas. Subjectivity can be curtailed through the adoption of rules and guidance in
the collection and analysis. A simple example is the progress measurement of development artefacts as
shown in Figure 16, and this also incorporates the principle of creating meta-data during the activities
themselves. It this case the artefact owner would track progress according to the relevant guidelines which
could be placed on a shared database for ready collation and analysis.
Artefact completion milestones Percentage complete
Completion of first draft of artefact 50
Completion of first review 60
Inclusion of first comments 80
Completion of second review 85
Inclusion of second review comments and submission for approval 90
Artefact approved 100
Figure 16. Simple method of measuring progress in development artefact.
Many Performance Measures are highly abstract. There are a number of these that relate to DSMs and these
should be considered for use. They however largely relate to complexity so while they may provide a
validation of the other analysis that has been undertaken this will relate to areas of risk within the technical
development. Examples of the areas of the development process that should be measured include [80]:
Schedule and cost planning – will measure risks, bottlenecks and the critical path as well as the overall
cost. It should also consider process iterations and associated uncertainty;
Resource allocation – will measure resource and capability levels as compared with the plan and
issues such as resource smoothing/levelling, removal of redundancy and accessibility of resource;
Quality – this will measure the consistent flow of information, completion of documentation in line
with process and meeting requirements and the distribution of risk amongst processes;
Flexibility – will measure the status of buffers to absorb delays and defences against individual errors
and general process resilience;
Organisational decomposition – will measure to determine whether the organisation of workgroups
and teams is adequate, efficient communication is in place;](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-41-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 38 of 69
Interfaces – will measure which entities need synchronising, speed of communication across
interfaces and the relevant communication paths in place;
Transparency – will measure whether the organisational units are aware of their impact on outcomes
and the mental model of the process organisation.
Decision making – will measure which decision points have a high impact on outcomes.
Also artefacts to be measured using a rating based on interdependency and risk:
Number of dependencies;
Origin of dependencies – with organisational area, interdepartmental, inter-contract, external
dependency;
Status of dependencies – i.e. the meta data derived for them;
Percentage of document that requires dependencies;
Identified risks.
Focusing effort on particularly critical areas of requirements management is achievable by using the
Complexity Assessment and identifying nodes on the WBS. This can identify when Performance
Measurement should commence as well as the depth and breadth of measurement that is appropriate.
5.3. Selection of System Performance Measures from literature
In this section, we discuss the measures which have been selected from both the previous literature survey
[4] and further research as a part of this dissertation as being significant for inclusion in the Complexity
Management Framework. Each measure will be categorised against the previously described criteria in
Section 5.2. They will be chosen for their credibility of previous application and breadth of methodology and
aspects of development for which they are used. These will be verified by a questionnaire survey in terms of
their comparative influence. EVM will not be described in detail but will be used as a benchmark in the
questionnaire survey against which the influence of other Performance Measures can be compared.
5.3.1. Requirements
The management of requirements is a fundamental part of technical development and the two central
techniques relate to the measurement of outcomes (requirements satisfaction) and process (linked to a
variety of attributes and metadata).
5.3.1.1. Satisfaction of Stakeholder and System Requirements
A fundamental tenet of technical development is the elicitation and satisfaction of requirements. In turn
these requirements can be broadly sub-divided into stakeholder requirements and system requirements.
Requirements, especially system requirements, are prone to successive iterations and coupling behaviours
as represented in the DSM model.
The measurement of requirements is described within a number of systems engineering manuals and lately
INCOSE’s System Engineering Handbook. They observe both stakeholder and system requirements from the
perspective of their success in fulfilling the overarching business requirements [28][54] and as such are
quality orientated measures. The technique identifies a limited number of particular Stakeholder and
Systems Requirements whose realisation (satisfaction) are critical or important to overall success. Selection
of these can be influenced by Criticality Assessment though they are often closely linked to high-level
business and operational level requirements that are determined at the very beginning.
With respect to complexity criteria, the analysis of requirements allows monitoring and control of the system
development and particularly the technology and internal interfaces. Other complexity criteria will have a
direct impact on requirements with excellent examples of such criteria being development process, internal
organisation, organisation and stakeholders. Measurement is facilitated through design review type activities
and provides lagging measures. This period of lag is of course dependant on the frequency of design review.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-42-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 39 of 69
5.3.1.2. Requirements attributes
Another method of measuring System Requirements is through the collation of associated information to be
stored in a matrix, spread sheet or database. This is sometimes known as the Requirements Verification
Traceability Matrix [81]. The metadata can be compared against the plan or trends monitored to identify
areas of concern. The complexity of the RVTM should be proportional to that of the technical development
and may include fields that are specifically chosen to assist in the management of foreseen project and
system development risks identified early using techniques including the complexity assessment. Suggested
fields to be collated include [81]:
System Requirement unique identifier and name;
Requirement description;
Requirements specification (document reference);
Overall requirement status (such as detailed in design, implementation or integration);
Trace to overarching Stakeholder Requirement (unique identifier and document reference);
Verification and validations procedures (document references);
Verification strategy status (such as undefined, strategy only, procedure completed);
Verification status (such as not started, failed, completed with reservations or completed and with
document references as appropriate);
Validation purpose (such as for acceptance, certification, readiness for use or Qualification);
Validation status (such as not started, failed, completed with reservations, completed and with
document references as appropriate).
Other fields that could be employed in either the same artefact or complementary artefacts, including:
Stability (susceptibility to change);
Design compliance;
Interface compliance;
Process compliance;
Risk and risk status;
Safety case and licensing compliance;
Procurement and contractual compliance;
Importance or criticality of requirements within the project schedule.
These measures may give an indication of requirements against the planned development and are leading in
nature as they indicate potential issues ahead of requirements satisfaction. They do however allow for the
identification of areas of concern. More generally requirements can be given a wide variety of metadata that
can be used to interpret the overall status of the requirements or a particular sub-system or area of
development. Not all of these attributes should be chosen due to the overhead required to maintain
complete and accurate data sets. Attributes should be chosen early in the development lifecycle to ensure
that data does not require to be retrofitted at a later date [81].
Both the RVTM and requirements attributes are primarily concerned with quality and identifying issues
relating to progress. Many of these attributes, together or independently, can be used to infer the efficacy
of resource allocation, organisational decomposition, interfaces and Decision making. For instance, a backlog
of requirements with a particular status may indicate one or more bottlenecks in the process. This may be
due to issues with resourcing, organisational efficiency or decision making. It can be used to identify risk at a
high level of decomposition within the system of interest which can be rolled up to show particular sub-
systems which deserve additional attention [81]. If trends are identified early enough, for example a backlog
of under review requirements within a particular development area, then action can be taken to reduce or
eliminate delays to the schedule.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-43-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 40 of 69
5.3.2. Development health
Development health is likely to contain a high degree of subjectivity. Facets such as adherence to the planned
cost and schedule can be measured through EVM or other similar techniques. Other less tangible aspects will
require a framework to provide context and guidance in which they can be assessed. Using a defined method
Kline et al [76] formed a weighted matrix of design performance characteristics that can be used to measure
development health. Each high-level measure was formed of five individual factors with each of these being
assigned an individual text description as a basis for scoring. The high-level measures are as follows:
Problem Definition;
Prior Knowledge;
Divergent Thinking;
Professional Analysis;
Decision Making;
Create & Follow Plan;
Iterate & Assess;
Validate Solutions;
Communication;
Teamwork.
By observing attributes of the development process and team the technique is a leading indicator and should
be able to identify risks to the development before they are fully realised. It should be undertaken reasonably
infrequently, say at the beginning of a phase or mid-development phase, at three or four month intervals, if
it is particularly long in duration. It is proposed that the Kline et al methodology is applied, partly or in full, to
form new measures or tailor exist measures for use in technical development. One candidates for this
approach, mentioned in the Literature Survey [4], is Torbet et al’s six criteria of ‘Design Performance
Measurement’ [82];
Client needs (stakeholder requirements);
Integrating design into objectives (system requirements);
Internal design processes (suitability and effectiveness of internal design);
External design processes (suitability and effectiveness of external design);
Profitability and efficiency (of the design);
Learning and innovation.
There is of course considerable overlap between these and requirements orientated measures and with the
criteria within the complexity assessment. Another way of identifying the subject for measurement is their
identification during complexity assessment and designation of CSFs. Alignment of specific performance
measures with area of concern broadly follows the principles described in planning via the use of DSM. That
is focus on areas where there is particular risk of deviation from the plan while planning (in this case
monitoring) at a lower level of decomposition level across the entirety of the development. The headings
within Kline et al’s ‘Creating and Using a Performance Measure for the Engineering Design Process’ relate
particularly to the complexity criteria of the development process and internal organisation.
5.3.3. Process maturity
A framework for process maturity is the ‘Project Definition Rating Index’ or PDRI [83][84]. Maturity can apply to
both the development in terms of where it is within the process or the technology that is to be used within
the system of interest. There will necessarily be some overlap and influence between the two. Though not
directly related to development health or schedule, there can be some inference as to the status of these.
For instance, in an extreme case technology of low maturity which is approaching validation suggests either
problems within the development process or either badly planned or incompletely executed activities leading
to that particular point in time.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-44-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 41 of 69
Of course there is still much potential for subjectivity within the scoring of the PDRI framework but this can
be further refined by providing individual guidance for each criterion as is documented in Kline et al’s
development health methodology. The result of
the scoring can then be compared with the
‘maturity value rating’ and ‘qualitative criteria’
[84]. A maturity rating that does not correspond
with the particular phase of the development that
is currently in progress indicates a concern. This
can be further analysed to determine the areas
where the issues reside by looking at the individual
criteria. Though similar in methodology to Kline et
al’s method it is a lagging indicators as it essentially
observes the status of the development at a
particular point in time. Again a review at the
beginning of a development phase or every three
or four months would be appropriate. Process
maturity measures relate directly to the
complexity criteria of development process but
also indirectly to the three complexity criteria.
5.3.4. System maturity
A popular method of assessing the system with the United States, and especially within military and space
industry domains, is that of the Technology Readiness Assessment (TRA) also known as Technology Readiness
Level (TRL). Specifically TRL has been developed for use by both the US Department of Defense and NASA
[86][87][88]. TRL assesses the maturity of the system as a means of analysing development risk and used in
a similar way as the PDRI methodology. That is if the TRL is below that required for the development phase
there are serious risks to the realisation of the technology which will be ultimately be reflected in an inability
to verify or validate the technology.
TRLs generally have ratings between TRL 1 and TRL 9 with subjectivity of analysis constrained through the
use of textual guidance. TRL 1 is the least mature and TRL 9 denotes technology proven within its intended
operational environment. In both of these applications of TRL a team of subject matter experts will assist in
providing the actual rating which will form the basis for stop/go type decisions by a board of appropriate
managers at predetermined points in the development such as development stage boundaries. TRLs relate
directly to the complexity criteria of technology. A development of TRL, and an attempt to address to address
TRL’s inherent limitation, is the System Readiness Assessment or SRA. SRA provides a ‘whole system
perspective’ [89]. It assesses all system components, the integration of them along with external
dependencies and is designed to be undertaken more frequently than those using solely TRLs [89].
TRLs are assigned to the system of interest as before along with an Integration Readiness Level or IRL. Criteria
are provided to guide the assessment as per the traditional TRL method. From these three System Readiness
Level Metrics are derived through calculation. Component SRL looks at individual components and how they
are integrated to identify elements of the system that are lagging behind the system as a whole. Composite
SRL is concerned with the overall integration of the system and this is then converted into an integer between
1 and 9, called the SRL [88]. Assessments and calculations are performed across the system architecture and
the ratings are similar to those of TRLs. The IRL relates to the complexity criteria of internal interfaces while
SRL is relates to the three system development complexity criteria as a whole.
5.3.5. Organisation, process and schedule complexity
DSMs were the subject to discussion in Section 4.2 and are useful in both defining areas of the organisation,
process or schedule that require particular attention and in providing useful measures to provide a basis for
Figure 17. Maturity value rating criteria [85].](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-45-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 42 of 69
management action. As they relate to the plan they are reasonably static in nature and as such lend
themselves to plan validation type activities at the beginning of a development phase or at a new iteration
of the schedule. Previously the use of DSM related performance measures were discussed in Section 4.2.5
and suggested calculating median and peak quantities for measuring plan complexity. Observed DSM primary
measures can be either trended through successive iterations or compared against acceptable absolute
values that were determined through previous developments. These can either consider the phased activities
as a whole or the current schedule critical path. The secondary measures of likelihood of change across a
feedback cycle and vulnerability to change propagation could be used to inform rescheduling or other
interventions to be undertaken to reduce development risks to acceptable levels.
The basic properties of the DSM itself can be used to provide an insight into the schedule and its complexity.
The relevant proportion of sequential, parallel, coupled and conditional activities within a development
phase can be used to provide an indication of its status. A high incidence of coupled activities especially, gives
a clear indication of complexity. Again this could be applied against the critical path activities only to provide
information on these priority activities. Finally, Kreimeyer and Lindemann [73] list four metrics using the
language of DSM optimisation activities:
Sequencing – the number of ‘ideally sequenced’ activities within the DSM, i.e. sequential activities;
Tearing – the number of activities that have been subject to tearing due to them being a barrier to
sequencing;
Banding – the number of activities that are independent to each other, i.e. parallel activities;
Clustering – the number of ‘mutually related’ i.e. coupled activities.
As a way of identifying areas of concern within the plan this is a leading indicator and should prompt re-
planning such as further optimisation of the DSM. Process and organisational-architecture DSM related
performance measures are concerned with controlling development process and internal organisation
complexity criteria.
5.4. The verification of performance measures through questionnaire
The performance measures chosen from the literature reviewed during the project were verified with the
practitioner group as a part of the questionnaire described in Chapter 3 above. The objective was to rank
influence and comparison with the most widely used performance measure from the EVM methodology. The
questionnaire can be seen in Appendix E with performance measures relating to questions 13 and 14. Results,
ranked by influence, are shown in Appendix F.
Influences were generally lower than that attributed to CSFs, with results across the entire 122 respondents
ranging from 3.04 (just above medium influence) for the use of PDRI and 3.77 (with 4.0 indicating high
influence) for measurement of requirements process. This may suggest a general dissatisfaction with the
identification and practical use of performance measures and in agreement with the author’s own
experiences. No other measures were suggested for use, providing some assurance that the literature survey
was comprehensive in its review of the subject. The measurement of the realisation of stakeholder and
system requirements was also rated highly and just below the top performance measure at 3.76.
Measurement of plan complexity through determination of proportion of sequential, parallel, coupled and
conditional activities on the schedule was third with 3.69. This be achieved through deeper analysis of
planning through traditional Gantt charts or, as advocated here, using DSM methodologies.
EVM did surprisingly poorly at eighth of the twelve measures considered with a rating 3.56 It did rate the
third highest number of very highs but suffered a lot of lower scores which offset this. This suggests the high
value of integrating requirements management into a project and that EVM is not sufficient alone. This is not
reflected in the practice where Earned Value is often the only performance measure used and requirements
require scant attention within project management methodologies [4]. Complexity measurement using
heuristics against the WBS elements also rated relatively poorly but only just below EVM at 3.48. This](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-46-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 43 of 69
supports the author’s opinion that absolute values of complexity are of limited value alone but support the
following sub-processes. Its value is qualitative rather than quantitative.
There were possibly shortfalls in the survey and barriers to Performance Measure use that influenced
individual scoring of some of the Performance Measures. Providing a sufficiently brief description and
without additional explanatory text may have created a bias against several of the measures. The PDRI
project maturity and project health measures scored very poorly and this along with the apparent effort of
application (200 individual criteria for PDRI and 10 high-level and 50 low-level organisational and process
characteristics for process health) may have depressed the final scores. Further research would be needed
to either confirm or disprove this.
Two comments from question 14 provided further validation of the author’s approach and initial
observations.
“Keep the complexity measures as simple as possible!”
“Many forms of performance measures out there, I think you have captured most, personally I don't rate them
and although their aim or managing system interfaces and relative maturity thereof is correct - they generally
don't add much value. There would definitely be mileage in getting back to basics, systems definition,
boundary limit definition and design review.”
5.4.1. Performance measures by age, role and industry
Findings were consistent over age, role and industry and especially over the top three rated performance
measures across each dataset. Considering the top three performance measures notable exceptions were as
follows:
Earned Value Management was considered the third most influential performance measure’ by over
65s’. This elevated the importance of EVM over any other sub-dataset.
The measurement of percentage complete of project artefacts and the measurement of IRLs were
considered the most important second most important performance measures respectively by
‘engineering personnel’. These reflect the concerns and challenges that are particularly faced by
engineering disciplines over other project personnel.
‘Defence’ valued TRL, IRL and SRL over all other performance measures which was not particularly
surprising when it is considered that these measures originated from the defence sector.
5.4.2. Conclusion
The findings of the questionnaire support the assertion that further work is required to develop performance
measures that better represent the challenges faced by today’s projects. Requirements in particular should
receive greater attention which in turn will encourage their better management. Plan complexity is the other
measure highly worthy of development. The combination of the measures of requirements and plan
complexity in conjunction with EVM should provide an excellent way of monitoring and thus controlling many
project types. This should be supplemented with TRL and IRL methodologies when the novelty of technology
dictates. This of course needs to be considered against the resource and cost overheads involved in such an
undertaking.
5.5. The collective use of performance measures
The use of the Balanced Scorecard [78] is much publicised within industry and has been further refined into
‘The DMI Design Value Scorecard’ [90]. These concepts provide little information as to what and how
measurements are taken but the principle of presenting together a broad selection of relevant measures has
merit. Using the broad principles identified from through the concept of Balanced Scorecard it is proposed
that a Development Scorecard could be developed. This would present a variety of metrics together to best
represent status across the breadth and depth of the development. It would present a high-level view of the
development for director level management, along with a text-based summary to provide context and
indicate underlying areas of concern that otherwise may be hidden. Below this the measures would be](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-47-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 44 of 69
presented at a number of levels of decomposition. Strategic measures would be primarily of interest to the
management leading the development to provide data on particular areas of concern within the process or
organisation. Tactical measures would provide feedback for team leaders and individuals to make
interventions at the level of the work being undertaken. The performance measures should be chosen to
address concerns raised during the complexity assessment with regards areas of high complexity or criticality.
For example, the following measures could be combined in a development scorecard:
Plan complexity consisting of:
o Average uncertainty, ambiguity, emergence and non-linearity as derived from DSM
(leading);
o Proportion of sequential, parallel, coupled and conditional activities within DSM (leading);
Conformity with the plan consisting of one or more of the following:
o Earned Value Management
Cost Performance Index (lagging);
Schedule Performance Index (lagging);
o Overall progress using heuristics with guidelines to aid analysis of individual development
artefacts;
o Actual resource and capability compared to plan;
Process consisting of:
o Others such as numbers and processing times for engineering change control requests
and quality non-conformances;
o Process health using Kline et al method (leading);
o Process maturity using PDRI (lagging);
Requirements consisting of:
o Realisation – relating to MOE, MOS, MOP and TPM (lagging);
o Execution – relating to selected requirements metadata (leading);
System maturity consisting of:
o Technology Readiness level (lagging);
o Integration Readiness level (lagging);
o System Readiness Level (lagging);
Bold type indicates measures assessed as having the greatest influence from the questionnaire survey. The
high-level measures are in italics and would be rated from very low to very high. Strategic measures are
contained below each of these. This would provide coverage of most of the complexity criteria.
6. Managing system development risks
6.1. Introduction
‘Management of risk should be systematic and not based on chance. It is about the proactive identification,
assessment and control of risk that might affect the delivery of the project’s objectives’ [53].
The process described so far identifies ways of analysing the technical development for areas of concern
against which effort can be best directed. An important part of this is identifying development risks which is
traditionally achieved through a risk management processes that uses techniques such as workshops and risk
reviews. This process can be used to supplement this approach, principally in the identification of risks, as it
prompts analysis of various aspects of the development and at varying levels of decomposition.
The initial complexity assessment is used to instruct on the areas of particular complexity and CSFs necessary
to increase the likelihood of success. Residual risks will remain should the CSFs be imperfectly implemented
or only partially address the raised concern. These risks can then be placed upon the risk register. This initial
complexity assessment is likely to generate high-level risks only due to the low amount of decomposition
possible and available detail on the development process. Successive iterations of the complexity assessment](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-48-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 45 of 69
however will allow risks to be targeted and existing risks will be elucidated upon, confirmed or dismissed as
not relevant. The planning process will identify still more risks and these will be focussed upon particular
types of interactions or clusters of activities. Again these should be compared to risks identified earlier as a
means to improving the quality of the information upon the risk register. A recognised and respected
approach for managing risk is contained within the PRINCE2 project management methodology. PRINCE2
bases much of this approach on Management of Risk: Guidance for Practitioners as published by the UK
Office for Government Commerce. Their guidance follows a number of important principles which are
satisfied by risk management through the complexity management process [53]:
Understand the project’s context – complexity assessment looks across the breadth of the technical
development;
Involve stakeholders – achieved at complexity assessment and particularly during DSM planning
activities;
Establish clear project objectives – as highlighted during the initial complexity assessment;
Develop the project risk management approach – partially satisfied through complexity management
process in conjunction with the general risk management process;
Report on risk regularly – dependant on the frequency of complexity assessment and planning
iterations (possibly partially satisfied);
Define clear roles and responsibilities - as highlighted during the initial complexity assessment
(possibly partially satisfied);
Establish a support structure and a supportive culture for risk management – through an independent
risk management strategy;
Monitor for early warning indicators – through complexity management process monitoring
activities;
Establish a review cycle and look for continual improvement – partially satisfied by complexity
assessment and planning iterations.
Incorporation of the contribution of complexity management within the risk management strategy will
provide additional support to the process and prevent duplicated effort. Complexity management will aid
the early identification of risks. Conventional risk management techniques are well described in literature
and risk and project management methodologies. Rather than define a particular approach to be used
alongside complexity management only the few relevant points will be discussed. A template of the risk
register, the artefact ultimately produced by any risk management process, is contained within Appendix M.
6.2. Important concepts
The accommodation of risk management alongside complexity management should not present too much
additional effort. Two of the components of the process are particularly important. The classification of risks
can be undertaken using a pre-determined risk breakdown structure or RBS. The RBS is a hierarchical
structure following the broad principles of other breakdown structures such as the WBS. The RBS contains
successive levels of decomposition and there are as many levels to the RBS as is useful for the purposes of
risk management. An example of high-level risk categories within a RBS are those of ‘schedule’, ‘resource’,
‘technical’ and ‘licensing’ [91]. Below this other risk category would decompose risks still further. This
approach ensures consistency of risk category assignment throughout the project and allows the tracking of
trends.
These categories can then be further sub-divided as required. Other options include categorising by function
or discipline. The goal of such an exercise is so ensure the consistency of categorisation and to allow trending
of risks to show where risk particularly resides within the development. The artefact that is ultimately
generated by the risk management process is the risk register. An example of the attributes that can be
assigned to each risk is shown in Appendix N. The risk matrix; against which risk likelihood, impact and priority](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-49-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 49 of 69
7.2. Analysis of framework against criteria within existing literature
In this section the complexity management framework is compared with the recommendations and criteria
for successful projects as discussed within the literature survey and from a wide variety sources. Any obvious
deficits are discussed and additional measures proposed that may benefit the future development of the
complexity management framework. Other recommendation should be satisfied by activities outside the
scope of this framework.
The 2012 ‘thisiswhatgoodlookslike’ Gartner survey [92] made five high-level observations relating to high-level
project strategies. The first concerned the size, complexity and duration of individual projects. Determination
of size and duration, and to a lesser extent the complexity, of the overall project is very often outside the
control of the technical development and indeed of the project itself. Complexity can be controlled once it is
recognised and at the very least prevented from growing unduly. The survey describes that a project must
‘stay on top of costs’ and describes that there should be measures for early identification of cost overruns.
This can be achieved through the adoption of leading performance measures that can identify delays to the
development that inevitably lead to additional costs due to acceleration of the schedule. The potential for
rework can also be highlighted which again can have significant associated costs. A realistic schedule can best
be achieved through better planning. The targeted use of DSMs identifies coupling behaviours and
dependencies to allow both optimisation of the schedule and improved representation of the schedule when
used as an input in the production of Gantt charts. Using the measures previously described it does at the very
least provide an early warning of discrepancies between early high-level schedule commitments and likely
actual schedule.
The fourth of the Gartner survey’s recommendation related to stakeholder and system requirements and the
funding that the full realisation of these dictates. Requirements management is not within the scope of this
dissertation. However, the assignment of requirements related performance measures and enhanced
planning alongside project management of the budget should assist in this. Additionally, effective and rigorous
requirements management would very likely be a development CSF which could then be monitored over time.
The last recommendation refers to frequent ‘project status and review meetings’ and ensuring alignment of
requirements with the business case. Better use of performance measures, as discussed within the
dissertation, should facilitate this. Ultimately the identification of a failing project, unlikely to satisfy the
business case, may lead to its cancellation.
The Royal Academy of Engineering’s 2004 ‘The Challenges of Complex IT Projects’ [93] also had five high-level
findings. ‘Lack of constraints’, relating to the definition of reasonable requirements, is outside the scope of
this framework. Again this can be partially mitigated through effective planning and performance monitoring.
Complexity assessment should highlight where complexity is highest which may indicate unrealistic
expectations. This would apply especially to the aspects of environmental constraints and technology,
especially where technological demands may be unduly constrained. Similarly, ‘Visualisation’ and ‘Flexibility’
will be again addressed by effective requirements management alongside the framework’s emphasis on
performance monitoring. The impact of changes can also be readily assessed in portions of the development
planned using DSM. High-impact change requests can then be identified and approvals made accordingly.
Lastly both ‘complexity’ and ‘uncertainty’ can be assessed early using the top-down complexity assessment
and later, using bottom-up as DSM planning is undertaken. CSFs and detailed planning can both reduce and
manage both these properties along with the inevitable changes to development requirements and scope.
Denker [94] cites the components of ‘Colossal Complexity’, ‘Invisibility’, ‘Over-Optimism’, ‘Extreme
Uncertainties from the Kickoff’ and ‘Rework’. Mirroring the factors mentioned previously, these can be
managed through the recognition of development complexity through complexity assessment, determination
of ways to manage it through identification of CSF and through better planning. The paper specifically
mentions the imbalance between complexity and imposed timescales. DSM should more completely identify
coupling and emergence that are features of complex schedules. Other themes are the lack of timely feedback](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-53-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 50 of 69
mechanisms and the identification of cause and effect. Both of these are addressed through the targeted
application of performance measures. Lastly the reuse of development patterns can be partially addressed
through the complexity assessment, assignment of CSFs and the review of their efficacy for future use. While
‘control mechanisms’ and ‘feedback capabilities’ are directly addressed through the use of performance
measures, the remainder Jiang’s 13 [95] should be addressed through general management arrangements.
Deficits in Gantt chart schedules, as previously discussed, can be addressed through the use of targeted
planning and specifically through the use of DSMs and appropriate performance measures. In it critical
activities that are outside of the critical path can be identified and coupling behaviours better represented.
The use of appropriate leading and lagging performance measures can measure the quality of processes and
its outputs. The high degree of ‘information requirements’ associated with planning are a function of Program-
size Complexity. Though it may be assessed it can only be managed, through appropriate resources and
technology, and not reduced. Limited modelling of the ‘behavioural aspects of management’ may be
represented using numerical type organisational-architecture DSM or process-organisation MDM. This does
not preclude the use of other modelling techniques, such as OBM, when deemed appropriate.
8. Case study application of framework
8.1. Purpose
In this section we evaluate the framework by using it to assess an existing complex project using publically
information. The case study uses publically available information on the development of Boeing’s Dreamliner
and will illustrate the use of the framework and identify areas of potential concern that require process
improvement or additional guidance. The WBS will necessarily be at a high-level/strategy level due to
constraints of information availability and the effort required to undertake analysis relating to detailed WBS
structures. As such it will be assumed that the assessment is being undertaken during the concept design
phase and used to analyse development up to the point of manufacture. The initial complexity analysis will be
followed by suggested CSFs and recommendations requiring areas of planning, performance measurement
and risk management. The assessment will be undertaken from the viewpoint of Boeing’s Dreamliner design
team and was chosen due to the well published development and operational issues [96][97][98], sheer
quantity and variety of literature, interesting characteristics and first-of-a-kind complexity. The findings will be
compared to the outcomes and assessed for efficacy and it will demonstrate the process, albeit at a low level
of detail. An additional case study, using the construction of a nuclear power plant, is contained in Appendix
Q. This project is distinctly different from the Boeing Dreamliner and identifies other aspects of the framework
requiring development but also demonstrates how the framework can be implemented.
8.2. Case study 1 – Boeing Dreamliner
8.2.1. Background
In the early 1990’s Boeing required a new commercial aircraft to replace the aging 767. Two options were
initially investigated; a sub-sonic cruiser with a speed of 0.98mach and similar fuel efficiency as the 767, and a
design similar to the 747 with increased capacity, designed to directly compete with the Airbus A380. Neither
was particularly well received by the commercial airline markets and when fuel costs escalated after 9/11 the
requirement for fuel efficiency became paramount. This led to the concept of the 787 Dreamliner in 2003 [96].
The requirements for the Dreamliner were as follows:
Lower fuel consumption;
Lower maintenance costs;
Longer range;
Improved passenger comfort.
This necessitated the use of technology not previously used extensively within commercial airliners. This
included advanced composite use, eventually resulting in more than 50% by weight [97], and the replacement
of pneumatic and hydraulic systems with electric architectures [98]. Additionally, the fuselage was to be](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-54-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 51 of 69
manufactured in ‘barrel sections’ and integrated later rather than being assembled by Boeing. These design
decisions were undertaken early in the development cycle for reasons of overall weight and reduced system
power requirements. Boeing’s initial estimate for the development was $6 billion over a four-year period,
representing a significant investment Manufacturing costs for the Dreamliner were also a significant factor in
order that it was to be both competitive and allow Boeing to make a return on its investment [99].
This prompted the adoption of a contracting strategy that was designed to make efficiencies in the
development and manufacture of the aircraft but spread the commercial risks [100]. It also allowed market
expansion by allowing the manufacturers in the countries that would buy the Dreamliner participate in the
project in ‘offset deals’ [100]. Contractual arrangements allowed Boeing to delay payment to partners until
aircraft were delivered [97], including risk-sharing partnerships, with payments depending on final outcome.
Manufacturers were only paid when the plane was certificated and delivered, thus Boeing avoided bearing
many of the up-front non-recurring research and development costs, encouraging efficiency in the supply
chain and could delay investment until later in the project lifecycle. In addition to higher returns on successful
deployment of aircraft the manufacturers also were allowed to retain intellectual property rights of their
contribution to the aircraft [100]. ‘Partner councils’ were established to aid the creation of an ‘integrated,
integrated, modularized supply-chain’ and mirror the practices used in military and business aircraft sectors
[97]. The use of globally based suppliers ensured that the necessary expertise could be sourced and also
allowed the development of many systems in parallel.
8.2.2. Work Breakdown Structure
To meet the considerable technological challenges and cost efficiencies it was decided that design
development and manufacturing, including research and development, of most of the aircraft would be
outsourced across the globe, to Tier 1 suppliers. The contracts were designed to allow Final assembly would
be undertaken by Boeing in the US. Ultimately this resulted in 70% of parts being manufactured by 50
companies in 28 countries [98]. The Tier 1 suppliers would in turn contract out manufacture and services to
Tier 2 suppliers as appropriate. A WBS using system engineering and project activities has been produced
decomposing the aircraft into systems and structures using the not-dissimilar Hyperion project [101] as a basis.
Hyperon was chosen because it is within the aviation domain (unmanned aerial vehicle in this case) and has
been developed internationally. This WBS has been adapted to include general commercial aircraft systems
[102] the Dreamliner’s more detailed ‘Global WBS’ [103]. This gives an emphasis to the technology and
contracting components of the Dreamliner project though common design phases have been included
including a research and development phase that was undertaken by both Boeing and its Tier 1 partners.
It is assumed that research and development would be undertaken after the concept had been agreed,
allowing the development of a business case and the commencement of marketing and sales, but before
scheme design. Based on the Dreamliner’s history the concept would have been available in late 2003/early
2004 before the first order with Nippon Airlines was agreed. An alternative would have been to decompose
the project with a higher emphasis on functions and processes using a combination of system engineering and
project management methodology. It should be noted that the contents of the WBS given below is not
exclusive, though it is complete enough to allow early analysis. As can be seen the level of detail increases
through the project’s phases as the aircraft’s systems are defined.
Design
Concept;
o Aerodynamics;
o Mass properties;
o Composites;
o Structures;
o Fuselage;
o Wings;
o Doors;
o Stabilisers;](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-55-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 56 of 69
System Integration - In a similar fashion to contractual management, integration will increase in all areas.
8.2.4. Critical Success Factors
The CSFs will evolve throughout the development though thought should be given to putting them in place in
advance of when they are particularly required and the phasing required. CSFs will be ranked and generally
taken from ‘high to very high’ rated CSFs from the full survey dataset Appendix F. CSFs within the top three
identified from each Complexity Theme will be shown in bold with the chosen CSFs in order of priority. The
full dataset was chosen in preference to that from the aviation industry dataset due to its low sample size.
Where the number of CSFs are overwhelming in number it is expected that these can be further ranked in
importance. CSFs assessed as having the greatest impact will be chosen in preference to the others.
Furthermore, it would also be expected that performance measures should be chosen to demonstrate the
implementation and impact of CSFs. Examples of how these CSFs may be further tailored towards the
Dreamliner project or otherwise supported are provided in italics with top three ranked CSFs identified in bold
Concept design
At this point in the development lifecycle the emphasis is on managing uncertainty. At this stage ambiguity
will also be high to the technology and there may be constraints of funding prior to a commitment to long-
term investment. Particular Environmental constraint related CSFs are as follows:
I. Clear realistic project objectives – planning undertaken with minimum participation of a defined
percent within the supply chain. Plan complexity measured and managed through use of DSMs;
II. Composition of project team in terms of experience and capability – minimum contractual pre-
qualification criteria for Tier 2 suppliers, Process-Organisation MDMs implemented.
The development process will need to tailored to suit the particular challenges likely to be faced and also
manage the uncertainty and ambiguity that will be present. There may be the opportunity to use methods
from other domains or industries such as the design of business aircraft in with respect to the use of composite
materials [104] or manufacture of electric cars with respect lithium ion batteries. Development CSFs may be:
I. Critical activities are identified – Process DSM applied in areas of high complexity;
II. Performance measures tailored to monitor areas of criticality and uncertainty – DSM related
Performance Measures are implemented where required.
The organisation would also need to cope with uncertainty and ambiguity relating to the new technology and
could be assigned the following CSFs.
i. Transparent definition of responsibilities - Process-Organisation MDMs implemented;
ii. Composition of development team in terms of experience and capability – as above.
Contractual management would be in its infancy at this point and would relate to the production of contractual
documentation and selection of potential Tier 1 partners. In terms of uncertainty and ambiguity the following
CSFs are important.
i. Clearly understood contractual interfaces - Process-Organisation MDMs implemented;
ii. Project breakdown into logical packages – complexity considered before allocation.
However, this is one of the most critical aspects of the development and decisions made at this point and
written into the contracts will have far reaching consequences. For this reason, CSFs need to be considered
for the entire project lifecycle at this point in time.
Neither the management of stakeholders or external interfaces are the project’s most challenging aspects and
no CSFs should be emphasised.
Regulatory interfaces are subject to some uncertainty due to the new technology and it treatment by the
various certification authorities across the globe. Early and thorough engagement would be vitally important.
i. Clearly identified and understood interfaces – MDM developed to model interfaces;
ii. Clear lines of communication with regulators.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-60-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 58 of 69
Changes in requirements can have a serious effect on the project lifecycle. Specific CSFs surround the effective
and timely communication and management of stakeholders.
i. Client/user acceptance– Requirements measurement is adopted;
ii. Decisions are agreed and documented;
iii. Early identification and management of conflicting interests – Requirements measurement is
adopted;
iv. Active management of client/user integration – MDMs adopted for critical systems.
External interfaces, while important, are not viewed as being as critical as other areas.
Regulatory interfaces, technology and Integration have the potential to have an increased impact though a
lower likelihood of change as the development proceeds and technology and design matures. As such the CSF
will remain the same as there is little that can be done to reduce the impact of emergent change other than
identify it early.
8.2.5. Planning technique
The sheer size, scale and criticality of the project would lend itself to the use a combination of Design Structure
Matrices. These could be targeted against particular areas of development and interfaces based on risks borne
from the constraints of schedule and relative criticality, both a technically and in relation to the overall project.
From the Criticality Assessment the three areas that deserve particular attention are management of Tier 1
suppliers, technology and integration. All have particular challenges arising from the project and development
strategies:
Organisation architecture DSM to consider how Boeing and the Tier 1 partners taking into account all
interdependencies and constraints arising from their diverse geographical locations;
Process architecture DSM to focus on particular groups of activities from the WBS. This might
particularly look at integration or assembly type activities and dependencies between Tier 1 suppliers;
MDM to look at particular parts of the development and be used to plan the interaction between
organisations and show the impact on the wider schedule.
Full integration of traditional scheduling and DSMs would allow a seamless blend of the two techniques to be
used with areas of contractual interfaces, technology centric development and integration activities being
prime candidates for binary MSMs and numerical DSMs. Further Complexity Assessment of the WBS could be
taken at the next lowest level or levels to identify areas of particular complexity. From the project’s
background information such candidates for enhanced planning include the following:
Electrical systems, including the lithium-ion batteries [105];
Composite material development and especially construction of barrel sections of fuselage;
Design integration activities;
Aircraft assembly including Tier 1 partner dependencies;
Certification related activities, such as quality assurance, across the entire development.
Enhanced planning would be chosen in other areas, depending on the WBS and on the most recent Complexity
Assessment findings. The scale and length of the development may justify specific tool support to be
developed.
8.2.6. Performance measurement
To meet the challenges of contractual management, technology and integration the following measures could
be combined into a Development Scorecard across the entire project:
Conformity with the plan consisting of:
o Earned Value Management
Cost Performance Index (lagging);
Schedule Performance Index (lagging);
System maturity consisting of:](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-62-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 59 of 69
o Technology Readiness level (lagging);
o Integration Readiness level (lagging);
Process Health to measure areas such as leadership and collaboration.
The outsourced nature of the project makes collecting information challenging and quality assurance of Tier 1
supplied information would be required. Other measures could be applied, with discretion, in areas of
particular concern and these include:
Plan complexity consisting of:
o Proportion of sequential, parallel, coupled and conditional activities within DSM (leading);
Requirements consisting of:
o Realisation – relating to MOE, MOS, MOP and TPM (lagging);
o Execution – relating to selected requirements metadata (leading).
Such data would need to be trended over time rather than compared with data over the entire project and
would be used where there are particular technical risks or evidence of poor project performance. This would
provide a mix of leading and lagging measures the purposes of monitoring, control and reporting.
8.2.7. Risk management
The identification of development risks will not be undertaken in detail due to lack of data. It is not difficult to
appreciate that there will be significant risks in the areas of contractual management, technology and
integration. Detailed descriptions, risk responses and related information would naturally fall out of more
detailed analysis and planning activities.
The creation of a hierarchical Risk Breakdown Structure would be an important early activity in the formation
of the risk management strategy. One suggestion for the high-level nodes (level 1) of the RBS could be those
of ‘schedule’, ‘resource’,’ technical’ and ‘certification’. Level 2 nodes would include contracting and Boeing
(below resource) and technology and integration (below technical). Certification could be divided amongst
particular systems within the aircraft. With the high reliance on Tier 1 Partners it might be appropriate to also
include this as a level 1 RBS node or else place it under resource risks alongside ‘Boeing organisation’. Similarly,
the importance of technology and integration can be elevated to that of a level 1 node or else included under
one or more of the other nodes.
8.2.8. Comparing framework findings with project outcomes
Boeing’s Dreamliner has received a large amount of publicity due the prevalence of problems pre and post
launch. A detailed time line of events can be found in Appendix P.
Delays almost doubled the ‘launch-to-delivery’ time from 49 to 89 months [97] due to a variety of problems
involving Tier 1 partners and the technology. Most notable were long standing issues with the supply of
fasteners and a number of incidences involving the battery technology causing safety concerns [106].
Recognised risks included the management of the supply chain across different countries, such extensive use
of outsourcing and assembly issues made worse by the extensive use of composite technology [97]. Costs
spiralled to approximately $40billion, which was twice the original estimates and included costs due to defects,
additional R&D, supplier support and buyout, customer contract penalties and cancelled orders [100]. This
included bailing out Tier 1 partners such as Alenia and Vought who designed and manufactured the horizontal
stabiliser and both central and rear fuselage.
There has been much speculation, research and analysis as to the root causes for the issues described. Many
relate to supply chain competence and Boeing’s own communication and procurement. Risk sharing neglected
to hold individual suppliers to account for their actions and so encouraged them to benefit from savings on
direct costs while the overall schedule was delayed at the same time Boeing placed a higher emphasis on costs
over schedule. As such optimisation of aspects of development did not equate to overall optimisation, rather
quite the opposite [100] Many of the additional costs thus related to Boeing requiring more effort than
anticipated managing the supply chain. Post-launch problems were also significant in increasing costs but also](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-63-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 60 of 69
in damaging Boeings reputation and future Dreamliner sales. Significant safety events included a number of
electrical fires relating to the batteries required for the aircraft’s electrical system. This has been attributed to
quality control rather than strictly technology based issue [107] and as such reflects badly upon both Boeing’s
control of its partners and the certification process [108] including the delegation for the provision of test data
to the project’s suppliers [109].
Other operational safety issues included engine shutdowns, fuel leaks, loss of transponder, electrical faults,
hydraulic failures, a cracked windscreen, cracks in wings discovered at the factory and a variety of on-board
management systems [110]. All in all, there have been close to 100 reported incidents as of August 2016 [111].
The Complexity Assessment described particular areas of concern surrounding Contractual Management,
Technology and Integration. It is fair to say that most of the issues encountered during the development of
the Boeing Dreamliner fell into one of these categories.
An additional case study describing the development and construction of AREVA’s Evolutionary Power Reactor
in Olkiluoto, Finland (OL3) has been undertaken within Appendix Q. OL3, though very different to the Boeing
Dreamliner, exhibits characteristics that are equally as complex. While it has similarities in the depth of
technical complexity OL3 differs in Areva’s choice of procurement and contracting strategy. Whereas Boeing
predominately contracted out design to Tier 1 partners, Areva undertook virtually all design in-house, creating
its own unique issues. OL3 is again well documented and has a reputation well deserving of a case study. Both
Boeing’s Dreamliner and Areva’s OL3 were briefly discussed within Section 1.5 of the Literature Survey [4].
9. Results and Evaluation
The dissertation has addressed the following objectives in an effort to develop a complexity management
framework:
Proposed a complexity assessment by decomposing a technical development by:
o Discussing the rationale for complexity management alongside current project management
methodologies and provide background to the complexity assessment proposal;
o Decomposing development by the WBS;
o Creating themes for each WBS element – (project) environmental constraints, development
process, internal organisation, contractual management, stakeholders, regulatory interfaces,
external (system) interfaces, technology and internal (system) interfaces;
o Creating complexity criteria for each theme – uncertainty, ambiguity, emergence, non-linearity
and programme-line complexity;
o Rating of criteria between very low and very high;
Describing how complexity may be profiled per criteria over the development lifecycle, how these
criteria interacts and how change may affect this;
Developing the concept of CSFs for application within the Complexity Framework;
o To indicate where extra planning effort should be applied;
o To direct where monitoring should be applied to a higher level of decomposition;
o To align with areas of high complexity deserving interventions and management effort;
o Testing and ranking the selection of CSFs through questionnaire using project and engineering
professionals;
Proposing the use of Design Structure Matrix to supplement and be used as an input into the
production of Gantt chart schedules;
o Describing the technique as it is already used for both process and organisational-architecture
DSM and Multi Domain Matrix;
o Proposing the rationale for the use of binary, numerical and MDM;
o Aligning DSM with complexity criteria and use it to derive planning indicators and
performance measures;](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-64-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 61 of 69
Propose the effective use of different types of performance measures to aid monitoring of
development activities;
o Discussing the varying types and purposes of performance measures to provide a way of
accurately representing status to allow control and mitigating actions to be implemented;
o Listing a number that have been previously proposed as suitable for use in this application;
o Testing the proposed Performance Measures by questionnaire survey to indicate those that
are considered most influential and to provide confidence in the measures selected;
o Proposing a development scorecard to consist of a number of the performance measures to
provide a balanced way of measuring status across the breadth and depth of a technical
development;
Describe how the entire process may inform risk identification as a part of the overall development
risk management;
Gather all these elements into the complete complexity management framework;
Test the complexity management framework against two case studies.
The dissertation has satisfied the objectives that were assigned within the introduction with varying degrees
of success. The individual components of the Complexity Management Framework will be discussed for their
strengths and weaknesses. Finally, areas of further research will be identified where they are appropriate.
Complexity assessment
The rationale for the assessment was derived from a combination of literature and the author’s personal
experience. The use of the WBS as a focus provides a convenient way of decomposing the activities while the
complexity themes allow the assessor to look at the WBS elements from a variety of important viewpoints.
The complexity criteria decompose the actual complexity further and introduce uncertainty and ambiguity as
discreet elements, which are very often barriers to good decision making. The criteria broadly follow the
principles the business assessment methodology called VUCA [37], using classical definitions of complexity
[37] and adopting with similarities to 5DPM [31]. The regime for scoring individual criteria is as precise as such
a heuristic can possibly be though of course it is still open to a high degree of subjectivity.
The assessment provides little in isolation but does prompt qualitative analysis of the development and the
identification of areas of concern for assignment of CSFs. Together they show potential areas of complexity
related risk and a description of the issues within the technical development. Difficulties arise in assigning
criteria ratings that relative to the entire development lifecycle. For example, providing a ‘very high’ rating
early in the development allows no scope for increasing the rating later on. This can be at least partially cured
through the development of assessment guidance. Another area requiring guidance is that pertaining to
System Development Themes, of which there can be seen to be some overlap. Practical application would be
preferred method of refining analysis and improving consistency of application. Focussed further research
could be used to aid lessons learned type exercises.
The nature of Complexity Criteria was discussed and how these evolve during a project. Uncertainty/Ambiguity
closely equates to likelihood of change while Emergence/Non-linearity related more closely to impact. In this
way it can be seen that Uncertainty/Ambiguity is most influential in early phases of development while
Emergence/Non-linearity becomes increasingly predominant as the development matures. Complexity
Profiles could be developed to allow baselines against which similar technical development could be assessed.
The use of case studies high-lighted a number of potential shortfalls in the Complexity Assessment. The WBS
is an important consideration during the Complexity Assessment as inconsistent or a poorly conceived
structure will prove problematic in the application of Complexity Assessment. Issues arise where the WBS uses
inconsistent methodology and omits development phases at a high level as can be seen in the OL3 case study
contained in Appendix Q. This makes Complexity Assessment more of a challenge at each point that it is done.
In the example of OL3 the WBS will have been created early in the project lifecycle and presumably by a variety
of business entities in parallel. In its favour it would have stayed largely unchanged throughout as it would](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-65-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 62 of 69
have been largely uninfluenced by contracting decisions, relying instead on established business functions for
its structure. In contrast the Boeing Dreamliner WBS would have been dependant on the ‘Global WBS’ that
was created as a consequence of how the aircraft was contracted out for development and manufacture. Later
interventions by Boeing to support the supply chain and provide additional supervision and management
would have also affected the WBS. Ideally early input into the WBS during initial complexity assessment would
alleviate these issues.
The case studies allowed an assessment of the technique but with the inevitable bias relating to knowledge of
actual real-life outcomes and limitations of high-level application. However, they were able to demonstrate
its use against a high-level WBS. Development of assessment against a detailed WBS would benefit from
practical application to identify further areas for improvement.
Critical Success Factors
This is a technique already established in industry [5][7][45] though without any defined methods to define
and implement. This dissertation was able to link it into a process designed to better guide its application and
also relate the realisation of CSFs to Performance Measures. The literature survey was effective in providing a
comprehensive list of CSFs. The validity of these CSFs was confirmed following the questionnaire survey with
only a single CSF from the 141 being identified as having low influence. The ranking of CSFs allows them to be
both prioritised when chosen for use and when applied in practice. Further development of CSFs will again
rely on project data and particularly the review of past development performance. The questionnaire has a
reasonable sample size that provides a good starting point for their application.
Another area of development that became obvious while undertaking the case studies is the need to
categorise the CSFs to match the Complexity Criteria to inform their selection. This would move the selection
of suitable CSFs from being highly subjective to that of a semi-formal method. Each CSF would be considered
against its potential to reduce or allow better management of Ambiguity, Uncertainty, Non-Linearity,
Emergence and Program-size Complexity. Initially this may take the form of having have simple categorisations
of positive, neutral or negative influence. There would then be the opportunity to develop this further using
lessons learned from the use of CSFs. Foremost the application of CSFs would benefit from a commentary
being given to each, proposing how it could be best implemented and under which circumstances. Ultimately
the CSFs could be given a more complicated rating against Complexity Criteria. Candidates for this include a
scale from high to low (with none and negative to denote where the CSF has no benefit) or a numerical scale
with positive and negative integers.
Identification of CSFs should not be based purely on the analysis of the current stage of development as
omission of CSFs for particular aspects can have far reaching consequences. Examples include Contractual
Management where the cognisance of CSFs at the time of contract documentation writing can influence
strategy and content. Another example may be the consideration of verification and validation strategy early
in the design where the management of uncertainty and ambiguity early in the design may overshadow that
of managing the same later on. The application of these generic CSFs to make them effective within particular
projects was not developed beyond some examples within the case studies.
The application of CSFs could be applied in isolation of the other sub-processes described within this
dissertation. As such there is much value in the findings of the questionnaire survey and their application even
if complexity management is not adopted as a whole. There is however a significant benefit of the selection
of CSFs being informed by a process such as Complexity Assessment.
Planning using Design Structure Matrix
While there is substantial literature on DSM and related techniques its use has yet to become commonplace.
Exceptions include a few specialised application examples, such as Boeing’s Unmanned combat aerial vehicle
(UCAV), Intel’s microprocessor development and BMW’s hybrid vehicle architecture concepts [67]. Evidently
its use does not yet approach the level that will enable tool support and training to be made widely available.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-66-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 63 of 69
DSMs can provide substantial value within planning development activities with complex relationships and it
should be used to supplement, rather than a replace, currently used techniques. It is supposed that a barrier
to its use is that of cost as an additional overhead, requiring additional resource and specialised skills to
implement. Therefore, the benefits of DSM need to be demonstrated for industry to justify investment to
develop and implement it. The section on DSM clarified the similarities between planning relationship
descriptions between DSM and traditional techniques. It showed how the creation of DSMs could be used as
an input into Gantt chart development. Finally, it proposed a number of Performance Measures relating to
plan complexity. The need for such Performance Measures was confirmed by the results of the questionnaire
survey which indicated that this was the third most influential, therefore desirable Performance Measure for
use.
The activity relationships sequential and parallel are commonly used within planning, the DSM process activity
types of coupled and conditional are not. There is an opportunity to better define the interfaces between Gantt
charts and both DSMs and MDMs. This would potentially lead to better tool support, including an interface
with the most popular planning software such as Oracle’s Primavera [69]. Such tool support for transposition
of activity relationships information from numerical DSMs into a Gantt chart format would improve the
popularity of DSM as a planning tool. Tool development should not be confined purely to that of Process DSMs.
Of equal benefit is the development of techniques and tooling to allow Organisational DSMs, and ultimately
Process-Organisational MDMs, to support resource loaded Gantt charts. This would benefit the application of
Earned Value management by improving the quality of the plan against which activities are to be monitored
and controlled against.
As well as providing information and data to support the implementation of EVM these techniques would
support the implementation of plan complexity Performance Measures. As the third ranked most influential
Performance Measure, behind both Requirements related measures, it should be considered as an important
method of monitoring and controlling activities. Indeed, its value is that it is very much a leading measure of
project performance. The benefit of using DSM and appropriate tooling is that the collection of these plan
complexity performance measures can be largely automated. This allows these measures to influence the
planning process itself in reducing complexity. It should influence the ultimate forming of the Gantt chart. It is
also a leading measure that can be used to inform management of areas of particular concern against which
resources and expertise should be expended.
In common with CSFs DSM can be applied outside complexity management and without the proceeding
Complexity Assessment and CSF identification. These sub-processes do facilitate the targeted and semi-formal
method of applying DSM against areas of particular complexity. Again there is additional value in the
application of DSM within this framework in that areas of low concern can be disregarded and planned using
more traditional techniques.
Performance measurement
From the questionnaire it is obvious that there is a strong need for techniques to supplement EVM. EVM was
used as a benchmark and rated relatively poorly and even amongst project personnel. This suggests there is
considerable room for improvement in this area. Requirements management figured most strongly overall
reinforcing the author’s view that the monitoring and control of requirements deserves greater emphasis
within project management methodologies. The measurement of plan complexity, which could be satisfied
using a number of DSM related measures, also figured strongly as third most influential. Requirements
management relates closely to that of scope management since all scope should be derived from development
and project requirements. This demonstrates there should be a lot stronger emphasis on the area of scope
within the traditional project management model of the ‘Iron Triangle’ [32]. The measurement of schedule
and cost, other than through EVM, received barely any attention within the questionnaire comments. In
contrast, scope, both directly through plan complexity and indirectly through requirements was mentioned in
the top three measures.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-67-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 65 of 69
documented a process to aid actual identification, management and the review of their impact is not. Analysis
of CSF efficacy would be highly useful and this sub-process provides a basis for such an activity that is not
discussed elsewhere in literature. It is also felt more work is required in the Complexity Themes of ‘technology
development management’ and ‘system integration management’. An increased sample size, including more
engineering personnel, would improve the findings in these areas
There is a deficit in the number of CSFs relating to the Criticality Themes under the External factors and System
Development groupings. Neither research of literature or the questionnaire survey identified any additional
CSFs so again it is felt only practical application of the framework would yield progress in this area. Guidance
should be developed for the application of CSFs against particular Complexity Criteria. This could take the form
of a simple categorisation of positive, neutral or negative influence to begin with. Additional instruction could
be developed for the use of CSFs in the form of a commentary for each one. Guidance should also be
developed in applying CSFs in specific technical developments. This should include appropriate wording and
content but should also relate to the use of Performance Measures.
Further work is required to develop suitable Performance Measures that will support the implementation of
CSFs with maximum effort applied to the most influential CSFs. Many of these CSFs are intangible and
subjective, though nonetheless important, making this task difficult. For some CSFs objective measurement
will be impossible with examples including ‘good leadership’ and ‘degree of collaboration’. However, the
identification of which CSFs can and cannot be measured would be useful in itself. Methods to assess those
that cannot be measured could be undertaken through appropriate review at Stage Gates or similar
development decision points. Examples of those that are readily measured include CSFs such as ‘critical
activities are identified’ and ‘clearly understood contractual interfaces’. In these examples the use of DSM
methodologies would be of assistance.
The application of DSMs has been developed over several decades though commercially available tooling is
weak. Development of tooling to allow DSMs to be used and be used as both an input into the development
of Gantt charts and Performance Measures would be useful. Other areas of research include the development
of numerical DSM methodologies, developed and improved over several development lifecycles, and its use
in combination with other modelling techniques. OBM [41] has been referenced earlier in the dissertation but
there are many other modelling techniques that may be more appropriate to integrate with DSM and
traditional planning techniques.
The practical application of complexity related performance measures against DSM would allow the proposed
technique to be developed further. There is considerable benefit in such a leading measure and, considering
its ranking in the questionnaire survey, its use may well be an attractive proposition within technical
development. It is suggested that trialling of this, in conjunction with the application of DSM, is the next step
in its development.
The development of Performance Measures to support the use of CSFs deserves further research. The
measurement of CSF effectiveness would encourage their use and be useful as an input into lessons learned
exercises. This can be undertaken alongside general development of Performance Measures. Another aim is
that of automation of the collection of data for the use in Performance Measures using existing a new
computer tooling. This would reduce the manual overhead and encourage their greater use. The survey
findings suggested significant benefits in further development of requirements and scope based Performance
Measures. Database and tooling support would allow some degree of automation. Further research should be
undertaken in the development of additional measures and those measures that did not receive particularly
good rankings in the survey. It is felt the application of these measures may be appropriate in some technical
developments.
Overall there are opportunities to apply several of the individual sub-processes in isolation to better develop
them, despite the benefits of combining them. This may be more viable for their development as the overhead](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-69-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 66 of 69
in terms of resources will be considerable and especially if they are applied in tandem with conventional and
proven project monitoring and control techniques.
Project management has failed to learn the lessons over successive high-profile project failures. Many of these
failings were discussed within the literature survey [4] and so it may be surmised the discipline has been
resistant to change for some considerable time. Without sufficient data, preferably corroborated, the benefits
of Complexity Management via this framework will be intangible as best. This combined with project
management’s inherent conservatism may be the greatest barrier to the acceptance to the tools and
techniques described within this paper. An iterative and considered approach to the management of
complexity will be required to gain its support.
What is clear is that the current project management and systems engineering methodologies will need to
improve their efficacy in the face of increasing levels of complexity. The question that has to be asked before
commencing any complex project is what sets this undertaking apart from the high percentage of unsuccessful
projects? The recognition of complexity and tailoring of the technical development process to suit a project’s
unique complexity traits is one way this question may be answered.
References
[1] M Lee (2003). HRD in a Complex World, (Ed), London: Routledge.
[2] C E Shannon and W Weaver (1998). The Mathematical Theory of Communication, University of Illinois Press, Champaign.
[3] Australian Public Service Commission (2012). Tackling wicked problems: A public policy perspective, Last updated: 31 May 2012,
Available at: http://www.apsc.gov.au/publications-and-media/archive/publications-archive/tackling-wicked-problems, Accessed on:
25th October, 2016.
[4] N Brook (2015). Integration of technical development within complex project environments Literature Survey, MSc, Safety Critical
Systems Engineering, University of York.
[5] Mindtools (2016). CSFs; Identifying the Things That Really Matter for Success, Available at:
https://www.mindtools.com/pages/article/newLDR_80.htm, Accessed on: 28th October, 2016.
[6] L S Pheng and Q T Chuan [2006]. Environmental factors and work performance of project managers in the construction industry,
International Journal of Project Management, Volume 24, Issue 1, January 2006, Pages 24–37, Available at:
http://www.sciencedirect.com/science/article/pii/S0263786305000633, Accessed on: 25th October, 2016.
[7] BMG Research (2014). Factors in project success; Research Report, Prepared for: The Association for Project Management (APM),
November 2014, Available at: https://www.apm.org.uk/sites/default/files/APM%20Success%20report_NOV%2014.pdf, Accessed on:
25th October, 2016.
[8] OGC (2012). PRINCE2; Directing Successful Projects, 5th Ed, TSO, Norwich.
[9] aceproject (2014). Project management and firefighting, Available at: http://www.aceproject.com/blog/2009/04/29/project-
management-and-firefighting/, Accessed on: 25th October, 2016.
[10] M Ramsden (2013). Ten rules for smart bowtie analysis, 31 October 2013, ERM, Available at: http://www.erm.com/en/news-
events/platform/ten-rules-for-smart-bowtie-analysis/, Access on: 8th November 2016.
[11] N Brook (2015). Presentation: Failure curves and bowtie analysis (adapted from ERM (no date). Introduction to Bow-tie Diagrams,
Available at: http://events.r20.constantcontact.com/register/event?llr=6quxcycab&oeidk=a07e5zzlwto9ff6b679), Northumbria Water
[12] B Juttehttps (no date). 10 Golden Rules of Project Risk Management, Project Smart, Available at: http://www.projectsmart.co.uk/10-
golden-rules-of-project-risk-management.php, Accessed on: 25th October, 2016.
[13] K Jackson (no date). Plan-Do-Check-Act (PDCA), Mindtools, Available at: http://www.mindtools.com/pages/article/newPPM_89.htm,
Accessed on: 25th October, 2016.
[14] N Settle-Murphy (no date). 10 Top Tips for Leading Great Lessons Learned Reviews in a Virtual World, Guided Insights, Available at:
http://www.guidedinsights.com/10-top-tips-for-leading-great-lessons-learned-reviews-in-a-virtual-world/, Accessed on: 25th October,
2016.
[15] J Shane et al (2015). Guide to Project Management Strategies for Complex Projects, SHRP 2 Report S2-R10-RW-2, Institute for
Transportation, Iowa State University, Available at: http://www.trb.org/Main/Blurbs/167482.aspx, Accessed on: 26th October, 2016.
[16] Helmsman Institute (2012). Why Complexity Matters, Available at: http://www.apmg-
international.com/nmsruntime/saveasdialog.aspx?lID=5479&sID=6815, Accessed on: 26th October, 2016.
[17] N Goldenfeld and L P Kadanoff (1999). Simple Lessons from Complexity, Science 02 Apr 1999: Vol. 284, Issue 5411, pp. 87-89, Available
at: http://science.sciencemag.org/content/284/5411/87, Accessed on: 25th October, 2016.
[18] D W Oliver et al (1997). Engineering Complex Systems with Models and Objects, McGraw-Hill Inc.,US, Available at:
https://oldsite.incose.org/ProductsPubs/DOC/EngComplexSys.pdf, Accessed on: 26th October, 2016.
[19] D Rind (1999). Viewpoint: Complexity and climate. Science, 284, 105-107, Available at:
http://pubs.giss.nasa.gov/docs/1999/1999_Rind_ri04100j.pdf, Accessed on: 25th October, 2016.
[20] S A Kauffman (1993). The Origins of Order: Self-organization and Selection in Evolution, Oxford University Press.
[21] J H Holland (1996). Hidden Order: How Adaptation Builds Complexity, Addison-Wesley, Reading, Massachusetts.
[22] Y Bar-Yam (1997). Dynamics of Complex Systems, Addison-Wesley, Reading, Massachusetts, Available at:
https://fernandonogueiracosta.files.wordpress.com/2015/08/yaneer-bar-yam-dynamics-of-complex-systems.pdf, Accessed on: 25th
October, 2016.
[23] G Weng et al. (1999). Complexity in biological signaling systems, Science. 1999 Apr 2;284(5411):92-6, Available at:
https://www.ncbi.nlm.nih.gov/pubmed/10102825, Accessed on: 26th October, 2016.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-70-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 67 of 69
[24] M Brugnach (2008). Complexity and Uncertainty: Rethinking The Modelling Activity, Published in Environmental Modelling, Software
and Decision Support: State of the Art and New Perspectives, Amsterdam et al.: Elsevier, 2008.Available at:
http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1071&context=usepapapers, Accessed on: 26th October, 2016.
[25] M T Pich (2000). On Uncertainty, Ambiguity, and Complexity in Project Management, Published Online: August 1, 2002, Management
Science, Available at: http://dx.doi.org/10.1287/mnsc.48.8.1008.163, Accessed on: 26th October, 2016.
[26] The British Computer Society (2006). Case Study of Successful Complex IT Projects, Lancaster University, August 2006, Available at:
http://www.bcs.org/upload/pdf/casestudy2.pdf, Accessed on: 26th October, 2016.
[27] SEBoK (2016). Guide to the Systems Engineering Body of Knowledge, (SEBoK), Available at:
http://sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_Body_of_Knowledge_(SEBoK), Accessed on: 26th October, 2016.
[28] INCOSE (2015). Systems Engineering Handbook; A Guide for System Life Cycle Process and Activities, 4th Ed, INCOSE‐TP‐2003‐002‐04,
January 2015, John Wiley & Sons, Hoboken, New Jersey.
[29] Project Management Institute. (2004). A guide to the project management body of knowledge (PMBOK guide). Newtown Square, Pa:
Project Management Institute.
[30] D A Buchanan and A Huczynski (1997). Organisational Behaviour, 3rd Edition, Hemel Hampstead, Prentice Hall.
[31] D D Gransberg and H D Jeong (2015). Managing Mega-Project Complexity in Five Dimensions, Conference: The 6th International
Conference on Construction Engineering and Project Management (ICCEPM 2015), At Busan, Korea, October 2015, Available at:
https://www.researchgate.net/publication/284533244_Managing_Mega-Project_Complexity_in_Five_Dimensions, Accessed on: 26th
October, 2016.
[32] Mindtools (2016). The Iron Triangle of Project Management, Balancing Your Budget, Scope, and Schedule, Available at:
https://www.mindtools.com/pages/article/newPPM_54.htm, Accessed on: 26th October, 2016.
[33] PCubed (2016). Nuclear Industry Sector Challenges, Available at: http://www.pcubed.com/bulletins/nuclear-industry-sector-
challenges, Accessed on: 26th October, 2016.
[34] ICCPM (no date). About ICCPM, Available at: https://iccpm.com/content/about-iccpm, Accessed on: 26th October, 2016.
[35] A R McGowana et al (2013). A Socio-Technical Perspective on Interdisciplinary Interactions during the Development of Complex
Engineered Systems, 2013 Conference on Systems Engineering Research, Volume 16, 2013, Pages 1142–1151, Available at:
http://www.sciencedirect.com/science/article/pii/S187705091300121X, Accessed on: 26th October, 2016.
[36] N Bennett and G J Lemoine. (2014). What VUCA Really Means for You, Harvard Business Review, January–February 2014 Issue,
Available at: https://hbr.org/2014/01/what-vuca-really-means-for-you, Accessed on: 27th October, 2016.
[37] G Satell (2013). Management Has to Change In An Increasingly Complex World, June 9th 2013, Available at:
http://www.businessinsider.com/how-to-manage-complexity-2013-6?IR=T, Accessed on: 27th October, 2016.
[38] L Fortnow (no date). Kolmogorov Complexity, Available at: http://people.cs.uchicago.edu/~fortnow/papers/kaikoura.pdf, Accessed on:
27th October, 2016.
[39] P Holman (2010). Engaging Emergence: Turning Upheaval into Opportunity, Chapter 1. What Is Emergence? Berrett-Koehler
Publishers, San Francisco. Available at: http://peggyholman.com/papers/engaging-emergence/engaging-emergence-table-of-
contents/part-i-the-nature-of-emergence/chapter-1-what-is-emergence/, Accessed on: 27th October, 2016.
[40] P Reason (1991). Human Error, Cambridge University Press, Cambridge.
[41] J S Warboys and J S Keane (1993). OBM: A Specification Method for Modelling Organisational Process, Published in the Proceedings of
the Workshop on Constraint Processing at CSAM’93, St. Petersburg, 1993, Available at:
[42] ITIL (2016). Back to basics (People, Process and Technology), Available at:
http://www.itilnews.com/index.php?pagename=ITIL__Back_to_basics_People_Process_and_Technology, Accessed on: 27th October,
2016.
[43] M Frank et al (2011). The Relationship among Systems Engineers’ Capacity for Engineering Systems Thinking, Project Types, and Project
Success, Project Management Journal, Volume 42, Issue 5, pages 31–41, September 2011, Available at:
http://onlinelibrary.wiley.com/doi/10.1002/pmj.20252/abstract,, Accessed on: 26th October, 2016.
[44] Techopedia (2016). Scope Creep; Definition - What does Scope Creep mean? Available at:
https://www.techopedia.com/definition/24779/scope-creep, Accessed on: 26th October, 2016.
[45] Association for Project Management. (2012). APM body of knowledge, 6th ed., High Wycombe.
[46] A Bassiouny (2008). UAV Stability Augmentation System, Published on Nov 16, 2008, Available at:
http://www.slideshare.net/ahmad1957/uav-stability-augmentation-system-usas-presentation, Accessed on: 27th October, 2016.
[47] S Denning (2013). What Went Wrong at Boeing? Jan 21, 2013 Forbes, Available at:
http://www.forbes.com/sites/stevedenning/2013/01/21/what-went-wrong-at-boeing/#30d2a3465aad, Accessed on: 27th October,
2016.
[48] J Laaksonen (2010). Lessons Learned from Olkiluoto 3 Plant, 9th October, 2010, Power Engineering, Available at:
http://www.power-eng.com/articles/npi/print/volume-3/issue-3/nucleus/lessons-learned-from-olkiluoto-3-plant.html, Accessed on:
27th October, 2016.
[49] L Jun-yan (2012). Schedule Uncertainty Control: A Literature Review, 2012 International Conference on Medical Physics and Biomedical
Engineering (ICMPBE2012), Available at: http://www.sciencedirect.com/science/article/pii/S1875389212016069, Accessed on: 27th
October, 2016.
[50] L P Leach (1999). Critical Chain Project Management Improves Project Performance‖, Project Management Journal, vol. 30, no. 2, pp.
39-51, June 1999, Available at: https://www.pmi.org/learning/library/critical-chain-pm-improves-performance-5305, Accessed on:
27th October, 2016.
[51] R de Neufville and S Scholtes (2011). Flexibility in Engineering Design, The MIT Press, Cambridge, Massachusetts, London, England.
[52] D Cleland and W R King (1997). Project Management Handbook, Chapter 20, Critical Success Factors in Effective Project
implementation by J K Pinto and D P. Slevin, 2nd Edition, Wiley.
[53] OGC (2012). PRINCE2; Directing Successful Projects, 5th Ed, TSO, Norwich.
[54] G J Roedler and C Jones (2005). Technical Measurement, A Collaborative Project of PSM, INCOSE, and Industry, 27th December 2005,
INCOSE-TP-2003-020-01, http://www.incose.org/docs/default-source/ProductsPublications/technical-measurement-guide---dec-
2005.pdf?sfvrsn=4, Accessed on: 28th October, 2016.
[55] D Dvir et al. (1998). In search of project classification: a non-universal approach to](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-71-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 68 of 69
project success factors, Research Policy, Volume 27, Issue 9, December 1998, Pages 915–935, Available at:
http://www.sciencedirect.com/science/article/pii/S0048733398000857, Accessed on: 12th November 2016.
[56] T Chow and D-B Cai (2007). A survey study of critical success factors in agile software projects, Journal of Systems and Software,
Volume 81, Issue 6, June 2008, Pages 961–971, Available at: http://www.sciencedirect.com/science/article/pii/S0164121207002208,
Accessed on: 12th November 2016.
[57] G L Ragatz et al. (1997). Success Factors for Integrating Suppliers into New Product Development, Product Innovation Management,
Volume 14, Issue 3, May 1997, Pages 190–202, Available at: http://onlinelibrary.wiley.com/doi/10.1111/1540-5885.1430190/abstract,
Accessed on: 12th November 2016.
[58] Koutsikouri et al (2008). Critical success factors in collaborative multi-disciplinary design projects, Journal of Engineering, Design and
Technology, Available at: http://www.emeraldinsight.com/doi/abs/10.1108/17260530810918243, Accessed on: 12th November 2016.
[59] J Fortune and D White (2006). Framing of project critical success factors by a systems model, International Journal of Project
Management, Volume 24, Issue 1, January 2006, Pages 53–65, Available at:
http://www.sciencedirect.com/science/article/pii/S0263786305000876, Accessed on: 12th November 2016.
[60] SurveyMonkey (2016). Home, Available at: https://www.surveymonkey.co.uk/, Accessed on: 27th October, 2016.
[61] Dr. B Mehlenbacher (2002). Communication Disasters, ENG 421: Computer Documentation Design, Available at:
http://www4.ncsu.edu/~brad_m/teaching/eng%20331/Lessons/communication.html, Accessed on: 28th October, 2016.
[62] M V Malko (no date). The Chernobyl Reactor: Design Features and Reasons for Accident, Joint Institute of Power and Nuclear Research,
National Academy of Sciences of Belarus, Available at: http://www.rri.kyoto-u.ac.jp/NSRG/reports/kr79/kr79pdf/Malko1.pdf, Accessed
on: 28th October, 2016.
[63] D Haughey (2014). A Brief History of Project Management, Project Smart, Available at: https://www.projectsmart.co.uk/brief-history-
of-project-management.php, Accessed on: 28th October, 2016.
[64] C Borysowich (2008). Pros & Cons of Gantt Charts, Toolbox.com, Feb 2, 2008, Available at: http://it.toolbox.com/blogs/enterprise-
solutions/pros-cons-of-gantt-charts-22233, Accessed on: 6th November 2016.
[65] T Browning (2012). The Design Structure Matrix: A Tool for Managing Complexity, Scientific American, 15th September 2012, Available
at: http://blogs.scientificamerican.com/guest-blog/the-design-structure-matrix-a-tool-for-managing-complexity/, Accessed on: 28th
October, 2016.
[66] S D Eppinger et al (1992). Organising the Tasks in Complex Design Projects: Development of Tools to Represent Design Procedures, NSF
Design and Manufacturing Systems Conferences, Atlanta, Georgia, January 1992, Available at:
http://web.mit.edu/people/eppinger/pdf/Gebala_NSF1992.pdf, Accessed on: 28th October, 2016.
[67] S D Eppinger and T R Browning (2014). Design Structure Matrix Methods and Applications, The MIT Press, Cambridge, Massachusetts,
London, England.
[68] DSMweb.org (2009). The Design Structure Matrix (DSM), Available at: http://www.dsmweb.org/, Accessed on: 28th October, 2016.
[69] Oracle (2016). Primavera Enterprise Project Portfolio Management, Available at:
https://www.oracle.com/uk/applications/primavera/index.html, Accessed on: 31st October 2016.
[70] Planning Planet (2009). Resource Loading, Posted Thu, 2009-02-05, Available at:
http://www.planningplanet.com/wiki/422401/resource-loading, Accessed on: 28th October, 2016.
[71] GanttChartExample.com (2016). Gantt Chart Example, Available at: http://www.ganttchartexample.com/, Accessed on: 28th October,
2016.
[72] S Vajna et al (2010). Designing the Solution Space for the Autogenetic Design Theory (ADT), International Design Conference – Design
2010, Dubrovnik, Croatia, May 17-20, 2010, Available at: https://www.designsociety.org/download-
publication/29490/designing_the_solution_space_for_the_autogenetic_design_theory_adt, Accessed on: 29th October, 2016.
[73] M Kreimeyer and U Lindemann (2011). Complexity Metrics in Engineering Design; Managing the Structure of Design Process, 1st Ed,
Springer-verlag Berlin Heidelberg.
[74] D Simon and F Simon (2005). Das Wundersame Verhalten von Entwicklern beim Einsatz von Quellcode-Metriken. Proccedings in
DASMA Software Metrik Kongress: Metrikon 2005, Shaker Verlag, Aachen, 2005, pages 263-272, Available at:
http://www.softwarekompetenz.de/servlet/is/28003/?print=true, Accessed on: 29th October, 2016.
[75] M Bremer and B McKibben (2011). Escape the Improvement Trap: Five Ingredients Missing in Most Improvement Recipes. Boca Raton,
London, New York.
[76] A Kline et al (2003). Creating and Using a Performance Measure for the Engineering Design Process, American Society for Engineering
Education, Available at:
http://www.webpages.uidaho.edu/ele/scholars/Results/Publications/asee/ASEE_2003_Creating_Performance_Measure.doc,
Accessed on: 29th October, 2016.
[77] Balanced Scorecard Institute (2016). Balanced Scorecard Basics, Available at: http://balancedscorecard.org/Resources/About-the-
Balanced-Scorecard, Accessed on: 1st November, 2016.
[78] D Huether (2010). How Do You Know Your Metrics Are Worth It, Make Things Better, Available at:
http://www.derekhuether.com/2010/02/11/how-do-you-know-your-metrics-are-worth-it/, Accessed on: 2nd December 2016.
[79] R Navon (2005). Automated Project Performance Control (APPC) of construction resources, Automation in Construction 14(4):467-476 ·
August 2005, Available at:
https://www.researchgate.net/publication/223091830_Automated_project_performance_control_of_construction_projects,
Accessed on: 29th October, 2016.
[80] A Yassine et al (2003). Information hiding in product development: the design churn effect, Research in Engineering Design 14 (2003)
145–161, DOI 10.1007/s00163-003-0036-2, Available at: http://necsi.edu/affiliates/braha/RED03_Info.pdf, Accessed on: 28th October,
2016.
[81] N Brook (2016). Guidelines for Requirements Engineering and Management, Tractebel Engineering.
[82] R Torbett et al (2001). Design Performance Measurement in the Construction Sector: A Pilot Study, Science and Technology Policy
Research, Sussex University, Available at: http://www.sussex.ac.uk/spru/documents/sewp66, Accessed on: 29th October, 2016.
[83] US Department of Energy (2010). DOE G 413.3-12, U.S. Department of Energy Project Definition Rating Index Guide, Available at:
https://www.directives.doe.gov/directives-documents/400-series/0413.3-EGuide-12, Accessed on: 29th October, 2016.
[84] NASA (2010). PDRI; Project Definition Rating Index Use on NASA Facilities, April 2000, Available at:
http://www.hq.nasa.gov/office/codej/codejx/Assets/Docs/ProjectDefinitionRatingIndex.pdf, Accessed on: 29th October, 2016](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-72-320.jpg)
![Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
Page 69 of 69
[85] Construction Industry Institute (2015). Project Definition Rating Index (PDRI), Available at: https://www.construction-
institute.org/scriptcontent/more/rr113_11_more.cfm, Accessed on: 29th October, 2016
[86] Department of Defense (2010). Technology Readiness Levels in the Department of Defense (DoD), Defense Acquisition Guidebook),
Available at: https://www.army.mil/e2/c/downloads/404585.pdf, Accessed on: 31st October.
[87] NASA (2007). Systems Engineering Handbook, NASA/SP-2007-6105 Rev1, December 2007, Available at :
http://foiaelibrary.gsfc.nasa.gov/_assets/doclibBidder/tech_docs/5.%20NASA%20SP-
6105%20Rev%201%20(Sys%20Eng%20Handbook).pdf, Accessed on: 31st October, 2016.
[88] NASA (2012). Technology Readiness Level, Oct. 28, 2012, Available at:
http://www.nasa.gov/directorates/heo/scan/engineering/technology/txt_accordion1.html, Accessed on: 31st October, 2016
[89] M F Austin and D M York (2015). System Readiness Assessment (SRA) An Illustrative Example, 2015 Conference on Systems Engineering
Research, Volume 44, 2015, Pages 486–496, http//www.sciencedirect.com/science/article/pii/S1877050915002677, Accessed on: 31st
October, 2016.
[90] M Westcott et al (2013). The DMI Design Value Scorecard: A New Design Measurement and Management Model, Design Management
Review, Volume 24, Issue 4, pages 10–16, Winter 2013, Available
at:.http://onlinelibrary.wiley.com/wol1/doi/10.1111/drev.10257/full, Accessed on: 7th November 2016
[91] N Brook (2016). Risk Management – Management Procedure, ref 3887; PALLAS Project, Tractebel Engineering.
[92] L Mieritz (2012). Thisiswhatgoodlookslike; Gartner Survey Shows Why Projects Fail, Gartner, 1th June 2012, Available at:
http://thisiswhatgoodlookslike.com/2012/06/10/gartner-survey-shows-why-projects-fail/, Accessed on: 29th October, 2016
[93] Royal Academy of Engineering (2004). The Challenges of Complex IT Projects, , Available at: http://bcs.org/upload/pdf/complexity.pdf,
Accessed on: 29th October, 2016
[94] A Denker (2007). The Challenge of Large-Scale IT Projects, World Academy of Science, Engineering and Technology, International
Journal of Social, Education, Economics and Management Engineering Vol:1, No:9, 2007, , Available at:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.193.3858&rep=rep1&type=pdf, Accessed on: 29th October, 2016
[95] R Frese (2003). Project success and failure: What is success, what is failure, and how can you improve your odds for Success, 16th
December 2003, , Available at: http://www.umsl.edu/~sauterv/analysis/6840_f03_papers/frese/, Accessed on: 29th October, 2016
[96] Modern Airliners (no date). Boeing 787 History, the creation of the Dreamliner, Available at:
http://modernairliners.com/boeing-787-dreamliner/boeing-787-dreamliner-history, Accessed on: 29th October, 2016.
[97] M Mecham (2011). 787: The Century’s First Jet to Fly. Sep 26, 2011, Aviation Week & Space Technology, Available at:
http://aviationweek.com/awin/787-century-s-first-jet-fly, Accessed on: 29th October, 2016.
[98] A Casey (2012). Boeing 7E7 case study, 21 November 2012, Available at: https://prezi.com/64hzwunhvlap/boeing-7e7-case-study/,
Accessed on: 29th October, 2016.
[99] P Ausick (2014). Why a Boeing 787-9 Dreamliner Costs $250 Million, June 17, 2014, 24/7 Wall St, Available at:
http://247wallst.com/aerospace-defense/2014/06/17/why-a-boeing-787-9-dreamliner-costs-250-million/, Accessed on: 29th October,
2016.
[100] Y Zhao, PhD (2013). Why 787 Slips Were Inevitable? Available at: http://zhao.rutgers.edu/787-paper-12-02-2013.pdf, Accessed on:
29th October, 2016.
[101] J Koster et al (2012). Hyperion UAV: An International Collaboration, Conference: AIAA-ASM, Available at:
https://www.researchgate.net/publication/263045116_Hyperion_UAV_An_International_Collaboration, Accessed on: 31st October
2016.
[102] I Moir and A Seabridge (2008). Aircraft Systems Aircraft Systems: Mechanical, electrical, and avionics subsystems integration, 3rd Ed,
John Wiley & Sons, Ltd.
[103] J Koster et al (2011). Workforce Development for Global Aircraft Design , Full-text available · Conference Paper · Jan 2011, Available at:
https://www.researchgate.net/figure/267593377_fig1_Figure-1-Boeing-787-Global-Work-Breakdown-Structure-1, Accessed on: 31st
October 2016.
[104] M Thurber (2009). An All-composites Learjet, Business Jet Traveler, Wednesday, April 1, 2009, Available at:
http://www.bjtonline.com/business-jet-news/an-all-composites-learjet, Accessed on: 31st October 2016.
[105] U Irfan (2014). How Lithium Ion Batteries Grounded the Dreamliner: Official report on Boeing 787 fires tells a cautionary tale about
advanced batteries, ClimateWire on December 18, 2014, Scientific American, Available at:
https://www.scientificamerican.com/article/how-lithium-ion-batteries-grounded-the-dreamliner/, Accessed on: 31st October 2016.
[106] Telegraph Travel (2013). Boeing 787 Dreamliner: a timeline of problems, Telegraph Travel, 28 July 2013, Available at:
http://www.telegraph.co.uk/travel/comment/Boeing-787-Dreamliner-a-timeline-of-problems/, Accessed on: 29th October, 2016.
[107] C Negroni (2015). FAA Is Doing Nothing About Continued Boeing Dreamliner Battery Failures, 29 October 2015, Gizmodo, Available at:
http://www.gizmodo.com.au/2015/10/faa-is-doing-nothing-about-continued-boeing-dreamliner-battery-failures/, Accessed on: 29th
October, 2016.
[108] P Marks (2013). Grounded: Where the Boeing Dreamliner went wrong, 6 February 2013, New Scientist
Available at: https://www.newscientist.com/article/mg21729036-700-grounded-where-the-boeing-dreamliner-went-wrong/,
Accessed on: 29th October, 2016.
[109] W Kaufman (2013). Dreamliner Woes Expose FAA's Potential Weak Spots, January 23, 2013, npr Business
Available at: http://www.npr.org/2013/01/23/170096977/dreamliner-woes-expose-faas-potential-weak-spots, Accessed on: 29th
October, 2016.
[110] D Rushe (2013). Why Boeing's 787 Dreamliner was a nightmare waiting to happen, 18 January 2013, The Guardian, Available at:
https://www.theguardian.com/business/2013/jan/18/boeing-787-dreamliner-grounded, Accessed on: 8th November 2016.
[111] AeroInside (no date). Airline Incidents for aircraft type Boeing 787-8 Dreamliner, Available at:
https://www.aeroinside.com/incidents/type/b788/boeing-787-8-dreamliner, Accessed on: 31st October 2016.](https://image.slidesharecdn.com/integrationoftechnicaldevelopmentwithincomplexprojectenvironment-projectdec2016finalrev2-161230132554/85/Integration-of-technical-development-within-complex-project-environment-73-320.jpg)
Integration of technical development within complex project environment
- 1. i Integration of technical development within complex project environments Project Department of Computer Science at the University of York Author: Nick Brook Project Supervisor: Katrina Attwood Abstract The integration of design and safety functions within complex technical system development and project methodologies is still undeveloped. Deficits in management are primarily manifested through large numbers of major unanticipated changes. This variously results in overrunning costs and schedule and compromised fulfilment of important project requirements. These undesirable consequences are greatly magnified by complexity. This project will attempt to address this deficit. The high-level objectives are to develop a framework to facilitate better planning, monitoring and control of technical activities. This will be achieved through the identification, development and integration of suitable existing concepts and techniques into a complexity management framework. This should apply to all complex projects and particularly those relating to safety critical engineering projects. Primarily, the project builds upon the research which has been previously undertaken by this author in project failings, complexity and existing tools and techniques. The framework will be evaluated through questionnaire survey of several of its component parts, a review against the recommendations from literature and through case study application of the tools and techniques. It is the goal that all or parts of this framework can be put to practical application within industry. Statement of Ethics This dissertation, including literature research, questionnaire survey and conclusions, has carefully considered and adhered to the three principles of ethics specified by the University of York. These are - to do no harm, to ensure informed consent from human participants and to uphold sound principles of data confidentiality: Do No Harm No physical system or entity has been developed which could cause harm of any kind. The work undertaken pertains to literature research, the gathering of non-sensitive and non-confidential data via a questionnaire survey, and the development of research conclusions. Informed Consent Information which is freely available in the public domain has been appropriately referenced within this dissertation. Some information was acquired by direct requests to, and discussions with, the authors; at all times the authors have been made aware of the purpose of the request and the nature of the critical evaluation. Questionnaire respondents freely participated and were made aware of the purpose of the survey beforehand. Confidentiality of Data No sensitive or confidential data has been acquired, used or released during the course of this dissertation. Number of words is 39,473 as counted by the MS Word word count. This includes all of the body of the report, but excludes the appendices which are included in the project submission for completeness and interest.
- 2. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 ii Left intentionally blank
- 3. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 iii Table of Contents Table of Contents........................................................................................................................iii Acknowledgements.....................................................................................................................iv 1. Introduction .........................................................................................................................1 1.1 Overview............................................................................................................................................ 1 1.2 Objectives.......................................................................................................................................... 1 1.3 An understanding of complexity ....................................................................................................... 3 1.4 The importance of considering the organisation .............................................................................. 4 1.5 Existing management frameworks.................................................................................................... 5 1.6 Devising the Complexity Management Framework.......................................................................... 8 2. Identifying and quantifying project complexity......................................................................8 2.1. The complexity assessment matrix ................................................................................................... 8 2.1.1. Complexity themes.................................................................................................................... 9 2.1.2. Complexity Criteria.................................................................................................................. 10 2.1.3. Complexity Matrix ................................................................................................................... 11 2.2. How, where and when to assess..................................................................................................... 14 2.3. Outputs from complexity assessment............................................................................................. 15 2.4. Complexity profile and the interaction of complexity criteria........................................................ 15 3. Critical Project Success factors and their application in system development .......................17 3.1. Selection of success factors from literature.................................................................................... 17 3.2. Verification of success factors through questionnaire.................................................................... 18 3.2.2. Full dataset .............................................................................................................................. 19 3.2.3. By age ...................................................................................................................................... 22 3.2.4. By role...................................................................................................................................... 22 3.2.5. By industry............................................................................................................................... 23 3.2.6. Conclusion ............................................................................................................................... 24 4. System development planning techniques...........................................................................24 4.1. Overview.......................................................................................................................................... 24 4.2. Design Structure Matrix................................................................................................................... 25 4.2.1. Introduction............................................................................................................................. 25 4.2.2. Design Structure Matrix principles.......................................................................................... 26 4.2.3. Creating and applying the organisational architecture DSM .................................................. 27 4.2.4. Creating and applying process architecture DSM ................................................................... 27 4.2.5. Application of the process DSM within a complexity management framework..................... 30 4.2.6. Process-organisational MDMs and their application .............................................................. 33 5. System Performance Measures ...........................................................................................34 5.1. Introduction..................................................................................................................................... 34 5.2. Desirable properties of Performance Measures ............................................................................. 35 5.3. Selection of System Performance Measures from literature.......................................................... 38 5.3.1. Requirements .......................................................................................................................... 38 5.3.1.1. Satisfaction of Stakeholder and System Requirements .......................................................... 38 5.3.1.2. Requirements attributes ......................................................................................................... 39 5.3.2. Development health................................................................................................................ 40 5.3.3. Process maturity...................................................................................................................... 40 5.3.4. System maturity....................................................................................................................... 41 5.3.5. Organisation, process and schedule complexity ..................................................................... 41 5.4. The verification of performance measures through questionnaire................................................ 42 5.4.1. Performance measures by age, role and industry................................................................... 43
- 4. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 iv 5.4.2. Conclusion ............................................................................................................................... 43 5.5. The collective use of performance measures ................................................................................ 43 6. Managing system development risks...................................................................................44 6.1. Introduction..................................................................................................................................... 44 6.2. Important concepts ......................................................................................................................... 45 7. Complexity orientated development framework .................................................................46 7.1. Integrating the sub-processes together .......................................................................................... 46 7.2. Analysis of framework against criteria within existing literature.................................................... 49 8. Case study application of framework...................................................................................50 8.1. Purpose............................................................................................................................................ 50 8.2. Case study 1 – Boeing Dreamliner................................................................................................... 50 8.2.1. Background.............................................................................................................................. 50 8.2.2. Work Breakdown Structure..................................................................................................... 51 8.2.3. Complexity assessment ........................................................................................................... 53 8.2.4. Critical Success Factors............................................................................................................ 56 8.2.5. Planning technique.................................................................................................................. 58 8.2.6. Performance measurement .................................................................................................... 58 8.2.7. Risk management .................................................................................................................... 59 8.2.8. Comparing framework findings with project outcomes.......................................................... 59 9. Results and Evaluation........................................................................................................60 10. Conclusion..........................................................................................................................64 References…………………………………………………………………………………………………………………………………..66 Appendix A Glossary of terms and acronyms Appendix B Existing management frameworks Appendix C Critical Success Factors Appendix D Response summary Appendix E Complexity questionnaire Appendix F Questionnaire results ranked by influence Appendix G Dataset sample sizes Appendix H Questionnaire results filtered by age Appendix I Questionnaire results filtered by role Appendix J Questionnaire results filtered by industry Appendix K Design Structure Matrix Appendix L System Performance Measures Appendix M Risk register template Appendix N Risk attributes as metadata Appendix O Criticality Assessments for Boeing Dreamliner case study Appendix P Timeline to Boeing Dreamliner Appendix Q Complexity management case study using OL3 and AREVA’s EPR Appendix R Full questionnaire results per respondent Acknowledgements I would like to thank Lana and Anna for coping with my extended period of further education and putting up with my many grumpy moods. I also have the utmost respect for the University of York and the teaching and administrative staff who have made my studies so enjoyable and rewarding. Finally, my Mum and Dad have been extremely supportive and I could not have done this without them. Continuous effort - not strength or intelligence - is the key to unlocking our potential. - Winston Churchill.
- 5. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 1 of 70 1. Introduction 1.1 Overview The characteristics of complex systems include a difficulty in determining outcomes from the inputs [1] and there being present a ‘degree of disorder’ [2]. The relationships and dependencies between processes, organisations and other external systems describe perfectly that of any complex system. What is more, the general trend is for development complexity to increase over time. This is while simultaneously catering for advances in technology and the downward pressures on development timescales and cost. The result is greater prevalence of the wicked problem, that is a technical system development that is ‘highly resistant to resolution’ [3]. This can be thought of as Technical Development Complexity. This dissertation builds on the research undertaken within the previous literature survey [4]. This work described the high failure rate amongst projects, a summary of the main reasons for this and areas of further investigation. Also a variety of definitions were given for complexity and compelling reasons that this should be the focus of attention when planning for the development of a system. The dissertation develops the literature survey’s conclusions, describing a framework for the treatment of complex system development within a wider project environment. As the network of individual activities and their interactions increase it is reasonable to expect that the mechanisms that are used to achieve a successful outcome will be greater in terms of their magnitude and resources to undertake them. Therefore, complexity inevitably influences the effort required to plan, monitor and control development activities. These mechanisms and conditions can be specified within development processes but it would appear reasonable that these also need to be tailored to suit the particular development characteristics and especially those relating to its complexity. These are known as Critical Success Factors (CSF) and describe ‘essential areas of activity that must be performed well if you are to achieve the mission, objectives or project’ [5]. However, it should be noted that outside describing their form and importance, literature provide no satisfactory methods or process for determining suitable CSFs. 1.2 Objectives It will be proposed that complexity comprises a number of aspects and that, for the purpose of this dissertation, these can be comprised of themes and criteria. These aspects will not be homogeneous across all the development activities nor the development lifecycle. This variation across a project can be seen as a complexity profile that will evolve throughout the development lifecycle. Outwardly similar developments, even within the same organisation, may not exhibit the same complexity characteristics. This can often be attributed to the presence (or lack) of constraints outside the development team’s control and the result of influential management decisions. Consideration of complexity will have a number of goals: Identify the most influential Critical Success Factors (CSF) to put in place environmental conditions required for a successful outcome; Identify methods to put the responsibility for interventions where it is best placed through early identification the establishment of environmental factors is often beyond the direct influence of the development team or its managers [6][7]; Recognise how complexity evolves throughout the development and how some aspects increase as others diminish. As such the methods to plan, monitor and control development activities will need to evolve accordingly throughout the development lifecycle; Describe how interventions can influence complexity and displace its effects elsewhere; Recognise and anticipate development risks identified through the consideration of complexity early in the development lifecycle and during detailed planning activities; Develop a framework for assigning interventions to defined risks more closely, at the appropriate level within the Work Breakdown Structure (WBS). The identification of interventions will be initiated by the recognition of CSFs and risk planning;
- 6. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 2 of 70 Propose methods to guide where the most effort should be applied in planning, monitoring and controlling activities through the identification of areas of maximum complexity. This is where the greatest number of residual risks is likely to be. This is most important where there exists a constraint of available resource which inevitably means effort must be prioritised. This is a factor of varying significance in all but the most exceptional of technical developments; Ensure the framework is scalable with regards to the size of the development and to the level within the WBS Structure to which it can be applied. This is closely related to the preceding item; Provide a framework to supplement planning techniques to better capture aspects of complexity, such as coupling behaviours between activities. This will also allow analysis to identify individual activities or clusters of planned activities that have the potential to influence the overall development outcome disproportionately and as such deserve enhanced management effort; Guide the selection and implementation of development status measures. These should be determined on the basis of the impact of particular areas of technical development and the key activities within these areas. These should be selected primarily for their use in controlling the development and initiating avoiding action and not merely for purpose of reporting status. An underlying theme of the framework will be that the activities it directly influences (planning and monitoring), and indirectly influences (controlling), are chosen to balance risk and benefit with effort involved. It should complement current techniques, and absolutely not conflict with them. It is the aim that this will allow their use with a minimum of additional resource overhead. It is recognised that the framework will not be implemented in isolation of existing project management and system engineering tools and techniques. A secondary objective is to allow project and process complexity to be understood through practical usage of the techniques. This can be achieved through the feedback of learning back into the process to improve it and develop better responses for future system development. There are often latent issues within system development that are only manifested through delays, cost overruns or quality issues. This is problematic for several reasons. The intervention required is generally greater the later it is left. Also as the development tempo intensifies, the effort required to change the process and organisation will inevitably draw on the very same resources as those developing the system itself. The disruption and additional uncertainty of making these changes will exacerbate the impact of the original latent issue. In fact, the scope of such a change can be seen as a project in its own right [8] and is as undesirable as it is unnecessary. Following on, there is evidently great benefit in balancing the effort between undertaking risk management as opposed to that of issue management. This balance is often forgotten and especially where there is insufficient substantiating data to support the mitigation of risks or to support change. The focus instead is on constant management of issues arising from bad risk planning, commonly known in project management as ‘firefighting’ [9], leading to no long-term sustained improvements in performance. A goal of the framework is to provide strong enough indicators to prompt this early management action. Many of the principles in the management of system related risks are equally applicable to project risks. This can be illustrated by an annotated Bow Tie diagram [10][11] as shown in Figure 1. The role of complexity management is both to identify the threats on the left hand side that are brought about by complexity and to determine effective methods for controlling them. Selecting CSFs should be followed by an understanding of residual risks and ways of providing an early warning of their realisation. A semi-formal method for identification of CSFs within a framework will encourage their wider use and raises the possibility of their use at development and project review points, such as design reviews and stage gates. An objective should be the early identification and best possible assignment of responsibility for the of the mitigation of development risks, not only across the breadth of the project but to a sufficiently senior level of management [12].
- 7. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 3 of 70 Heuristic techniques will be chosen and developed as a means of guiding decision making and providing inputs into traditional project management and system engineering processes. These will anticipate areas within system development that require the most effort and where effort will bring about the greatest impact. This will provide multiple Plan- Do-Check-Act (PDCA) cycles, an iterative four-step management method used to control and allow continual improvement of processes and product development [13]. Also known as the Demming Cycle, it is equally applicable within the development process. In this application it will be used to influence the development environment, processes and planning for complex system development. The techniques described will use a mix of decomposition and reductionist methods, along with a heuristic approach, to optimise resource usage required both of the framework and by any additional development effort that results from its application. Heuristic techniques are well suited to the problems associated with planning and executing a complex system development and work well alongside iterative development techniques generally advocated for complex system development. There are many variables in play within complex technical development. While optimal solutions are achievable for a particular aspect, optimisation across all these often competing aspects is very difficult within the current methodologies. An important concept is that of feeding back learning, into the on-going development or subsequent developments, to improve future applications of the framework. The determination of CSFs or risks should include the intended or optimal outcome which can be compared against actual outcomes during the development lifecycle for the efficacy of the CSFs and their implementation to be assessed. This allows improvements to be made in a timely fashion for the benefit of the development. It also avoids the unsatisfactory analysis of lessons learned, often at the very end of the project when personnel drift away and the motivation to hold a review wanes [14]. A glossary of terms and acronyms used throughout this dissertation can be found in Appendix A. 1.3 An understanding of complexity Complexity must be understood before it can be modelled. It has previously been described and decomposed in a large number of ways, both within and outside the domain of engineering and many other overlapping fields of research. Examples include the Strategic Highway Research Program [15] and Helmsman Institute [16]. For the purpose of this framework it is important for complexity to be represented as simply as possible, while covering all the necessary aspects as effectively as possible. This may appear counterintuitive but is vital if the principles of the framework are to be applied in practice. This section will summarise research on the nature of complexity and will develop it for use in a complexity management framework. A more detailed explanation of the properties of complexity can be found within the literature survey [4]. Figure 1. Annotated Bow Tie diagram [11].
- 8. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 4 of 70 Definitions of complexity highlight is highly structured nature [17][18] which encompasses many interactions [19] and a large array of possible configurations [20][21]. Other characteristics include there being no relationship between the behaviour of the overall structure and that of its individual parts and the ‘response’ of system elements to changes to other interrelated system elements [20][21][22]. Together these aspects make the overall structure difficult to understand [23]. It is noted that uncertainty is often omitted from the definition of complexity though it is intertwined with several of the aforementioned characteristics [24], such as configuration variation. This project proposes that ambiguity be considered [25] as a distinct and separate characteristic, alongside Uncertainty. Ambiguity is a common property of system development, caused by the absence or unavailability of reliable information, and resulting in the formation of assumptions. Ambiguity is a dominant factor in the early stages before requirements have been agreed and also during development phases before coupling of large numbers of activities are fully understood. Assumptions cannot be completely validated until later in the development and only after ambiguity has been addressed. An understanding of how the individual characteristics of complexity interact with each other will be pivotal to the determining of CSFs. Further dimensions of complexity will be elaborated upon in Section 2. 1.4 The importance of considering the organisation The main two components of a system development, apart from the actual system being developed, are the development process and the development organisation. These can be considered to be within the direct control of the management team responsible for system development. Additionally, there will be a number of external interfaces and constraints that will be outside direct control of the system development management team. These may be from within the actual project itself, such as overall project budget or resource availability, or entirely outside of the project. Examples of the latter may be stakeholders, regulators and existing adjacent operating systems. An understanding of these boundaries will be important in assigning responsibility for interventions. Figure 2 shows the British Computer Society complexity model with tiered levels of external interface. This is a simplified representation and does not show how these tiers interact but nevertheless is useful in categorising project influences. Specifically, the model includes the ‘macro-environment’, which encompasses interfaces furthest from the control of the development team such market condition, legislation and regulatory frameworks. ‘Micro-environment’ relates to the customer and stakeholders and how requirements are captured and traded-off [26]. The ‘organisational environment’ and the ‘project environment’ relate to the system development and the resource and process constraints imposed on it. The organisational relationship Figure 2. The British Computer Society’s complexity model [26]. Figure 3. Type of relationship between engineering and project management teams (adapted from SEBoK [27]).
- 9. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 1 The ‘Iron Triangle’ describes The three related constraints of scope, budget, and schedule [32]. Page 5 of 70 between system engineering and the project environment is also an important consideration and something which this model omits. Figure 3 shows three distinct project organisational types from the viewpoint of the technical development and is adapted from the Systems Engineering Book of Knowledge (SEBoK) [27]. These organisational types strongly influence what is and what is not within the direct control of the system engineering team. The definition of the inputs and outputs between project and system development organisations will allow a better understanding of where the responsibility for putting in place and implementing CSFs and risk mitigations resides. These interfaces should be defined within such documents as the Systems Engineering Management Plan [28] and the Project Management Plan [29]. In the first diagram organisational behaviour is typified by transactional type relationships. Engineering may be contracted out or undertaken by a different organisation, possibly located elsewhere. There may be more than one engineering organisation, with complexity increasing according to the number of interfaces. The project management organisation is responsible for managing the outputs only and there is a strong emphasis on functional organisational structures [30]. In the second diagram the overlap in responsibilities and accountabilities can also result in a form of complexity. There is a lesser emphasis on transactional relationship and the project management team have a greater influence over the management of engineering activities. The degree of overlap will affect this relationship. The third diagram could be a matrix type structure. The project management team has overall control and the engineering management team has the least scope to determine how the development will be managed. The type of relationship will thus affect the scope and the method of managing complexity. In the first example system engineering is entirely self-sufficient in terms of contractual complexity. In the last example this is far less likely. Thus the treatment should be amended accordingly. 1.5 Existing management frameworks There has been a large volume of literature on the subject of managing projects and technical development activities based on a blend of theory and practical experience. The framework presented in this project will take the existing management frameworks and models, many discussed in the literature survey [4], and will attempt to reconcile inconsistencies in approach and content. This section will both consolidate the discussion in the literature survey [4] and introduce concepts that have been found subsequently. The predominant systems engineering model is the ‘V-model’ as shown in Figure 4. It does not specifically address complexity but has some concepts that will be applied within the developed framework. Specifically, its definition of the development lifecycle and requirements-centric view is useful. The satisfaction of requirements is a convenient way of looking at complexity and the monitoring and control of requirements should form an integral part of the way that a complex system is managed. The V-model represents a typical idealised development lifecycle, so the phase descriptions within this model will be used to profile complexity. Of the system engineering and project management methodologies that were previously discussed in the literature survey [4] the Strategic Highway Research Program’s 5DPM was of particular interest. This methodology is described in some detail in both their ‘Guide to Project Management Strategies for Complex Projects’ [31] and also in ‘Managing Mega-Project Complexity in Five Dimensions’ [31]. Rather than considering the usual three project dimensions, i.e. the iron triangle1 [32], it Figure 4. The V-model [26].
- 10. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 6 of 69 introduces the concepts of ‘financing’ and ‘context’ as shown in Figure 5. 5DPM also considers CSFs from the perspective of complexity. Furthermore, this is done within several feedback cycles. Together these represent considerable synergy with many of the concepts that the author previously felt were worthy of consideration. Finance becomes of utmost consideration as both the overall magnitude of the development and risk increases. This is exemplified by the difficulties experienced in securing investment for the UK’s nuclear new- build programme [32]. For the purposes of technical development, it is an imposed constraint by virtue of being outside the direct control of the management and will be treated as such in the model. The fifth constraint of the 5DPM model is that of context, describing the project environment, including potential constraints and interfaces. The process can be decomposed as follows: Review project factors in each of the five areas; Identify and prioritise complexity factors; Develop 5DPM complexity map; Define CSFs with sub-process steps such as assemble project team, select project arrangements and prepare early cost model and finance plan; Develop project action plan to address resource issues; Re-evaluate complexity map on commencement of project. 5DPM describes reviews at defined intervals with an iteration of the above steps. This would most naturally align with a Stage Gates type governance structure [29], but it could be envisaged that interim reviews would be undertaken on a periodic basis if stage gates were deemed too far apart for effective control. Overall the model does not try to model complexity itself but rather attempts to model the component parts (within the five dimensions) and manage them to better manage a complex project environment. By focussing on these five dimensions the resulting assessments may provide a superficial view of project management orientated aspects only. Despite this there is considerable merit to the high-level approach and synergy with the principles of the PDCA Cycle [4]. The Helmsman Institute complexity assessment [16] is advocated by the International Centre for Complex Project Management [34]. This again has five areas of general consideration consisting of: Context Complexity; People Complexity; Ambiguity; Technical Challenge; Project Management Complexity. Within these categories there are specific factors. Those solely relating to technical development are: Integration Complexity; System Development Complexity; Impact on Infrastructure. Each of the categories is in turn considered against a large number of factors relating to such concepts as stakeholder numbers and alignment, uncertainty and abstraction. Importantly this framework introduces the concept of ambiguity as does Pich et al [25] and McGowana et al [35]. In each ambiguity is seen as a distinctly separate, though closely coupled, property to that of uncertainty. The consideration of complexity is of interest within the wider domain of business in general. The concept of VUCA (volatility, uncertainty, complexity and ambiguity) [36], as shown in Figure 6, considers many of the previously discussed themes. Volatility can be viewed as the result of one or the product of several complexity characteristics. Traits attributable to volatility include a general lack of understanding of the likely impacts of an event Figure 5. Five dimensional project management model [15].
- 11. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 7 of 70 along with the impacts being ‘unexpected’ and ‘unstable’. Here volatility is described as a factor in its own right and complexity is described alongside the others rather than being composed of them. Nevertheless, the presentation of an unpredictable business environment in terms of a limited number of characteristics is useful. Much of the literature references complexity without sufficiently breaking it down into its component parts. Classical definitions of the components of complexity were described detail within the literature survey and are as follows [4][37]: The ‘Kolmogorov-Chaitin complexity’, also known as Program-size Complexity [38]; ‘Nonlinearity’; ‘Emergent Complexity’. Essentially the first component is the number of interfaces relating somewhat to the development magnitude but also its nature. A large technical development may contain duplicate tasks and low interdependency, thus limiting its Program-size Complexity, while a comparably smaller technical development may conversely have many interdependencies and highly intricate activities. The latter two components collectively can be closely aligned with the previously described volatility within the VUCA framework. A relatively small change results in far-reaching or a disproportionately large impact. Expanding the definition of emergence for the benefit of its use in an assessment we may say that it has a number of properties [39]: Radical novelty – new and unanticipated properties emerge from each interaction; Coherence – though unexpected the new properties are consistent with rules and behaviours; Wholeness – emergence causes the new system that is greater than the sum of its parts; Dynamic – changes from emergence will continue to evolve as long as there is emergence; Downward causation — the system as a whole dictates the behaviour of its parts as well as the system being influenced by the interaction of its parts. This may be caused by two common dynamics, either separately or in conjunction: No one is in charge – thinking in terms of a technical development organisation this could be characterised by a network-centric organisation with a decentralised structure; Simple rules engender complex behaviour – using the example of an organisation this may be a large hierarchical organisation with many functions. This section describes the decomposition and treatment of complexity from varying perspectives. The development of the 5DPM process supports the view that simple consideration of the ‘iron triangle’ is insufficient. It also reinforces the importance of complexity and the validity of an iterative approach. However, its treatment of complexity is overly simplistic. The complexity framework developed in this project will adapt the broad methodology of the 5DPM process, while proposing a more precise definition for development complexity using a combination of prevailing theory and practical application. Specifically, definitions of complexity outlined either in the literature survey or in this section will be re-categorised to fit within the model, maintaining consistency. Figure 6. The VUCA framework [36].
- 12. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 2 The objectives described in the summary of the literature survey [4] will be adopted with a single change. The development of Object Based Modelling (OBM) will be omitted. DSM will be used alone as a flexible technique that can be used to model the development process. This of course does not preclude the use of techniques, such as OBM, in parallel with DSM. Further information on the use of OBM can be found in the paper by Warboys and Keane [41]. Page 8 of 69 Complexity will be decomposed by applying it against the relevant themes within the technical development. It must also be considered that complexity is in the eye of the beholder. The observer will have experience and a perspective that will influence subjective analysis based on their ‘bounded rationality’ [40]. The assessment criteria will ultimately require to be qualified as much as possible to guide interpretation. A more detailed literature review of this area of research can be found in Appendix B. 1.6 Devising the Complexity Management Framework This dissertation will define complexity in terms of technical development and combine a method of assessing complexity with a number of techniques to allow the planning, monitoring and controlling of complex technical developments. To summarise this dissertation will2 : a. Identify a method of quantifying Technical Development Complexity; b. Provide a semi-formal method for the identifications of the pre-requisite pre-planning environmental factors, or so called critical success factors (CSFs). The outcomes from this method will be influenced by an individual project’s characteristics, primarily in terms of its complexity; c. Propose methods for planning technical development and modelling interdependencies using the method of Design Structure Matrix (DSM); d. Provide Performance Measures that relate to the technical development and, where possible, to the assigned CSFs. It is preferable that these are totally objective but it is accepted that there will be considerable subjectivity involved which will need to be constrained to enhance consistency by suitable guidance. These performance measures will be both leading and lagging and provide coverage of the development process biased towards the parts that are particularly significant for reasons either of criticality or of risk of deviation in terms of criteria such as cost, schedule, scope and quality. Coverage of the remaining parts of the development process will be at a lower degree of decomposition commensurate with criticality and risk; e. Propose methods of identifying risks and how this influences CSFs and the Performance Measures. f. Demonstrate how all these (a. to e.) can be used together in a Complexity Management Framework. Together items a. and b. can be thought of as pre-planning activities and together are a strategy for putting in place an environment conducive for the project. Items a. to e. are developed in more detail throughout this project, culminating in Section 7, which includes a framework map integrating all these items together. 2. Identifying and quantifying project complexity 2.1. The complexity assessment matrix The complexity concepts described above will be encompassed within a Complexity Assessment Matrix. The matrix will form the most important part of the Complexity Assessment which in turn forms a part of the overall framework. The outputs from the assessment are as follows: 1. Undertake assessment of system development at a high-level within the WBS; 2. Identify of ‘hotspots’ both within the WBS and within the individual themes and direct additional assessments where they are required. It should be noted that focussed assessments will not be possible in very early development stages due to the low level of definition and decomposition of the WBS; 3. Identify development risks and propose CSFs to counter significant levels of complexity. Record additional information such as: a. Rationale; b. Residual risks that remain once CSFs have been put in place; c. Proposed outcomes that are to be brought about by the CSFs.
- 13. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 9 of 69 4. Reassess the project complexity profile with the CSFs in mind to ensure that no new significant unmitigated hotspots have been created. Revise the outputs of step 3 as appropriate; 5. Identify risks for inclusion within the risk register. Any consideration of complexity needs to be complete with an adequate explanation of categorisations. The process will be applicable across the entire development lifecycle and should be scalable. It is to be expected that there will be overlap between definitions due to a combination of minor shortfalls in the model and observer perceptions. It is for this reason that the reasoning behind decisions should be recorded to afford the best chances of consistency in applying the process in future applications. As discussed complexity will be viewed from the perspective of satisfying requirements, whether technical or project management orientated, and will form a part of the framework. There will be interactions between complexity types and it is highly likely that the nature of the complexity will evolve over the development lifecycle. We shall call this the Complexity Profile. Complexity assessment will have the potential to become more detailed as the requirements are defined. It can be reasoned that a pre-requirement, capturing the ‘business mission’ [28], can only be assessed at a very high level while detailed system requirements can be decomposed to much greater detail. It is important that the treatment of complexity can be targeted where it will represent the greatest value, as ambiguity reduces between the derivation of stakeholder requirements and the final validation of the system [28]. This can be achieved by considering assessment against either the development’s WBS or Product Breakdown Structure (PBS), depending upon which is used. Lastly there should be a comparison between actual and intended outcomes and feedback of learning to improve decision making in both the current and future development processes. It should be borne in mind that the objective of the assessment of complexity is to inform future decision making in order to enhance the likelihood of success. Section 3 details the use of CSFs as an established technique to do this by reducing either the probability or impact of complexity related events. Both complexity assessment and the assigning of CSFs can be viewed as pre-planning activities and should be reviewed periodically for applicability and appropriateness. 2.1.1. Complexity themes Using a combination of the methodologies described within the literature survey and Section 1.4, a typical technical development can be considered to comprise a number of themes. These are captured under the three broad headings of internal interfaces, external interfaces and system development. The themes are designed to provide breadth of coverage over all areas pertinent to system development. Some themes, such as stakeholders and regulatory interfaces could have been combined but it was felt that the identification of independent themes would prompt more detailed analysis in areas that are particularly influential. This also led to the apportioning of a third of the themes to the system development itself, an area underrepresented in other methodologies such as those from 5DPM and the Helmsman Institute [32]. This ensures that the assessment is more balanced overall and specifically in areas such as satisfying requirements and completing scope as a part of technical development. An important consideration is the project environment within which the development is undertaken. There will be constraints imposed upon the technical development that will, to varying degrees, be out with the control of the development management. The organisation type, as discussed in Section 1.4, will be a significant factor but others include the traditional business-level requirements of cost, schedule and scope [32]. Organisational culture and business-level governance, such Stage Gate type processes [29], will also be influential in shaping the technical development. The investment constraints and funding profiles, as elevated to one of the 5DPM dimensions [31], must also be considered. The remaining themes used in the framework generally follow the commonly used themes of people, processes and technology [42]. These are used in business architecture and can be decomposed further to better suit the purposes of the complexity assessment.
- 14. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 10 of 69 People can be decomposed into the internal organisation, i.e. the development team and those external to it. External elements include contracting organisations, regulatory interfaces and all other remaining stakeholder groups that contribute to the development. The properties of the development process are important in terms of detail and number of interfaces. Lastly the technological aspect can be thought of as the nature of technology that the system will comprise, as well as the integration of the technology (internal interfaces) and external interfaces, i.e. how the system integrates with other systems. The nine Complexity Themes used in the framework are listed below. These include high-level considerations that may impact the analysis. 1. Internal factors a. (Project) environmental constraints – external factors (inputs and outputs only), schedule (such as pace and phased handover requirements), the imposed funding profile; b. Development process – how prescriptive and many interfaces; c. Internal organisation – ranging from few disciplines within same organization to many and organisation type; 2. External factors a. Contractual management – ranging from a few simple relationships to many relationships with differing contractual terms; b. Stakeholders - relating to system definition and validation; c. Regulatory interfaces – how many and how influential they are on the development; 3. System development a. External (system) interfaces – the degree to which system impacts or depends upon external systems (from none to part of system of systems); b. Technology – system requirements definition and verification relating to technology type (low to very high tech and low to highly novel); c. Internal (system) interfaces – the level of internal integration required between sub- systems ranging from few simple relationships to many complicated relationships. 2.1.2. Complexity Criteria Complexity Criteria can be assessed against each of these themes. The twin themes of uncertainty and ambiguity will dominate the early phases of any project. Though they are often classically excluded from the definition of system complexity they have such a profound effect on complexity management and will be elevated in importance. Rather than be seen as sub-sets of complexity they will sit alongside the other criteria. Uncertainty and ambiguity will naturally diminish under normal conditions as requirements are defined, verified and validated. Conversely emergence and nonlinearity will tend to increase over time as requirements are progressively defined, implemented, verified and validated. They will reach their zenith as the system approaches handover when even small changes will have a significant impact. The final criterion is that of Program-size Complexity. This is essentially the quantity of information that represents the system development. Techniques to manage this include expenditure of resource and effort (personnel and/or computer-tooling) and the use of modelling techniques. The system development Complexity Criteria are: 1. Uncertainty – likelihood of unexpected events occurring, e.g. stakeholder requirements change significantly during system early development or assumptions are found to be incorrect; 2. Ambiguity – incompleteness of knowledge about functional variables [25], i.e., a lack of information upon which to make decisions. An example may be that stakeholder requirements are undefined in a particular area of system development leading to assumptions being made or to delays in the development schedule while the stakeholder requirements are defined; 3. Emergence – how change to system configuration leads to unexpected behaviours and interactions and re-evaluation of derived system requirements. High Emergence is defined as unexpected
- 15. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 11 of 69 behaviours and interactions in many other components or sub-systems; 4. Non-linearity – the effect of a single small change to system requirements on other system requirements. A non-linear relationship is one where a change results in a disproportionate change in affected components and sub-systems. High Non-linearity is where a small change results in a large impact elsewhere; 5. Program-size Complexity – relates to the minimum amount of information required to describe a process, system or organisational requirements. Manifests itself in shortfalls in modelling and a general low level of understanding in the system or its development even when uncertainty and ambiguity is low. High Program-size Complexity requires a high level of effort to model and to maintain the model. Examples where this may be manifested are requirements management, configuration management and scheduling. In summary, uncertainty and ambiguity dominate early in the development lifecycle when their major effect is lots of low impact changes. Later in the development lifecycle there are less, high impact changes generally brought about by emergent and non-linear behaviours. Program-line complexity impacts the ability to model, understand and control these changes, with higher levels of this criteria making schedules and requirements progressively more difficult to manage. 2.1.3. Complexity Matrix The Complexity Themes and Complexity Criteria can be presented in matrix form. A scoring system offering five choices, from very low to very high, has been chosen. Areas to be considered (considerations) for each of the themes has been chosen from the literature survey [4] and Section 1.5. These looked at sources such as the Helmsman Institute [16] and NTCP Framework [43]. Considerations provide guidance during application of the assessment and aid in the assessment of appropriate areas of the WBS against each theme. Risk has been omitted as a consideration from any of the themes as it is best understood only after complexity assessment has been undertaken. In this way the assessment will differ from its peers who tend to consider risk as an input into an assessment rather than an activity that is informed by an assessment’s results. Considerations can be seen against the themes within the Complexity Matrix as shown in Figure 7. The most obvious considerations for Project Environmental Constraints are those of imposed cost, schedule and scope. The funding profile that may constrain the development schedule, as well as overall cost, and introduce additional assumptions in lieu due to sub-optimal activity sequencing. Components of schedule include pace, where the development schedule may need to be accelerated to meet external milestones, and phased handover requirements. There may be additional activities and schedule implications due to project governance. Other constraints include those relating to procurement and contract usage which may impose constraints on who contracts are let to and how contracted activities are managed. There may be stipulations with regards to the technology that is used, for example technologies developed in-house. Lastly, organisational culture and politics can have a dramatic effect in areas such as communication and decision making. Tailoring of the development process to achieve the correct balance of governance and rigour against agility and flexibility is an important consideration as is facilitation of integration between the technical disciplines. The use of technology in areas such as requirements management, configuration control and safety case management will influence not only the development process but also the organisation. The organisation will strongly influence lines of communication. It is assumed that the fundamental type of organisation structure is decided early in a project lifecycle. It is generally a difficult undertaking to change it in its entirety as it is often dictated by the parent organisation undertaking the project. However, it is expected that it may be influenced by the findings of the Complexity Assessment. Traditional hierarchical structures allow ease of governance while decentralised network-type structures offer better flexibility [30]. Other aspects include the number of individual disciplines and their particular experience with regards the development type, the technology and integration challenges.
- 16. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 12 of 69 Figure 7. Complexity Matrix.
- 17. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 13 of 69 The internal organisation will interact with contracted development organisations and will be required to manage the transfer of information (development inputs and outputs) and the interfaces between areas of scope to ensure there are no omissions or inconsistencies. The development organisation will be shaped by its place in the wider organisation and particularly how it interfaces with the project organisation. This will dictate whether the responsibilities for areas such as resource allocation, contract procurement, cost estimation and scheduling are within the remit of the development organisation and the level of autonomy it holds. Other factors include geographical location of disciplines and other interfacing organisations. Contractual management will of course depend on how much of the development is actually contracted out versus activities that remain within the internal organisation. The reasons for contracting out of parts of the technical development will vary, for reasons such as low internal resource or capability or where particularly specialist or technology specific support is required. The nature of the relationship between the development organisation and contract organisations will be dependent on the composition of the project organisation. Responsibilities for actual contract management, including processing of contract variations and scope may be outside the scope of the development organisation in that they may manage technical interfaces only. Conversely this may all be within the remit of the development organisation. Again the division between scope being undertaken by internal resource and that contracted out should be considered against the outputs of the Complexity Assessment. The identification of significant areas of concern may initiate a reconsideration of contracting strategy or else prompt the establishment of measures to better manage it such as rigorous pre-qualification of suppliers. This is an example of a CSF that was introduced as a concept in Section 1.1 and is subject to more discussion in Section 3. Stakeholders can be thought of as the direct source of all high-level requirements. They inevitably have a strong influence on requirements as they are derived in greater detail throughout the development lifecycle. Factors that will affect stakeholder management include the number and alignment of stakeholders. Non- alignment of stakeholders’ needs leads to trade-off of requirements to gain the appropriate approvals. Large numbers of conflicting requirements will require careful management and introduces the risk for conflict and delays. Other factors include location and availability of access to stakeholders. Incomplete stakeholder identification may precipitate later changes to requirements and scope with potentially significant impact. Long development schedules can increase the likelihood of changes to important stakeholders which can lead to changing requirements also. Regulatory bodies are important stakeholders which can have a tremendous impact on system requirements. Technical developments may have weak interfaces, with only one or two regulators, or there may be strong interfaces, with many regulators. Changes in legislation and regulations may occur at any time due to events outside the control of the organisation and regulatory regimes may vary greatly between geographical areas. The development of a system for use in several countries can introduce much complexity surrounding the management of potentially conflicting regulatory requirements. External interfaces are systems that interface with the system of interest. There may be few interactions of a purely transactional nature with other systems or many interfaces. A system being developed as a part of a system of systems [28] will invoke the greatest management effort. Interfaces may be not properly understood or may evolve over time beyond the control of the development organisation. Technology is simply how ‘novel’ and ‘high-tech’ [43] the technology that has been chosen and dictated by the requirements is. Novelty and high-technology levels, as described by Frank et al [43], generally require highly iterative development methodologies and high levels of expertise. Short development schedules using such technology require greater effort, specialised systems engineering techniques and incur disproportionate level of risk. System integration represents the ‘synthesis of a set of system elements into a realised system that satisfies system requirements, architecture and design’ [28]. There are likely to be many interfaces that need to be managed through integration, verification and validation activities. There are techniques to
- 18. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 14 of 69 manage this complexity, including early integration planning. Examples of other factors include the integration of system elements from other contracted organisations and the careful management of stakeholders to minimise ‘scoop creep’ [44]. Examples of Complexity Matrices, completed for the case studies, can be found in Appendices O and Q. 2.2. How, where and when to assess Projects invariably utilise a WBS [45] to organise activities prior to cost estimation and scheduling activities. The assessment considers each Complexity Theme against the five Complexity Criteria at each node of the WBS on a particular level. Scores are assigned (very low, low, medium, high and very high) at each intersection of the matrix. As the WBS increases in detail so does the effort required to undertake the assessment. It may be justifiable to limit the assessment to relevant portions of the WBS or choose a high- level within the WBS to restrict the nodes to be considered. The complexity assessments should be undertaken against the portion of the WBS that describes the technical development aspects. The WBS, including that describing technical development, will be subject to refinement over time. As such any early assessment will be at a high-level on the WBS and necessarily at a corresponding low-level of overall detail. There will be business-level requirements at the offset and other factors, such as generic organisational processes and the existing regulatory landscape. Additionally, early planning may make assumptions with regards to the organisation and contracting mechanisms. This pre- requirements early assessment will identify areas of potential concern to enable early planning through the identification of high-level CSFs. As the project and the development progresses the WBS will be defined level by level with early development phases likely to receive the most detail. The example of a WBS shown in Figure 8 has technical development activities below several of the high-level nodes. In this example requirements definition would receive more attention than test and verification activities due to the dependencies between the two and the inherent difficulty in defining later activities because of this. Complexity assessments would be undertaken as a minimum at the end of development phases and would serve to inform the planning of subsequent phases and as an important input into Stage Gate type governance reviews. Long project phases may demand interim complexity assessments to catch any significant changes during the phase. Assessments would develop the earlier higher-level reviews to progressively lower levels of the WBS to identify hotspots of complexity within the development. This technique affords the opportunity to Figure 8. Sample WBS for an unmanned aerial vehicle (UAV) [46].
- 19. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 15 of 69 disregard areas of low complexity and allow expenditure of effort where it needed. For the purposes of this example WBS ‘control software’ and ‘verification software’ have been identified as having high levels of complexity. This could prompt the assessment of the next level of the WBS below to determine the particular node that is of particular concern. This can be repeated as required and, in the most extreme application, until the lowest level of the WBS has been reached. 2.3. Outputs from complexity assessment Identification of complexity should be used inform development planning, enabling the establishment of conducive project environment. An established method is that of CSFs as environmental influences that will enhance the likelihood of success. They may reduce either the probability or impact of the consequences of complexity. Both the rationale and the proposed outcome of the putting in place of CSFs should be recorded. This exercise should also prompt the identification of development risks. Some of these will be residual risks as a result of putting in place of CSFs. The putting in place of CSFs will influence the complexity profile. An example of this, developed later within the case study in Section 8, is that of the development of Boeing’s Dreamliner. Radical changes in Boeing’s procurement strategy were implemented at the very early stage of the project to improve schedule and costs. This changed the profile of complexity within the project’s supply chain, introducing ‘’coordination risks’, with well-publicised results [47]. Using the complexity themes, it may be that complexity within the internal organisation are merely transferred into contractual management. It is therefore important that the complexity assessment is repeated in areas where changes are proposed. Indeed, in the example of AREVA’s design and construction of the Olkiluoto 3 nuclear plant the issues were entirely the reverse of the Dreamliner and related with AREVA choosing to embark on a first of a kind project alone without the necessary experience as ‘architect and engineer’,’ without experienced partners’ and without having all the necessary competences [48]. Repeating the complexity assessment will identify any further areas of complexity that have resulted from the implementation of CSFs and allow the impact of changes to strategy to be understood. Additional CSFs can then be put in place or existing CSFs amended as appropriate. The designation of CSFs is discussed further in Section 3. 2.4. Complexity profile and the interaction of complexity criteria Complexity is influenced by many factors and is not a static property. It will evolve over the development lifecycle and its nature may also be transformed by imposed changes. It can be theorised that Complexity Criteria will interact in specified ways and will follow individual general profiles. Each criterion applies equally to modelling of the development process through planning and scheduling as it does to the definition of system elements and their interactions. Uncertainty relates to the likelihood of change of development requirements and it will be the greatest at the beginning when definition of requirements and scope is at its lowest. Naturally as definition increases and the number of assumptions are reduced, uncertainty is also reduced. Uncertainty can relate to changes prompted from sources such as stakeholders, regulators or rework due to iteration or errors. Uncertainty should reduce as the system is verified, and should diminish to its very lowest level as the system approaches final validation. Changes, especially those later in the development, will reintroduce uncertainty. Ambiguity is closely related to uncertainty and relates to the amount of information that is unknown. In the absence of verified information, it is necessary that assumptions are made to enable aspects of the development to progress. Without assumptions other activities will be delayed. Ambiguity will reduce as requirements are Figure 10. The changing nature of the project process [25].
- 20. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 16 of 69 defined but as with uncertainty it will be increased with the introduction of changes. Early definition activities will reduce both uncertainty and ambiguity together. Two methods of detail with relatively low levels of uncertainty and ambiguity include the inclusion of ‘float’ [49] within development schedules (sometimes known as slack) and ‘contingency’ [50] within the cost plan or the inclusion of flexibility within the design to accommodate a range of design definition outcomes [51]. Emergence, as defined in Section 1.5, will increase throughout the development lifecycle as ambiguity is reduced. This property is invoked by the interaction of requirements or by the introduction of change in ways that cannot be anticipated. It specifically relates to the impact and the likelihood of emergent behaviours arising. The impact of emergence can be managed through the decoupling of development activities or system elements. Examples of managing emergence are that of removing activities with likely emergent behaviours from the critical path of a development schedule or controlling the interfaces between sub-systems through choice of system architecture. Non-linearity also increases as ambiguity is reduced. The impact of non-linearity can be managed by identifying such interfaces and introducing extra capacity in the affected system elements to manage potential future change. This should be undertaken on the basis of risk and the extra capacity can be either built into the changed requirement to prevent the change being propagated or those affected by the non-linear behaviours to absorb the change with minimal impact. It is highly desirable to reduce uncertainty and generally control the incidences of change as non- linearity and emergence increase. Considering this from the point of view of requirements and scope it can be reasoned that Program-size Complexity of the development will begin and end low. Requirements will start as those necessary for the business or operational need and will be relatively few and at a low level of detail. Requirement Program- size complexity will rise as requirements are defined, Conceptofoperations High-levelrequirements Detailedrequired High-leveldesign Detaileddesign Implementation Integrationandtesting Sub-systemverification Systemverification Operations&maintenance Figure 11. Complexity Profiles over the development lifecycle with no major changes and an incidence of major change during sub-system verification (bottom). Point of major change Figure 9. Level of organisational effort over time [52].
- 21. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 17 of 69 increasing in both number and detail. Conversely this property will reduce as requirements are verified until the business and operational level requirements are finally validated. Scope is similarly represented via the WBS and activity scheduling. Again these begin at a low-level of detail until the development is fully defined and planned. As activities (scope) is completed the Program-size Complexity again reduces as the number of activities to complete and their interactions reduces. This is broadly in line with organisational effort [56] as can be seen in Figure 9. Changes will again increase Program-size Complexity with the example of previously verified requirements needing to be reanalysed and adapted to accommodate a late change. Program-size Complexity is managed through the application of organisational or computing resource and/or modelling techniques. The magnitude and impact of Program-size Complexity over the design definition phases is shown in Figure 10 [25]. Pich et al propose that not only does the complexity of information increase over time but that its relative impact of information content on outcomes diminishes over time. This assertion chimes with that of INCOSE who propose that approximately 70% of committed lifecycle costs are due to concept design against 95% [28]. Figure 11 suggest how the profile of a typical technical development may be represented over its lifecycle with an indicative measure of the magnitude of the particular complexity criteria against time with the development phase also shown. The first graph shows no major changes while the second graph shows a major change that is brought about during sub-system verification. This is manifested by an increase in program-line complexity, uncertainty and ambiguity. Development phasing has been taken from INCOSE as per Figure 4 on page 5. Such late changes present issues are there are now additional activities to manage and potentially a large number of system elements, previously verified, need either to be completely re- evaluated or re-verified against the new requirements. Uncertainty and ambiguity are re-introduced into system elements that had previously been defined and built or constructed. The impact of non-linearity and emergence is high due to the timing of the change and the number and nature of system dependencies. 3. Critical Project Success factors and their application in system development Critical Success Factors were first proposed by D. Ronald Daniel in the 1960’s, after which they were further developed by John F. Rockart of the Sloan School of Management [5]. Their application was primarily intended within the areas of business strategy but have also been adopted in project management as success factors [45][53]. It is the intention that the selection and implementation of CSFs should put in place an environment more conducive with a successful outcome. 3.1. Selection of success factors from literature Two mistakes that should be avoided are those of confusing a CSF with a requirement and assigning the CSF as too high a level. The latter make the CSF too general for useful application. Requirements are distinctly different from CSF and though their satisfaction is as at least as important, their management is handled elsewhere. Examples include particularly important stakeholder requirements being associated with ‘Measures of Effectiveness’ (MoE) and system requirements with ‘Key Performance Parameters’ (TPP) of the system of interest [54]. These will be discussed more fully in a later section but as performance measurement metrics. A CSF that is described at too high-level across all activities can be viewed as pre-requisites for any technical development, project or business endeavour. Examples of high-level CSFs are provided by the APM Body of Knowledge [45]: Defining clear goals and objectives; Maintaining a focus on business value; Implementing a proper governance structure; Ensuring senior management commitment; Providing timely and clear communication.
- 22. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 18 of 69 While these obviously worthy objectives, they apply to every project that the author has worked on. Furthermore, they give no indication of how they might be judged as being satisfied. It is therefore important that the CSFs do not merely represent best practice but apply to the demands of the particular technical development as indicated by complexity assessment. As such, the CSFs selected for inclusion in the framework will need to be supplemented with enough detail for implementation and, if possible, measurement. I have selected 141 CSFs for inclusion in the framework from a variety of sources, including APM [7], Pinto and Slevin [52], Dvir et al [55], Chow and Cai [56], Ragatz et al [57], Koutsikouri and Dainty [58] and Fortune and White [59. These are in turn divided amongst the Complexity Themes. Some of these CSFs are repeated in more than one Complexity Theme. Ultimately each will be need to be developed with the appropriate detail if chosen in practice and adopted for use. For example, ‘support from senior management’ should include whose support should be in place and when and what this support will achieve. It should be targeted against particular activities and should recognise the influence that development phasing may have on the nature of management support that is required. This who, what, how and why approach can be applied to most items on the list. The full list of CSFs for the framework has been grouped into the applicable Complexity Themes and can be found in Appendix C. There are a number of important concepts that should be borne in mind when applying CSFs. Merely stating development best practice will dilute the effectiveness of the identification of CSFs and instead only particularly areas of the process should be subject to analysis. As such general process improvement should be addressed elsewhere. In the spirit of the incorporation of the principles of the Demming cycle, and the resulting continual improvement, the proposed and the actual effect of the CSF should be recorded. This will enable the effectiveness of individual CSFs and allow them to be catalogued within a toolbox of CSFs for future use. To facilitate in the analysis of the effectiveness of CSF they should have performance measure associated with them where possible. 3.2. Verification of success factors through questionnaire Potential CSFs identified in the literature were evaluated for their potential utility in the framework via a questionnaire. The questionnaire used the criterion of perceived impact on the likelihood of project success. The questionnaire was posted online using the SurveyMonkey application [60] and a total of 122 responses were achieved by requests via email and Linkedin social media. An overview showing the source of replies can be seen in Appendix D. Respondents were a varied mixture of engineering and project management professionals. Respondents were asked to rank the CSFs listed in Appendix C according to their potential influence on a project into five categories from very low to very high. Any CSFs not considered relevant or without an impact were removed from the list as being non-applicable. The questionnaire also gave respondents the opportunity to identify other CSFs that the literature survey and subsequent analysis had missed. The questionnaire and explanatory text is contained with Appendix E. The unprocessed results, per respondent, are contained in Appendix R. Processing of the CSFs was undertaken and they are categorised in Appendix F according to their scores: 4.0 and above - shaded in green (high to very high influence); 3.5 and above - shaded in yellow (upper scoring of medium to high influence); 3.0 and above - shaded in orange (low scoring of medium to high influence); Below 3.0 - no shading and red type (low to medium influence). These categories could then be filtered, sorted and compared across age, role description and industry. Further processing was then undertaken to aggregate relevant age and role group. Data was disregarded where this was not possible for particular role and industry groups. This ensured that data groups were statistically significant, allowing the data from the selected groups to be meaningfully compared to the full dataset. Information relating to the size of the aggregated datasets can be found in Appendix G.
- 23. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 19 of 69 3.2.1. Overview The findings were analysed across the full dataset and then these results compared to the various categories sorted by age, role description and industry. In this way trends within the sub-groups could be identified. It may be surmised that CSFs and performance measures for different industries will be different and as such complexity managed differently across different domains. Also, the influence of age and role description on CSFs and performance measures may skew complexity management due to the biases inherent with them. Examples being project management professionals placing greater importance on established project management techniques such as Earned Value Management while engineering personnel may place a greater emphasis on the management of requirements. Identification of such biases will aid in the composition of teams determining how complexity management will be undertaken. Known biases can then be predicted and defended against. Comparisons were made between the filtered data and the full dataset. This was done for the high to very high (shaded green) CSFs and then for the top three CSFs in each category. 83% of the respondents were from the age categories of ‘35 to 44’ (25%), ‘45 to 54’ (27%) and ‘55 to 64’ (31%). The results were consolidated from seven into five age categories. Consolidation of results outside of the popular categories formed ‘up to 35’ and ‘over 65’ categories. This ensured sample sizes were statistically significant. ‘Up to 35’ was the smallest data group with just below 7% of responses. Role descriptions were similarly consolidated to ensure that samples for each category were sufficiently sized. The data from 12 respondents was disregarded as not fitting with any one of the three aggregated high-level role description that were chosen, reducing the dataset to 110. The groups derived from across full dataset were ‘senior personnel’ (41%), ‘project personnel’ (40%) and ‘engineering personnel’ (19%). Industry categories with the largest sample sizes were considered for analysis. Any groups below 10 in number were disregarded resulting in a total of 83 respondents. Those chosen based on sample size were ‘construction’ (20%), ‘decommissioning’ (16%), ‘defence’ (16%), ‘energy generation’ (18%), ‘oil and gas’ (12%) and ‘professional services’ (18%). Obviously there is some overlap between these industries and particularly between ‘construction’ and ‘professional services’, which are generic terms that may relate to undertaking of particular activities across most industries. The chosen data group sizes were generally of similar in size, with ‘oil and gas’ being the smallest with 10 responses. Further work, through a larger sample size, could be undertaken to increase the number of industries that can be analysed. This would also improve the general confidence in the findings across age, role description and industry. Furthermore, biases could be more completely determined across individual role descriptions rather than consolidated into the groups that were chosen. 3.2.2. Full dataset The summarised results, as discussed here, are contained within Appendix F. This section will discuss the top three CSFs for each complexity theme and the trends that were identified by the author from this data. The analysis was across the data from all 122 respondents. Imposed project constraints Clear realistic project objectives with an average (mean) score of 4.51 and 68 of 118 (58%) rating it very high; Adequate budget with an average (mean) score of 4.39 and 62 of 120 (52%) rating it very high; Composition of project team in terms of experience and capability with an average (mean) score of 4.38 and 57 of 120 (48%) rating it very high. In general planning and general business and governance processes appeared a lot less important than a good organisation, certain project management processes (change and risk management) and the overall viability of the development. Strong business case/sound basis for project was notable for having the second- highest number of very high ratings but equal tenth-highest number of low ratings amongst the 4.0 and above CSFs. This gave it a relatively lowly seventh most influential overall. This demonstrates how the perceived
- 24. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 20 of 69 importance of a business varies amongst respondents. Project management respondent’s generally assigned this very high-importance, in line with the teachings of project management methodologies, while other groups gave it a much lower score. Planning related CSFs were ranked as a high ‘medium to high’ influence or a low ‘high to very high’ influence which was unexpected considering the relative importance of planning within project and engineering management. A project cancellation process was assigned a ‘low to medium’ influence so should be discounted from the list of CSFs. Evidently the cancellation of an ill-conceived or no-longer-required project was considered either not relevant or of a low overall influence. None of the comments provided CSFs not already included here or within another Complexity Theme’. Technical development processes Critical activities are identified with an average (mean) score of 4.47 and 63 of 115 (55%) rating it very high; Clear realistic development objectives with an average (mean) score of 4.32 and 53 of 116 (46%) rating it very high; A well understood and mature design review process is in place with an average (mean) score of 4.27 and 56 of 118 (47%) rating it very high. Unsurprisingly the general trends of the first Complexity Theme were evident in this one. Identification of areas of risk and criticality ranked higher than general planning. Common Systems Engineering techniques such as ‘test early, test often’, ‘modelling and prototyping’ and ‘standardisation and/or modularisation’ were assigned a relatively low importance overall. Examples of additional comments include the use of ‘state of the art’ development processes and ‘clearly defined and mature functional requirements’. Organisational Good leadership with an average (mean) score of 4.62 and 72 of 115 (63%) rating it very high; Transparent definition of responsibilities with an average (mean) score of 4.26 and 51 of 115 (44%) rating it very high; Degree of collaboration with an average (mean) score of 4.26 and 47 of 114 (41%) rating it very high. Leadership was by some way the most important influence. Co-location of teams was not thought to be important which may be reflected by the global nature of modern projects and the availability of better technologies to allow collaboration. Surprisingly ‘competence in technology and technology management’ and ‘domain-specific know-how’ were not considered as ‘high to very high’ influences. Contractual management Clearly understood contractual interfaces with an average (mean) score of 4.45 and 64 of 114 (56%) rating it very high; Good performance by suppliers/contractors/consultants with an average (mean) score of 4.43 and 58 of 114 (51%) rating it very high; Effective monitoring/control with an average (mean) score of 4.38 and 54 of 114 (47%) rating it very high. The ranking of influences did not show any unexpected trends which were in essence the principles of selecting a competent supplier, ensuring the contract was clearly understood and managing it well. Stakeholder management Client/user acceptance with an average (mean) score of 4.58 and 74 of 113 (65%) rating it very high; Decisions are agreed and documented with an average (mean) score of 4.48 and 64 of 113 (57%) rating it very high; User/client involvement with an average (mean) score of 4.44 and 64 of 113 (57%) rating it very high.
- 25. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 21 of 69 Stakeholder management reinforced the need for early and effective communication. The representation of the full system lifecycle was not thought to be particularly important though the involvement of operators was considered as a relatively low in ‘high to very high’ category. The conclusion here is that the eventual decommissioning of a system is often of low concern during its design and implementation. A respondent’s comment that may be worthy of further investigation is that of ‘no informal communications under any circumstances and record all meetings’. The use of formal versus informal communication is an area that is rarely discussed. Informal communication infers frequent and relaxed use of the mediums of telephone calls and email, which suggests trust and collaboration. The inverse is contractually driven, with an administrative burden and an impact on factors such as frequency and content. External interface management Clear communication is established with an average (mean) score of 4.41 and 56 of 112 (50%) rating it very high; Clearly identified and understood external interfaces with an average (mean) score of 4.33 and 55 of 113 (49%) rating it very high; Defined process for managing external interfaces with an average (mean) score of 3.96 and 29 of 115 (26%) rating it very high. This theme mirrored the need for structured and effective communication as discussed in the previous theme. Only two of the six were rated above 4.0. Regulatory interface management Clearly identified and understood interfaces with an average (mean) score of 4.45 and 59 of 111 (53%) rating it very high; Clear lines of communication with regulators with an average (mean) score of 4.36 and 52 of 110 (47%) rating it very high; Good relationship with regulators with an average (mean) score of 4.28 and 48 of 111 (43%) rating it very high. All categories within this theme were assigned a relatively high score and even more so than other external interfaces. This is obviously down to the criticality of the regulator’s role in many projects. Technology development management Well-defined standards up front with an average (mean) score of 4.04 and 31 of 111 (28%) rating it very high; Proven/familiar technology with an average (mean) score of 4.01 and 38 of 111 (34%) rating it very high; Pursuing a simple as possible design with an average (mean) score of 3.98 and 38 of 111 (34%) rating it very high; 3.98 and 38 of 111 rating it very high. The conclusion when considering the highest scoring CSFs within this theme was the avoidance of unnecessary complexity while using established and thoroughly documented design standards. This is often not possible and many projects have elements of research and development within them. These were generally rated a lot lower than the other themes with approximately half the number of very highs assigned. System integration management Well-defined standards up front with an average (mean) score of 4.17 and 37 of 112 (33%) rating it very high; Pursuing a simple as possible design with an average (mean) score of 4.01 and 36 of 113 (32%) rating it very high; Proven/familiar technology with an average (mean) score of 3.98 and 30 of 111 (27%) rating it very high.
- 26. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 22 of 69 The scoring was similar to that within technology management. Only two of the eight were rated above 4.0. Again the numbers of very highs assigned were around half that as other themes. 3.2.3. By age Results filtered by age can be seen in Appendix H. Sample sizes per any age group were at best a third the size of the whole dataset. Instead of analysing absolute ratings the top rated CSFs for each theme were compared against the full dataset for consistency. Imposed project constraints - Up to 35 generally scored lower with a lot fewer ‘high to very high’ category CSF. This could be an anomaly as a consequence of the relatively low sample size. ‘35 to 45’ were representative of the overall rankings. Age ‘55 to 64’ and ‘above 64’ advocated ‘strong project sponsor/champion’ and ‘competent and qualified project manager’ respectively. Both these age groups rated ‘strong business case/sound basis for project’ which perhaps reflects their position and seniority within their respective organisations, introducing a related bias. Technical development processes - The ‘below 35 age group assigned more importance to planning in areas of criticality and the use of past experience, while the ‘above 64’ age group gave more importance to planning and responsiveness. Other age groups were broadly representative of the trends within the full dataset. Organisational - Findings across all age groups were closely in agreement with the full dataset and especially near the top ranked CSFs. Contractual management - Findings across all age groups were closely in agreement with the full dataset and especially near the top ranked CSFs though the position of ‘rigorous pre-qualification process’ varied greatly across age ranges. Stakeholder management - Findings across all age groups were closely in agreement with the full dataset though the ’35 to 44’ age group assigned a lot greater importance to the managing of expectations and the representation of operators. External interface management - Findings across all age groups were closely in agreement with the full dataset Regulatory interface management - Findings across all age groups were closely in agreement with the full dataset Technology development management - Findings across all age groups were closely in agreement with the full dataset though the ‘up to 35’ age group assigned high importance to ‘continuous improvement process for products’. System integration management - This theme was very similar to technology and was again closely in agreement with the full dataset with the ‘up to 35’ age group assigning high importance to ‘continuous improvement process for products’. 3.2.4. By role Results filtered by role can be seen in Appendix I. Similarly, to by age these sub-datasets were a fraction of the full dataset at less than half the size. The sub-data sets were compared against the full dataset results. Imposed project constraints - The findings were closely in agreement with the full dataset across the role descriptions with the exception of an understandable bias towards a single CSF in each role. Upper management advocated a ‘strong business case/sound basis for project’. Project personnel ‘ranked most highly the use of a ‘competent and qualified project manager. ‘Support from senior management’ was the third most important CSF for engineering personnel which perhaps indicates its importance gained from previous experience. Technical development processes - Findings across all role descriptions were closely in agreement with the full dataset across senior management and project personnel but perhaps understandably engineering
- 27. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 23 of 69 personnel had a very different perspective. The following CSFs, were not present in the other role descriptions. The first two of these were second and third most important while the others descended in relative influence within the ‘high to very high’ category. Enhanced planning is applied against areas of criticality and uncertainty; Fast transfer of information; Test early, test often philosophy is used during development; Past experience of management methodologies and tools is available; Correct choice of management methodologies and tools; Trouble shooting mechanisms in place. Organisational - Findings across all role descriptions were closely in agreement with the full dataset with the exception of the inclusion of’ composition of development team in terms of experience and capability’ within engineering personnel. Contractual management, Stakeholder management, External interface management and Regulatory interface management - Findings across all these themes and across all role descriptions were closely in agreement with the full dataset Technology development management - Findings across all age groups were closely in agreement with the full dataset though engineering personnel rated the CSFs general lower with only ‘test early, test often philosophy’ achieving the highest ranking. System integration management - Findings across all age groups were closely in agreement with the full dataset though engineering personnel rated the ‘modelling and prototyping’ higher than the other role descriptions in the top three. 3.2.5. By industry Results filtered by industry can be seen in Appendix J. Sub-datasets were again compared. Imposed project constraints - The findings were closely in agreement with the full dataset across the role descriptions with a few notable exceptions within oil and gas. Perhaps as a reflection of an environment reliant on oil prices and commercial pressures ‘effective change management’ and ‘political stability’ ranked relatively highly. Technical development processes - Construction differed significantly with the following CSFs figuring in the rankings as ‘high to very high’, with responsiveness being the third most important. This reflects the dynamic nature of a construction environment and requirement to act quickly during the construction phase to minimise or avoid rework. Responsive and flexible process to meet client needs; Strong, appropriately detailed and realistic development plan kept up to date; Appropriate development planning technique has been chosen; Correct choice of management methodologies and tools. Organisational - Energy generation elevated the ‘re-use knowledge and experience from previous projects’ which is a strong component of the nuclear following events such as Three Mile Island and Chernobyl [66][67]. Oil and gas elevated the ‘collocation of teams’, perhaps due to the international nature of many of its projects. Professional services included ‘domain-specific know-how’ and ‘appropriate techniques to aid identification of organisational dependencies and interfaces’. Otherwise there was commonality between the industries and most influential CSFs. Contractual management - With some minor differences findings across industries were closely in agreement with the full dataset. Stakeholder management - With some minor differences findings across industries were closely in agreement with the full dataset.
- 28. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 24 of 69 External interface management and Regulatory interface management - Findings across all industries were closely in agreement with the full dataset. Technology development management - Energy generation attributed some Systems Engineering type CSFs which reflects the complex nature of many projects in this domain. These included ‘modelling and prototyping’ and the use of ‘system element maturity’. ‘Continuous improvement process for products’ was also ranked third, well above any other industry. System integration management - Findings across all age groups were closely in agreement with the full dataset though energy generation rated the ‘system element maturity’ highly again. 3.2.6. Conclusion Analysing the datasets by age, role description and industry does not yield any particular trends nor show any of the sub-datasets deviate significantly from the findings over the full dataset. The following are very high level observations: The age groups ’55 to 64’ and ‘above 64’ show two thirds of all deviations from the overall sub- dataset; Particular biases were evident in the Complexity Theme ‘imposed project constraints’ that relate to the ‘role description’ dataset; The role description ‘engineering personnel’ shows half of all deviations from the overall sub-dataset; The industries of ‘energy generation’ and ‘oil and gas’ show two thirds of all deviations from the overall sub-dataset; The Complexity Themes of ‘Imposed project constraints’ and ‘Technical development processes’ tended to deviate more across all sub-datasets than any the other themes. From this is could be seen that particular biases should be guarded against when determining CSFs. A workshop approach should balance age groups and role descriptions to ensure particular pertinent CSFs are not overlooked. Furthermore, particular industries may place more emphasis on some CSFs over others. The formation of a ranked list of CSFs will need to be tailored for its application. It would be advised that adoption of the CSFs begin from either this list or, for a particular industry, from the nearest sub-data set as filtered by the closest applicable industry type. Ranking of influence of CSFs could be further refined through subsequent iterations of the Complexity Management process. Stakeholders could be consulted before and after application. 4. System development planning techniques 4.1. Overview An important aspect of the framework is that it will support and enhance planning activities. It has been recognised that planning related objectives cannot be achieved in isolation of established planning tools and techniques. Complexity influenced planning will need to be undertaken in parallel with current practice and indeed maximum benefits can only be realised if they can provide readily used and value-adding inputs into these established planning activities. This overview discusses the shortfalls and benefits of current practice, the opportunities in adopting a suitable supplementary technique and the requirements that the technique must fulfil. A suitable technique called Design Structure Matrix, also commonly known as DSM, will then be discussed along with ways it can be integrated into current practice. Gantt charts [63] are the dominant method of project and system development planning. The inherent advantage over other techniques is their relative simplicity, which allows it to be an effective communication tool. The use of Gantt charts also underpins project monitoring and reporting techniques such as Earned Value Management (EVM) [45]53]. This popularity has precipitated the development of advanced tooling, with packages such as Primavera, providing a way of creating the plan and integrating it with other processes. Overall they are a good way of managing the overall duration of a project.
- 29. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 25 of 69 There are however limitations in the techniques’ use that are increasingly exposed as complexity increases. Although relationships between activities are represented they are restricted to simple dependencies [64]. These dependencies describe how an activity influences the start (and occasionally the finish) of other activities including the use of lag. The nature of these dependencies cannot be further explored using this planning technique alone. Thus the relationship between coupled activities is much simplified and likely to be misunderstood without further analysis. As complexity and the number of dependencies increase, the effort required in the production and management of the Gantt chart increases significantly. This leads to more and more assumptions and simplifications of the nature of dependencies which can often result in the actual start and duration of activities progressively deviating from the plan as it proceeds. This is obviously a state of affairs to minimise if not to avoid. It is unlikely that the dominance of Gantt charts in industry as the primary method of planning will change anytime soon. Moreover, they have served as an incredibly useful way of coordinating and representing activities across diverse disciplines and functions. With this in mind, other techniques should supplement or feed into, rather than replace, Gantt charts, and should provide a way of enhancing the planning where required. Planning of all of the development activities across a project is a considerable undertaking which only increases as complexity increases. It may be surmised that there is a positive relationship between the effort that is required for effective planning and the effect of the number and potential impact of dependencies. Other factors that deserve consideration within planning include the varying uncertainty and number of ambiguity related assumptions. There is also an inverse relationship between the effort expended and the risk of deviation from the plan. The balance between effort and risk is an important consideration in any project where there are finite resources available. Such detailed planning has a high overhead and in a complex project unfocussed planning can diffuse efforts leading to essential dependencies being overlooked until it is too late. The direction of maximum effort to areas of risks should improve the likelihood of an overall successful outcome. Further to the earlier discussions within a survey of literature [4] and within the objectives within Section 1.2, it would be sensible to use any enhanced planning techniques in areas identified as being either particularly complex, or moderately complex but with a high potential impact in the event of deviation from the plan. Using the principles of the Demming cycle, it is also beneficial to understand the impact of the use of additional planning techniques to allow their use to be developed and improved for later planning of development activities or future technical development. Lastly there may be opportunities to reuse particular activity patterns or structures to reduce future planning effort that needs to be expended. To summarise it is proposed that detailed development activity planning follows these concepts: Will prioritise enhanced planning in areas identified through complexity assessment; Determine where and which planning techniques to be used should be influenced by choice of CSF; Uses enhanced planning to supplement rather than replace traditional project planning techniques; Keeps a record of where system development planning has influenced the overall planning; Records and uses learning to assess effectiveness of techniques and improve subsequent planning; Considers opportunities to develop frameworks findings for re-use in subsequent projects. 4.2. Design Structure Matrix 4.2.1. Introduction Design Structure Matrix (DSM) has been chosen to supplement traditional planning methods due to its high functionality and flexibility, allowing it to be used over several domains of planning. DSM is based on a simple graphical representation of the relationship between activities, called the square N x N matrix [65], and has its origins in Graph Theory. It is also variously known as the N2 diagram or coupling matrix [28]. It was developed by several independent parties but most notably by Eppinger and Browning who have refined the
- 30. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 26 of 69 technique, independently and in collaboration, through a number of papers and publications since the 1990s [66]. The nature of DSM’s notation conventions varies and should be designed to suit the particular application. For this reason, detailed descriptions will be omitted. For the purpose of this project the DSM as developed by Eppinger and Browning, will form the basis of future discussions. The IR/FAD convention (as opposed to the IC/FBD convention) will be adopted. Both variants are broadly similar but the use of rows and columns is transposed [67]. Full and detailed descriptions of the technique can be found within their recent publication, ‘Design Structure Matrix Methods and Applications’ [67]. They are also associated with DSMweb.org, which aims to ‘promote and foster’ further development of DSM [68]. DSM is flexible enough to cater for any type of system and can be further adapted for a particular process and organisation. Furthermore, it is ‘highly compact, easily scalable’ [67] and can be applied where needed at the level of detail required. The low level of abstraction, intuitive and graphical representation and the relative density of information that a DSM contains makes it ideal for conveying important information as both a standalone model or as a precursor for further analysis. It can provide system-level views of areas of heavy interaction [67] but for the purposes of this project it is also ideally suited for focussed views on particular aspects of the development. 4.2.2. Design Structure Matrix principles DSM is a highly flexible technique that can be applied to provide a view of aspect of a technical development either independently or in combination with another aspect, and can be summarised as follows: Product architecture, i.e. the interactions of the system of interest itself; Organisation architecture, i.e. how the development organisation interacts; Process architecture which describes the development process ranging from high-level to detailed planning; Multi-domain Matrix, also known as MDM, which combines several of the above to show interactions between the elements of the various matrices. Process can be described as having ‘temporal flow’ architecture due to the inherent time-based interactions that occur, with product and organisation possessing ‘static architecture’ [67]. MDM will be either temporal flow or static, depending on its component DSMs. We will primarily be interested in DSMs associated with process and organisational architecture and process/organisational MDMs as a tool to aid in planning of activities. This does not however preclude the use of product DSMs or the inclusion of product architecture in a MDM if the complexity assessment highlights an area of concern. The basic principles of the DSM are very simple are as follows: • Matrix elements being considered are represented along the diagonal of the matrix from upper left to bottom right; • Element names shown on rows and columns with the ordering of elements kept consistent on both the rows and columns; • Inputs are from left and right of the diagonal (rows) and outputs (columns from above and below as per IR/FAD convention; • Interaction between elements is shown in the matrix cells with the diagonal cells being blank. A simple DSM, which merely acknowledges the existence of a relationship between inputs and outputs, is called a binary DSM. The DSM however can be further developed into a numerical DSM, including attributes such as importance, number of interactions and impact or strength of interaction. Additional attributes can be linked to cells if required and elsewhere stored in a database. Whereas binary DSMs are qualitative, numerical DSMs can be designed to be highly quantitative. The process for creating and managing a DSM is relatively straightforward and follows a five step process. Before this is begun all conventions will need to be agreed including supplementary information to be
- 31. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 27 of 69 contained within numerical DSM models. Suggested additional data to be collected includes interaction strength as a single integer or as a combination of probability and impact. Later changes to notation and the scope of information to be collected and model will be difficult to reconcile with previously completed DSMs, which will potentially lead to confusion and duplicated effort. The inherent limitation of the DSM relates to it describing the activities in terms ‘edges’ rather than ‘nodes’ [67]. For example, a binary process architecture DSM cannot describe the duration of an activity. This can be overcome through the development of a numerical DSM that includes information such as duration or resource. The following sections will outline the essence of the DSM technique. A survey of literature relating to DSM, describing the technique in more detail, can be found in Appendix K. 4.2.3. Creating and applying the organisational architecture DSM The steps in the process for creating the Organisational DSM are as follows: Decompose; Identify; Analyse; Display; Improve. Once the DSM is clustered it can be used to inform management decisions with regards to the composition of teams, geographical location and the types and scope of communication. The integration of development activities can be extremely challenging. It can be used to shape the formation of teams or clustering of teams based on concentrations of interactions. The application of collaborative tools, such as databases, will be more appropriate for large groups while meetings and informal methods are more effective for smaller groups. Lastly the co-location of particular organisational elements may be beneficial. 4.2.4. Creating and applying process architecture DSM Process DSMs are created using the same five steps as are used for organisational DSM. The level of decomposition will be dictated by the previous complexity assessment and assignment of CSFs. The level decomposition will also be influenced by availability information during the particular phase of development. It is advisable that the elements are kept at a consistent level within the respective breakdown structures. If a model of activities already exists this should be used and used as the planning baseline. Key activities, pivotal to delivery of the entire process, should be identified. These are chosen based on dependencies, coupling and risk. As such disproportionate effort should be placed on ensuring that these are completed as per the plan over less consequential activities. These will be tracked throughout the project with evidence of slippage or increases in risks above the accepted residual risk prompting additional interventions. Analysis of DSMs should enable a better understanding of the interactions between elements and allow the plan to be amended accordingly. There are several relationship types within DSMs: Sequential – where activities follow on from each other in a finish to start type dependency relationship. There may be some overlap may be possible between activities which would be seen as negative lag on a Gantt chart; Parallel activities – these activities may rely on same resource but there is actual no dependency. Resource constraints would normally be considered later using organisational architecture DSM or on a resource loaded Gantt chart schedule; Coupled – with iterations between the outputs and inputs between two or more activities. This is the most difficult relationship to represent in traditional Gantt charts; Conditional – execution of later activity dependant on decision made in an earlier activity. This may or may not be sequential and in confined to the transfer of information.
- 32. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 28 of 69 The traditional interaction considered within planning is that of a finish to start dependency where one activity needs to be completed before another can commence, known as sequential within a DSM. This can be further refined by the inclusion of positive or negative lag to either delay or advance the commencement of the following activity by a specified duration. Other variations exist, including start to start (which suggests parallel activities) or finish to finish (which suggests a coupling type relationship). It is these relationships that define the effectiveness of a process or plan and it follows that the greatest impact is achieved by better managing the interfaces between activities. It will also be worth considering representing iterations as separate activities Of the relationship types it is coupling that is very often the most troublesome to manage and commonly causes delays. These relationships most often rely on iterative outputs from one activity into one or more other activities. It is difficult to predict the number of iterations that may be ultimately required. Other variables include the duration to review and accept an iterative input into another coupled activity and difficulties in planning the resource necessary for such interactions. The more activities involved in coupling, the more closely coupled activities are and the number of anticipated iterative cycles all potentially impact the overall duration. There are several types of coupling. Some of these can be eliminated or reduced in terms of their impact through effective planning and/or management of process. This is an example of a process behaviour that should be fed back through the process for CSF consideration. Coupling behaviours may be planned or, due to errors, unidentified interaction type or emergence, be unplanned. Coupling is generally as follows: 1. Inherent coupling – planned coupling behaviour with activities that are structurally interdependent; 2. Poor activity sequencing – information is created too late resulting in the delay of other activities. Though a planned coupling this type of behaviour can be minimised, though not entirely eliminated, through effective analysis; 3. Incomplete activities – where activities are unduly delayed with a similar impact as poor activity sequencing; 4. Poor communication – information or outputs are not passed on completely or in timely fashion again delaying subsequent activities; 5. Input changes – caused by changes to assumptions; 6. Mistakes – defective inputs created and discovered at a later date. Evidently coupling types 2 to 6 are to be avoided whether through planning or subsequent controlling of the plan. Delayed inputs/outputs can result in the formation of assumptions and indeed this is sometimes adjudged as being a desirable response to a coupling behaviour. Adopting such an action simply exchanges coupling behaviour behaviours 2, 3 or 4 for the potential for behaviour 5. Inherent coupling suggests necessary iterations of outputs/inputs between two or more activities. Though necessary, most often even desirable, convergence to a solution should be encouraged as quickly as is practicable. Additionally, the reduction in the number of feedback loops is highly desirable also. Indeed, this has the potential for use as a measure of development status and a measure of complexity in itself. One goal of the analysis of a process architecture DSM is to optimise sequencing of the activities so that as many interactions as possible are below the diagonal. Doing so ensures that activities only begin once all inputs are available and reduces the number of assumptions that need to be made (avoiding the potential Figure 12. Process architecture DSM [68].
- 33. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 29 of 69 for input changes type coupling) or activities being delayed unduly (avoiding poor activity sequencing type coupling). In the extreme example an output from an activity in the upper right hand corner indicates that assumptions will need to be made for an early activity to progress. If the assumption is subsequently proved incorrect there may be the propagation of changes throughout all the activities with substantial impact on resource usage and schedule completion. There will be conflicts between multiple activities that constrain the sequencing. The result will be a reordering of the rows and columns. Another aim of analysis is the removal of coupled relationships through a process call ‘clustering’ and ‘tearing’ [67]. Instead of removing unnecessary assumptions this process introduces assumptions to make the overall process or plan more efficient. Activities are rearranged into a ‘block’ so that they are grouped around the diagonal. Assumptions are then applied that remove the ‘inherent coupling’ behaviour. There is however a risk of the emergence of an ‘input change’ coupling behaviour at a later date which may reintroduce the coupling. The decision to engage in tearing of activities within the DSM will therefore be based on the risk of later input changes against the benefit of improving the efficiency of the process by the removal of inherent coupling. The discussion of where the process or plan may break down should be undertaken during the collation of information on activity interactions and particularly for coupling behaviours. The use of Failure Mode Effects Analysis (FMEA) may be of use to analyse potential points of failure leading to unplanned process iterations. If significant these should be fed back into the complexity assessment and CSF processes and recorded on risk register. Sequencing Sequencing optimises the ordering of activities. This can be done before the first draft of the DSM, in which case the effect will be dramatic. There is usually a natural or intuitive ordering of activities so sequencing performed on an established DSM is not likely to initiate wholesale change. It is a useful process nonetheless. The process follows a number of steps. The first step, and the easiest, is the identification of activities with either no inputs or no outputs. Activities with no inputs can be undertaken first and activities with no outputs undertaken last. The sequence of all remaining activities will be influenced by the number, strength and type of interaction with other activities depending on whether binary or numerical DSMs are being used. There are a number of heuristics that can be used to undertake sequencing. The first is minimising the number of assumptions that are required in the model by reducing the number of interactions above the diagonal. This reduces the number of activities that need to be completed later in the sequence that feedback to earlier activities. The second is reducing the distance of the remaining interactions above the diagonal. As described previously long feedbacks can be the cause of the propagation of change throughout the development process. These activities are then identified as ‘coupled blocks’. The identification of these coupled blocks can be used to identify early where additional management effort would be well spent. Examples include the co-location of those involved, the use of collaboration tools and application of status and progress measures. Heuristics are of primary interest in this project through constraints of space but also in fitting with the project’s overall philosophy. Algorithms and commercial applications are available with potential benefits in terms of both expended effort and accuracy over a large quantity of data. Clustering and tearing The identification of blocks of coupled activities can be used for further analysis of the DSM. Decomposition of coupled blocks of activities may yield a more manageable set of sub-activities. In contract aggregation can be used to simplify the DSM but this will obscure individual feedbacks and is not generally recommended. The addition of activities earlier in the sequence can be used to reduce the number of assumptions. The decision to add activities will usually be made on a cost-benefit basis and should be used to reduce uncertainty and risk of change. Tearing reduces the interaction between a block of activities by introducing assumptions in place of iterations. Following the clustering of activities into blocks the process is a follows:
- 34. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 30 of 69 Suggest interaction to be removed, called a ‘tear’. The link or links with the longest feedback loops back providing the greatest benefit and should be considered first; Analyse the tear and, with the agreement of the stakeholder within the process, accept or reject the tear considering the degree of confidence on the assumption that replaces the interaction. This is a risk-versus-benefit based decision. If the first tear is discounted move onto the next best tear and repeat the analysis; The tear removes an interaction which reduces the block of activities in a particular cluster. The DSM can now be sequenced; Feedback marks in the DSM are replaced with torn marks to provide a prompt to check activity outputs affected by the tear against the new assumptions; Disproved assumptions will result in input change type coupling behaviour and rework of early activities. If this occurs the tearing exercise was unsuccessful and the intended efficiencies were not realised. Single assumptions can apply across multiple tears Commercial DSM tooling software that incorporates many of the more advanced algorithms and concepts that have been developed for sequencing and clustering and tearing is available. The most popular and commonly used of these are referenced on DSMweb.org [68]. None of these packages can directly interface with the most commonly used scheduling application employed within complex projects is Oracle’s Primavera [69]. This suggests that the general maturity of the applications has some way to go. Several of the applications do profess outputs compatible with Microsoft Project, a scheduling application which (in the author’s experience) tends to be used to plan smaller projects. 4.2.5. Application of the process DSM within a complexity management framework The ultimate purpose of DSM analysis, in the general case, is to optimise the matrix which will influence the creation of a time bound schedule. Analysis, as a part of a complexity framework, is no different. There is however an additional rationale for analysis that is consistent with the principle of focusing on areas of concern against which management effort can be directed. Analysis can not only optimise the activity sequence and interventions, but can also identify clusters of activities and single activities that have either the potential to unduly influence the overall schedule or else are of particular concern. This then affords the opportunity to address current issues, mitigate risks or put in place measures to provide early issue identification. It is important to note that the DSM is less useful in showing attributes such as start dates, durations or the lagging or leading nature of dependencies. This is one of the most powerful reasons to retain Gantt charts alongside the use of DSMs. Thus the DSM can be used as an input into the Gantt chart and provide more information on interaction related risks. DSMs can also be used to drive techniques such as PERT. In this instance supplementary information relating to likelihood and impact would be contained with the DSM. Development of a process / organisation architecture MDM can be used to identify required expertise and resourcing requirements for particular activities and better manage the interface between organisational and activity driven requirements. It can be used for ‘resource loading’ [70] of the Gantt chart and schedule to facilitate resource smoothing or levelling of the schedule and implementation of EVM methodologies. It is proposed that there should be a tiered application of DSM following complexity assessment and assignment of CSFs. These increase in detail and also effort according to the risks and benefits of application within the particular area of the technical development. A suggested ranking, descending in terms of effort, is as follows: Not applied and traditional planning techniques are used; A high-level application within WBS between elements of the system development; Application of a binary process DSM at lower level of WBS; Application of a binary MSM between process and organisation; Application of a numerical process DSM;
- 35. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 31 of 69 Application of a numerical MSM (process and organisation). Areas of concern are analysed at a lower level in the WBS with integration of the process and organisation via the use of a MSM. The use of numerical matrices has the potential to provide extremely rich information for use within other planning processes. However, this detail inevitably comes at a cost so should be reserved for areas of the development that are likely to require the greatest degree of planning and control. It is likely that analysis will highlight additional areas of concern that should be fed back into the complexity assessment and CSF processes. It is also advised that the use of DSMs at a high-level with the WBS should be undertaken early in the development lifecycle. The identification of interactions at an early stage may influence the composition of the WBS and the ordering of WBS elements within the schedule. Since WBS elements are invariably placed in the Gantt chart in accordance with a WBS numbering scheme it is desirable to place as many of the WBS elements in the correct sequence as is practicable. This will then aid the logical flowing of activities from top left to bottom right of a Gantt chart. It is useful to develop a numerical process architecture DSM convention which incorporates the principles described within the complexity assessment. This will validate the initial assessment findings and allow for the identification of any additional CSFs that arise from changes after further analysis. For simplicity the measurement of uncertainty, ambiguity, emergence and non-linearity will be captured numerically variously using several convections. Some will be absolute integers while others will be subjective. The additional information should be determined alongside the inputs, outputs and interactions which are described above. Some or all of these values should be contained separately within a spread sheet or database to the DSM matrix itself as to do so would clutter the layout. Uncertainty is directly related to the probability of change related to a DSM activity. An integer relating to the likelihood of a change in an activity’s outputs or duration will be adopted. A percentage is often used and this can be translated into a whole number between 0.1 and 0.9 with 0.1 representing 10%, i.e. very low likelihood, and 0.9 representing 90%, which is indicates that the outputs are likely to change. No activities will be assigned 1, i.e. be planned with a 100% chance of change. Change may be due to input changes or mistakes. Ambiguity impacts the development in terms of the number of assumptions that are required and delayed start dates if assumptions are not to be used. The number of assumptions can be counted with high number of assumptions being related to higher levels of uncertainty. Emergence is where there are unexpected interactions [44], i.e. more activities are affected by a single change that is originally envisaged. High emergence is where many activities are so affected. Therefore, it is sensible to assign a value to an activity equivalent to the number of activities that rely on its outputs. Conversely a value can be assigned to each activity that represents the number of inputs it depends upon. The number of outputs from a single activity identifies which activities, if subject to change, are particularly influential on the wider DSM. The number of inputs shows how sensitive an activity is to changes elsewhere. Non-linearity is seen as a disproportionate impact resulting from a change. This can be represented by a ranking from 1 to 10 with 1 representing a slight impact on dependant activities and 10 being a substantial impact. Of course the impact of a change to an activity may not be homogeneous across its outputs to other activities. For the sake of simplicity an average measure of non-linearity will be proposed for this model. Figure 13. Assigning additional attributes to process architecture DSM activities. Activity Uncertainty (Also known as likelihood) Ambiguity (number of assumptions) Emergence Non-linearity (Also known as impact) System requirements – sub-system X 4 5 12 outputs 5 inputs 7 System safety assessment – sub-system x 3 1 5 outputs 5 inputs 9
- 36. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 32 of 69 Program-size Complexity is directly related to the number of activities to be managed. It is not considered useful to measure this attribute to manage at the level of activities. Measurement of Program-sized complexity would be more useful at a higher level, for example to measure the size of a work package, development phase or the project in its entirety. In these instances, suggested measures would relate to the number of activities or interactions between activities within the chosen area of interest. Figure 13 shows an example with fictitious data for two sub-systems and examples of data that could be assigned against the various properties. Completing a numerical DSM in this way will allow further analysis. Case studies [67] demonstrate that the assignment of probability and impact (called uncertainty and non-linearity in this text) to an activity can be used to perform PERT analysis on the schedule. There are however several other ways that these and the other remaining attributes can be used for analysis. In general, particular thresholds can be filtered and sorted to identify trends or group activities together for analysis and shading and colouring employed for ease of reference and communication of activities of particular concern. In each case there is the potential for tooling to automatically make the necessary calculations based on the inputs into the DSM itself. Additional management effort and status and progress measures can be introduced to specifically address areas of concern. The pairing of poor attributes relating to uncertainty/ambiguity and emergence/non-linearity is certainly not desirable. Section 2.4 discusses the profile of complexity across the development lifecycle. There is proposed that uncertainty and ambiguity should naturally reduce over the development lifecycle while emergence and non-linearity generally increases. High values of three or four of these properties against a group of activities, especially later in the development lifecycle, show an area of risk. Even a single activity with a similarly poor complexity profile, if on the schedule’s critical path, deserves closer attention. High uncertainty and ambiguity later in a development lifecycle may be the result of process or schedule optimisation activities, such as clustering and tearing, where assumptions are introduced to reduce the schedule duration and sequential activities are instead undertaken in parallel. This in turn may be as a result of either the original over-optimistic technical development completion milestone or because of delays demanding a new schedule. Another technique is to analyse activity feedback cycles to determine those that are particularly uncertain. Feedback cycles with a high likelihood of change that coincide with the critical path will be of particular concern as any delay, due to rework, will directly impact on the completion of technical development. To calculate the likelihood of change across a three activity cycle the following calculation can be undertaken: Likelihood of change across a feedback cycle = 1-{(1-a)(1-b)(1-c)} Where a, b and c are the uncertainties for the three activities in the particular cycle. This essentially calculates the likelihood of any of the activities changing within the feedback cycle. More activities and high likelihood of change will produce a low measure. One method of addressing this is through the introduction of additional activities to reduce the number of assumptions and thus the potential for input changes. The introduction of other management controls, described in additional CSFs, should also be considered. Finally, two or more sequential or coupled activities with large attributes of emergence have the potential for substantial propagation of change. It may be difficult to reduce the numbers of impacted activities (for example tearing and clustering to reduce dependencies may instead simply result in input changes) but further reducing uncertainty, even when already relatively low, is sometimes possible. Taking the analysis further other activity measures can be derived from its inputs and outputs. Activities particularly vulnerable to change propagation can be determined by calculating the product of uncertainty and non-linearity for each input to an activity and taking their sum. Large calculated amounts will highlight an activity that is at risk of change. For example, activity Y has four inputs. The inputs have products of
- 37. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 33 of 69 uncertainty and non-linearity of 1x8, 4x4, 3x6 and 7x3. The sum of these is 63 (8+16+18+21). This can be mapped across the DSM using colours or shading to denote activities of low or high vulnerability. Similarly, activities with high potential to influence change can be determined by calculating the product of uncertainty and non-linearity for each output and multiplying this by the number of outputs. In addition to allowing effort to be directed against particular activities these measures will readily allow change impact analysis to be undertaken and influence decision making. A relevant example of the potential benefit is that of a seemingly small non-essential change which has the potential for a substantial impact across the schedule. In this instance the change could safely be argued that it should be disallowed. So in summary the following measures can be calculated: Primary measures; o Uncertainty – subjective measure of likelihood of change, determined in conjunction with stakeholders during formation of DSM; o Ambiguity – determined through the number of assumption required for an activity; o Emergence – determined through the number of inputs and outputs for each activity; o Non-linearity – determined as the general impact of change of an activity’s outputs on other activities; Secondary measures; o Likelihood of change across a feedback cycle – particularly useful if activities coincide with the schedule critical path; o Vulnerability to change propagation – calculated from the uncertainty and non-linearity of an activities inputs; o Change propagation Influence - calculated from the uncertainty and non-linearity of the activity multiplied by the number of outputs it has. The calculation of mean amounts for the primary measures can give an indication of the complexity profile for a portion of the overall technical development based on the boundaries of the DSM. Complexity profiles were discussed in section 2.4. This could be across the entire phase or across the activities currently in progress and could have one of two purposes. The first would be to provide norms for technical development that could be used for later developments. Complexity profiles deviating markedly from the norm would require further investigation. The second would be the identification of trends. This is particularly useful in identifying when uncertainty and ambiguity remain consistently elevated or are increasing. Either would be a cause for concern. Coupled with natural increasing emergence and non-linearity, high uncertainty and/or ambiguity pose a high risk for significant change propagation. The identification of important activities, activity cycles or clusters of activities should direct the implementation of status and progress measures. These measures will be discussed in section 5. 4.2.6. Process-organisational MDMs and their application The creation of a MDM requires both DSMs to be available for the particular area of the development and for it to be of sufficient maturity. Much of the effort required is expended creating the donor DSMs and the principle of the MDM is similar to that of the DSM. Although there are a number of types of MDMs that can be created it is a process-architecture MDM that is particularly applicable to the planning and control of complex technical development processes. An MDM can be either of the binary or numerical type but unlike the DSM the MDM does not have a diagonal. It will show the Figure 14. Process-organisation MDM [68].
- 38. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 34 of 69 interaction between particular organisation elements and the overall process which is useful in determining the impact of a deficit in resource or capability across activities. It will also show the organisational requirements of a particular activity. Analysis of the MDM may show a cluster of organisational elements that are particularly influential during a particular phase. It may be beneficial to temporarily co-locate a number of teams or make additional efforts to integrate their interactions. The demands of the schedule can be inferred from the MDM allowing resource and capability requirements to be planned in advance. The MDM can be populated with the estimated level of effort that is required in terms of man hours or cost. Missing organisational requirements may be identified by the absence of interactions of MDM rows. The findings of this analysis can be used in cost and resource estimates and be used to resource-load the Gantt chart to allow the application of EVM. 5. System Performance Measures 5.1. Introduction The assigning of Performance Measures and the gathering and processing of project data to calculate them is the ‘check’ activity within Demming’s ‘Plan-Check-Act-Do’ methodology. It is a vital component between planning and controlling of a project that should inform if and when action should be taken. Effective Performance Measures should additionally provide information on what type of intervention may be effective. They may indicate a deviation from the plan but also the emergence of undesirable properties within the development process. Pertinent examples of such an undesirable property are poor outcomes after the implementation of a CSF or a large backlog of configuration change requests. The control and management of on-going development activities is dependent on relevant and accurate information on both the status and progress of the activities being undertaken. Technical development processes, as compared to normal business processes, are typified by a number of unique properties which makes effective process monitoring important [72]: ‘Dynamic, creative and chaotic’; Contain many feedback mechanisms; The process in its entirety is largely ‘virtual’ and ‘not always precise’; The likelihood of change is high due to feedback mechanisms, imperfectly defined requirements and customer led changes. Without the appropriate choice and effective implementation of Performance Measures issues may go undetected until they are manifestly apparent, by which time they may be unrecoverable. Additionally, it is useful to know where management effort needs to be directed most efficiently. Examples of existing frameworks used for the formation and collection of measures include House of Quality, known as Quality Function Deployment, Goal-Question-metric and Balanced Scorecard [73]. The use of metrics within concept and development activities is not widespread and can lead to a number of ‘coping’ mechanisms being adopted by personnel in response to them. Simon and Simon list a number of issues that have been identified during an empirical study [74] and as thus should be avoided. Similarly, Demming describes particular ‘traps’ to be avoided [75]. Conversely Kline et al [76] describe a method for designing performance measures that is applicable across any type of technical development. The design and implementation of performance measures should be designed to avoid aspects that reduce effectiveness and embrace those that improve their chance of having a positive impact. The measures that are chosen should also suit the particular development by its particular characteristics. The method recommended by this dissertation is to identify areas of complexity and criticality through the complexity assessment and to measure the implementation of CSFs. Two ways of looking at the performance measures is in terms of areas of the WBS and complexity criteria listed and described in Section 2. Both can be used to tailor the performance measures to balance effort of collection and analysis of data against the benefits of closer and more accurate monitoring and control. However, the constraints inherent in this
- 39. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 35 of 69 approach are extensive research, undertaken by the previous literature survey [4] and additional research within this dissertation, revealed the limited number of techniques currently available. This was reiterated by the findings of questionnaire survey which revealed no additional performance measures from amongst the 122 respondents. Additional research, undertaken since the literature review, can be found within Appendix L. 5.2. Desirable properties of Performance Measures Performance Measures, often call metrics, can be placed in several categories. Mathematical system performance measures are calculated from fundamental data. These metrics are either derived or combined and an excellent example is that of the metrics relating to the established method of EVM. In this methodology uses primary data, relating to the actual costs and physical progress, is compared with cost and schedule planning to derive measures such as cost and schedule variance and cost and schedule performance indicators. Practical system metrics are derived from the application of ‘empirically established factual logic’. Heuristic system metrics are similar to practical system metrics but the scope of the metrics is restricted to particular issues [73]. Depending on the level of decomposition the metrics described in this dissertation are most likely to be either practical system or heuristic system metrics. EVM is used extensively within projects and technical developments and the questionnaire survey will use it as a baseline against which the influence of other performance is measured. The properties of the measures are of importance. Foremost the measures should of course have a clear and obvious purpose and it is also beneficial that the measure is as ‘homomorphic’ as possible with the source data [73]. The nature of the measures should provide a view of developments objectives ranging from the short-term to the long-term [73]. Measures should be process orientated as well as schedule and cost orientated indicators [73]. In accordance with the concept of the balanced scorecard there should be breadth to the perspective of the measures that are adopted [77]. Measures may be either lagging or leading with staple measures inevitably relating to cost and schedule. An example of a common system of measurement is those used within EVM which reconciles planned cost and schedule against actual cost and progress. In this instance it is also a lagging measure. That is an issue is flagged only once activities fall behind schedule or estimated costs are exceeded. If the issue is determined early enough action can be taken to converge with the original plan. However, the severity of the issue is greater the later it is detected and both low sensitivity of data and the use of monthly reporting cycles may even delay this late detection by a month or two. In this instance it is much better to predict the occurrence of cost overruns and schedule delays through the use of leading measures. These measures will target areas of the technical development that will later influence the actual activity durations and costs. Other properties that should be considered are their frequency of application and it is important to balance the effort of application against the benefit that may be derived from the frequency of measurement. A measure may require a lot of time and effort to determine, for example the findings of an audit. It may measure the properties of the development that do not change frequently, such as the schedule for the current phase. In both instances it is appropriate to only determine the measure once per development phase or perhaps only three or four times per year. Exceeding this frequency of measurement may entail the use of inaccurate, incomplete or otherwise estimated data or not provide sufficient additional benefit to make it worthwhile. Conversely the measure may follow a dynamic development property. The status of development artefacts may be automatically recorded via the database or tooling that is being utilised. This allows the measures to be taken more frequently or even viewed real-time. The cadence of measure reporting may also be dictated by time-bound criteria. The frequency of EVM reporting is constrained by the monthly frequency that actual costs are most often calculated which in turn relates to the submission of invoices by contractors and consultants. Measures can be quantitative or qualitative. Often it is difficult to provide quantitative data and only a commentary on the on-going activities is possible. Even when quantitative is possible it is worth combining with qualitative to provide valuable context to the information.
- 40. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 36 of 69 Examples of leading measures include development resource and the quantity development change requests that have been submitted during a period. They should highlight early concerns either through an absolute quantity or through a trend over successive reporting periods. Using the first example from above a deviation below the planned level of development resource may not yet be reflected in a delay in the schedule. Early identification would allow management action to be taken to elevate resourcing to a suitable level thus avoid a later schedule deviation. An increasing trend in the number of development change requests may indicate inadequacies within the existing requirements or stakeholder management issues. Furthermore, Increases in resourcing requirements and funding may result from an increase in the amount of change. Again early identification of changes can be used to prevent or minimise an adverse impact on the development activities. It has been proposed that the selection of measures could be based on process analysis. There are several barriers to this approach [73]: There is a high overhead to process modelling which increases processes are already in place; Analysis and implementation of measures needs to be undertaken early for maximum effect but process uncertainty is high and change is likely. This can lead to wasted effort and/or incomplete analysis; Changes will continue to propagate through the project as contracts are let and strategies evolve; The measures chosen by such a method tend to be too abstract. Though some Performance Measures are used in almost each and every project (EVM for example) it is logical others are chosen on the basis of particular development characteristics. Within this framework the selection should be influenced by the complexity assessment and the subsequent sub-processes and following some simple concepts. Specifically, the goals of the measures will be determined with a focus on areas of particular complexity and risk, including the assessment of areas described in CSFs. The Goal-Question-Metric method is an exceedingly straightforward way of determining measures. The ‘goal’ of the measure consists of four components; purpose, issue, object and viewpoint and would be directly related to CSFs and risks. The ‘question’ typically asks for the current status and current trend in this status. The ‘metric’ is perhaps the most difficult aspect to determine as meaningful measures that are at a low level of abstraction can be difficult to determine. ‘metrics’ will be very strongly influenced by the range of measures that are currently available within industry or else are developed during the project itself. An illustration of the Goal-Question- Metric approach is shown in Figure 15 [78]. The level of decomposition should be considered and the WBS can again be used as the means for this. Low levels of decomposition provide an overview of development health but will not provide a real indication as to the source of any issues as the node within the WBS will be influenced by all of those below it. A high-level of decomposition may suffer from low sensitivity of the data due to lower sample sizes or chosen method of measurement and will certainly require a good deal more effort to implement. An appropriate level within the WBS should be chosen for the highest level of decomposition and it is perfectly feasible to use metrics following the same principles at simultaneously at different levels of decomposition for both management reporting and to facilitate targeting response to issues arising. Methods of data collection, and thus sensitivity and effort, should be commensurate with the overall goal of the Performance Measure. Performance Measures can be selected to focus on particular areas of concern within Figure 15. The Goal-Question-Metric method [78].
- 41. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 37 of 69 the development. This may be through a higher level of decomposition for a particular phase or cluster of activities or else the choice of a unique measure. Methods that can direct where measures are to be chosen include during the complexity assessment and assigning of success factors and during subsequent planning. In particular DSM can highlight areas of intense coupling and complexity. Such prioritisation of activities may be due to their properties of uncertainty, emergence or non-linearity and their proximity to the schedule’s critical path. The use of focused measures must be in conjunction with other less detailed measures to ensure coverage of the entire schedule of activities. It is also beneficial to overlap metrics, where possible, to provide a degree of confirmation of the findings [73]. Other factors to consider include: Metrics should be complete, correct, consistent and clear; [73]; As they will be communicated and reported metrics should be both ‘simplistic’ [73] and be chosen at an appropriate level of abstraction so they are readily understandable; The collection of meta-data during and as a part of the individual activities, as opposed to periodic and dedicated data collection, will reduce the management burden required. An example would be progress measurement during the production of development artefacts, at the end of each individual iteration. This will introduce a level of ‘automation’ [79] into the process; Consideration should be given to the collection of metrics by project assurance functions rather than the development team [58] to ensure impartiality. The reliability of some Performance Measures is necessarily subjective due to the scarcity of formal methods of collecting data in many areas. Subjectivity can be curtailed through the adoption of rules and guidance in the collection and analysis. A simple example is the progress measurement of development artefacts as shown in Figure 16, and this also incorporates the principle of creating meta-data during the activities themselves. It this case the artefact owner would track progress according to the relevant guidelines which could be placed on a shared database for ready collation and analysis. Artefact completion milestones Percentage complete Completion of first draft of artefact 50 Completion of first review 60 Inclusion of first comments 80 Completion of second review 85 Inclusion of second review comments and submission for approval 90 Artefact approved 100 Figure 16. Simple method of measuring progress in development artefact. Many Performance Measures are highly abstract. There are a number of these that relate to DSMs and these should be considered for use. They however largely relate to complexity so while they may provide a validation of the other analysis that has been undertaken this will relate to areas of risk within the technical development. Examples of the areas of the development process that should be measured include [80]: Schedule and cost planning – will measure risks, bottlenecks and the critical path as well as the overall cost. It should also consider process iterations and associated uncertainty; Resource allocation – will measure resource and capability levels as compared with the plan and issues such as resource smoothing/levelling, removal of redundancy and accessibility of resource; Quality – this will measure the consistent flow of information, completion of documentation in line with process and meeting requirements and the distribution of risk amongst processes; Flexibility – will measure the status of buffers to absorb delays and defences against individual errors and general process resilience; Organisational decomposition – will measure to determine whether the organisation of workgroups and teams is adequate, efficient communication is in place;
- 42. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 38 of 69 Interfaces – will measure which entities need synchronising, speed of communication across interfaces and the relevant communication paths in place; Transparency – will measure whether the organisational units are aware of their impact on outcomes and the mental model of the process organisation. Decision making – will measure which decision points have a high impact on outcomes. Also artefacts to be measured using a rating based on interdependency and risk: Number of dependencies; Origin of dependencies – with organisational area, interdepartmental, inter-contract, external dependency; Status of dependencies – i.e. the meta data derived for them; Percentage of document that requires dependencies; Identified risks. Focusing effort on particularly critical areas of requirements management is achievable by using the Complexity Assessment and identifying nodes on the WBS. This can identify when Performance Measurement should commence as well as the depth and breadth of measurement that is appropriate. 5.3. Selection of System Performance Measures from literature In this section, we discuss the measures which have been selected from both the previous literature survey [4] and further research as a part of this dissertation as being significant for inclusion in the Complexity Management Framework. Each measure will be categorised against the previously described criteria in Section 5.2. They will be chosen for their credibility of previous application and breadth of methodology and aspects of development for which they are used. These will be verified by a questionnaire survey in terms of their comparative influence. EVM will not be described in detail but will be used as a benchmark in the questionnaire survey against which the influence of other Performance Measures can be compared. 5.3.1. Requirements The management of requirements is a fundamental part of technical development and the two central techniques relate to the measurement of outcomes (requirements satisfaction) and process (linked to a variety of attributes and metadata). 5.3.1.1. Satisfaction of Stakeholder and System Requirements A fundamental tenet of technical development is the elicitation and satisfaction of requirements. In turn these requirements can be broadly sub-divided into stakeholder requirements and system requirements. Requirements, especially system requirements, are prone to successive iterations and coupling behaviours as represented in the DSM model. The measurement of requirements is described within a number of systems engineering manuals and lately INCOSE’s System Engineering Handbook. They observe both stakeholder and system requirements from the perspective of their success in fulfilling the overarching business requirements [28][54] and as such are quality orientated measures. The technique identifies a limited number of particular Stakeholder and Systems Requirements whose realisation (satisfaction) are critical or important to overall success. Selection of these can be influenced by Criticality Assessment though they are often closely linked to high-level business and operational level requirements that are determined at the very beginning. With respect to complexity criteria, the analysis of requirements allows monitoring and control of the system development and particularly the technology and internal interfaces. Other complexity criteria will have a direct impact on requirements with excellent examples of such criteria being development process, internal organisation, organisation and stakeholders. Measurement is facilitated through design review type activities and provides lagging measures. This period of lag is of course dependant on the frequency of design review.
- 43. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 39 of 69 5.3.1.2. Requirements attributes Another method of measuring System Requirements is through the collation of associated information to be stored in a matrix, spread sheet or database. This is sometimes known as the Requirements Verification Traceability Matrix [81]. The metadata can be compared against the plan or trends monitored to identify areas of concern. The complexity of the RVTM should be proportional to that of the technical development and may include fields that are specifically chosen to assist in the management of foreseen project and system development risks identified early using techniques including the complexity assessment. Suggested fields to be collated include [81]: System Requirement unique identifier and name; Requirement description; Requirements specification (document reference); Overall requirement status (such as detailed in design, implementation or integration); Trace to overarching Stakeholder Requirement (unique identifier and document reference); Verification and validations procedures (document references); Verification strategy status (such as undefined, strategy only, procedure completed); Verification status (such as not started, failed, completed with reservations or completed and with document references as appropriate); Validation purpose (such as for acceptance, certification, readiness for use or Qualification); Validation status (such as not started, failed, completed with reservations, completed and with document references as appropriate). Other fields that could be employed in either the same artefact or complementary artefacts, including: Stability (susceptibility to change); Design compliance; Interface compliance; Process compliance; Risk and risk status; Safety case and licensing compliance; Procurement and contractual compliance; Importance or criticality of requirements within the project schedule. These measures may give an indication of requirements against the planned development and are leading in nature as they indicate potential issues ahead of requirements satisfaction. They do however allow for the identification of areas of concern. More generally requirements can be given a wide variety of metadata that can be used to interpret the overall status of the requirements or a particular sub-system or area of development. Not all of these attributes should be chosen due to the overhead required to maintain complete and accurate data sets. Attributes should be chosen early in the development lifecycle to ensure that data does not require to be retrofitted at a later date [81]. Both the RVTM and requirements attributes are primarily concerned with quality and identifying issues relating to progress. Many of these attributes, together or independently, can be used to infer the efficacy of resource allocation, organisational decomposition, interfaces and Decision making. For instance, a backlog of requirements with a particular status may indicate one or more bottlenecks in the process. This may be due to issues with resourcing, organisational efficiency or decision making. It can be used to identify risk at a high level of decomposition within the system of interest which can be rolled up to show particular sub- systems which deserve additional attention [81]. If trends are identified early enough, for example a backlog of under review requirements within a particular development area, then action can be taken to reduce or eliminate delays to the schedule.
- 44. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 40 of 69 5.3.2. Development health Development health is likely to contain a high degree of subjectivity. Facets such as adherence to the planned cost and schedule can be measured through EVM or other similar techniques. Other less tangible aspects will require a framework to provide context and guidance in which they can be assessed. Using a defined method Kline et al [76] formed a weighted matrix of design performance characteristics that can be used to measure development health. Each high-level measure was formed of five individual factors with each of these being assigned an individual text description as a basis for scoring. The high-level measures are as follows: Problem Definition; Prior Knowledge; Divergent Thinking; Professional Analysis; Decision Making; Create & Follow Plan; Iterate & Assess; Validate Solutions; Communication; Teamwork. By observing attributes of the development process and team the technique is a leading indicator and should be able to identify risks to the development before they are fully realised. It should be undertaken reasonably infrequently, say at the beginning of a phase or mid-development phase, at three or four month intervals, if it is particularly long in duration. It is proposed that the Kline et al methodology is applied, partly or in full, to form new measures or tailor exist measures for use in technical development. One candidates for this approach, mentioned in the Literature Survey [4], is Torbet et al’s six criteria of ‘Design Performance Measurement’ [82]; Client needs (stakeholder requirements); Integrating design into objectives (system requirements); Internal design processes (suitability and effectiveness of internal design); External design processes (suitability and effectiveness of external design); Profitability and efficiency (of the design); Learning and innovation. There is of course considerable overlap between these and requirements orientated measures and with the criteria within the complexity assessment. Another way of identifying the subject for measurement is their identification during complexity assessment and designation of CSFs. Alignment of specific performance measures with area of concern broadly follows the principles described in planning via the use of DSM. That is focus on areas where there is particular risk of deviation from the plan while planning (in this case monitoring) at a lower level of decomposition level across the entirety of the development. The headings within Kline et al’s ‘Creating and Using a Performance Measure for the Engineering Design Process’ relate particularly to the complexity criteria of the development process and internal organisation. 5.3.3. Process maturity A framework for process maturity is the ‘Project Definition Rating Index’ or PDRI [83][84]. Maturity can apply to both the development in terms of where it is within the process or the technology that is to be used within the system of interest. There will necessarily be some overlap and influence between the two. Though not directly related to development health or schedule, there can be some inference as to the status of these. For instance, in an extreme case technology of low maturity which is approaching validation suggests either problems within the development process or either badly planned or incompletely executed activities leading to that particular point in time.
- 45. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 41 of 69 Of course there is still much potential for subjectivity within the scoring of the PDRI framework but this can be further refined by providing individual guidance for each criterion as is documented in Kline et al’s development health methodology. The result of the scoring can then be compared with the ‘maturity value rating’ and ‘qualitative criteria’ [84]. A maturity rating that does not correspond with the particular phase of the development that is currently in progress indicates a concern. This can be further analysed to determine the areas where the issues reside by looking at the individual criteria. Though similar in methodology to Kline et al’s method it is a lagging indicators as it essentially observes the status of the development at a particular point in time. Again a review at the beginning of a development phase or every three or four months would be appropriate. Process maturity measures relate directly to the complexity criteria of development process but also indirectly to the three complexity criteria. 5.3.4. System maturity A popular method of assessing the system with the United States, and especially within military and space industry domains, is that of the Technology Readiness Assessment (TRA) also known as Technology Readiness Level (TRL). Specifically TRL has been developed for use by both the US Department of Defense and NASA [86][87][88]. TRL assesses the maturity of the system as a means of analysing development risk and used in a similar way as the PDRI methodology. That is if the TRL is below that required for the development phase there are serious risks to the realisation of the technology which will be ultimately be reflected in an inability to verify or validate the technology. TRLs generally have ratings between TRL 1 and TRL 9 with subjectivity of analysis constrained through the use of textual guidance. TRL 1 is the least mature and TRL 9 denotes technology proven within its intended operational environment. In both of these applications of TRL a team of subject matter experts will assist in providing the actual rating which will form the basis for stop/go type decisions by a board of appropriate managers at predetermined points in the development such as development stage boundaries. TRLs relate directly to the complexity criteria of technology. A development of TRL, and an attempt to address to address TRL’s inherent limitation, is the System Readiness Assessment or SRA. SRA provides a ‘whole system perspective’ [89]. It assesses all system components, the integration of them along with external dependencies and is designed to be undertaken more frequently than those using solely TRLs [89]. TRLs are assigned to the system of interest as before along with an Integration Readiness Level or IRL. Criteria are provided to guide the assessment as per the traditional TRL method. From these three System Readiness Level Metrics are derived through calculation. Component SRL looks at individual components and how they are integrated to identify elements of the system that are lagging behind the system as a whole. Composite SRL is concerned with the overall integration of the system and this is then converted into an integer between 1 and 9, called the SRL [88]. Assessments and calculations are performed across the system architecture and the ratings are similar to those of TRLs. The IRL relates to the complexity criteria of internal interfaces while SRL is relates to the three system development complexity criteria as a whole. 5.3.5. Organisation, process and schedule complexity DSMs were the subject to discussion in Section 4.2 and are useful in both defining areas of the organisation, process or schedule that require particular attention and in providing useful measures to provide a basis for Figure 17. Maturity value rating criteria [85].
- 46. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 42 of 69 management action. As they relate to the plan they are reasonably static in nature and as such lend themselves to plan validation type activities at the beginning of a development phase or at a new iteration of the schedule. Previously the use of DSM related performance measures were discussed in Section 4.2.5 and suggested calculating median and peak quantities for measuring plan complexity. Observed DSM primary measures can be either trended through successive iterations or compared against acceptable absolute values that were determined through previous developments. These can either consider the phased activities as a whole or the current schedule critical path. The secondary measures of likelihood of change across a feedback cycle and vulnerability to change propagation could be used to inform rescheduling or other interventions to be undertaken to reduce development risks to acceptable levels. The basic properties of the DSM itself can be used to provide an insight into the schedule and its complexity. The relevant proportion of sequential, parallel, coupled and conditional activities within a development phase can be used to provide an indication of its status. A high incidence of coupled activities especially, gives a clear indication of complexity. Again this could be applied against the critical path activities only to provide information on these priority activities. Finally, Kreimeyer and Lindemann [73] list four metrics using the language of DSM optimisation activities: Sequencing – the number of ‘ideally sequenced’ activities within the DSM, i.e. sequential activities; Tearing – the number of activities that have been subject to tearing due to them being a barrier to sequencing; Banding – the number of activities that are independent to each other, i.e. parallel activities; Clustering – the number of ‘mutually related’ i.e. coupled activities. As a way of identifying areas of concern within the plan this is a leading indicator and should prompt re- planning such as further optimisation of the DSM. Process and organisational-architecture DSM related performance measures are concerned with controlling development process and internal organisation complexity criteria. 5.4. The verification of performance measures through questionnaire The performance measures chosen from the literature reviewed during the project were verified with the practitioner group as a part of the questionnaire described in Chapter 3 above. The objective was to rank influence and comparison with the most widely used performance measure from the EVM methodology. The questionnaire can be seen in Appendix E with performance measures relating to questions 13 and 14. Results, ranked by influence, are shown in Appendix F. Influences were generally lower than that attributed to CSFs, with results across the entire 122 respondents ranging from 3.04 (just above medium influence) for the use of PDRI and 3.77 (with 4.0 indicating high influence) for measurement of requirements process. This may suggest a general dissatisfaction with the identification and practical use of performance measures and in agreement with the author’s own experiences. No other measures were suggested for use, providing some assurance that the literature survey was comprehensive in its review of the subject. The measurement of the realisation of stakeholder and system requirements was also rated highly and just below the top performance measure at 3.76. Measurement of plan complexity through determination of proportion of sequential, parallel, coupled and conditional activities on the schedule was third with 3.69. This be achieved through deeper analysis of planning through traditional Gantt charts or, as advocated here, using DSM methodologies. EVM did surprisingly poorly at eighth of the twelve measures considered with a rating 3.56 It did rate the third highest number of very highs but suffered a lot of lower scores which offset this. This suggests the high value of integrating requirements management into a project and that EVM is not sufficient alone. This is not reflected in the practice where Earned Value is often the only performance measure used and requirements require scant attention within project management methodologies [4]. Complexity measurement using heuristics against the WBS elements also rated relatively poorly but only just below EVM at 3.48. This
- 47. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 43 of 69 supports the author’s opinion that absolute values of complexity are of limited value alone but support the following sub-processes. Its value is qualitative rather than quantitative. There were possibly shortfalls in the survey and barriers to Performance Measure use that influenced individual scoring of some of the Performance Measures. Providing a sufficiently brief description and without additional explanatory text may have created a bias against several of the measures. The PDRI project maturity and project health measures scored very poorly and this along with the apparent effort of application (200 individual criteria for PDRI and 10 high-level and 50 low-level organisational and process characteristics for process health) may have depressed the final scores. Further research would be needed to either confirm or disprove this. Two comments from question 14 provided further validation of the author’s approach and initial observations. “Keep the complexity measures as simple as possible!” “Many forms of performance measures out there, I think you have captured most, personally I don't rate them and although their aim or managing system interfaces and relative maturity thereof is correct - they generally don't add much value. There would definitely be mileage in getting back to basics, systems definition, boundary limit definition and design review.” 5.4.1. Performance measures by age, role and industry Findings were consistent over age, role and industry and especially over the top three rated performance measures across each dataset. Considering the top three performance measures notable exceptions were as follows: Earned Value Management was considered the third most influential performance measure’ by over 65s’. This elevated the importance of EVM over any other sub-dataset. The measurement of percentage complete of project artefacts and the measurement of IRLs were considered the most important second most important performance measures respectively by ‘engineering personnel’. These reflect the concerns and challenges that are particularly faced by engineering disciplines over other project personnel. ‘Defence’ valued TRL, IRL and SRL over all other performance measures which was not particularly surprising when it is considered that these measures originated from the defence sector. 5.4.2. Conclusion The findings of the questionnaire support the assertion that further work is required to develop performance measures that better represent the challenges faced by today’s projects. Requirements in particular should receive greater attention which in turn will encourage their better management. Plan complexity is the other measure highly worthy of development. The combination of the measures of requirements and plan complexity in conjunction with EVM should provide an excellent way of monitoring and thus controlling many project types. This should be supplemented with TRL and IRL methodologies when the novelty of technology dictates. This of course needs to be considered against the resource and cost overheads involved in such an undertaking. 5.5. The collective use of performance measures The use of the Balanced Scorecard [78] is much publicised within industry and has been further refined into ‘The DMI Design Value Scorecard’ [90]. These concepts provide little information as to what and how measurements are taken but the principle of presenting together a broad selection of relevant measures has merit. Using the broad principles identified from through the concept of Balanced Scorecard it is proposed that a Development Scorecard could be developed. This would present a variety of metrics together to best represent status across the breadth and depth of the development. It would present a high-level view of the development for director level management, along with a text-based summary to provide context and indicate underlying areas of concern that otherwise may be hidden. Below this the measures would be
- 48. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 44 of 69 presented at a number of levels of decomposition. Strategic measures would be primarily of interest to the management leading the development to provide data on particular areas of concern within the process or organisation. Tactical measures would provide feedback for team leaders and individuals to make interventions at the level of the work being undertaken. The performance measures should be chosen to address concerns raised during the complexity assessment with regards areas of high complexity or criticality. For example, the following measures could be combined in a development scorecard: Plan complexity consisting of: o Average uncertainty, ambiguity, emergence and non-linearity as derived from DSM (leading); o Proportion of sequential, parallel, coupled and conditional activities within DSM (leading); Conformity with the plan consisting of one or more of the following: o Earned Value Management Cost Performance Index (lagging); Schedule Performance Index (lagging); o Overall progress using heuristics with guidelines to aid analysis of individual development artefacts; o Actual resource and capability compared to plan; Process consisting of: o Others such as numbers and processing times for engineering change control requests and quality non-conformances; o Process health using Kline et al method (leading); o Process maturity using PDRI (lagging); Requirements consisting of: o Realisation – relating to MOE, MOS, MOP and TPM (lagging); o Execution – relating to selected requirements metadata (leading); System maturity consisting of: o Technology Readiness level (lagging); o Integration Readiness level (lagging); o System Readiness Level (lagging); Bold type indicates measures assessed as having the greatest influence from the questionnaire survey. The high-level measures are in italics and would be rated from very low to very high. Strategic measures are contained below each of these. This would provide coverage of most of the complexity criteria. 6. Managing system development risks 6.1. Introduction ‘Management of risk should be systematic and not based on chance. It is about the proactive identification, assessment and control of risk that might affect the delivery of the project’s objectives’ [53]. The process described so far identifies ways of analysing the technical development for areas of concern against which effort can be best directed. An important part of this is identifying development risks which is traditionally achieved through a risk management processes that uses techniques such as workshops and risk reviews. This process can be used to supplement this approach, principally in the identification of risks, as it prompts analysis of various aspects of the development and at varying levels of decomposition. The initial complexity assessment is used to instruct on the areas of particular complexity and CSFs necessary to increase the likelihood of success. Residual risks will remain should the CSFs be imperfectly implemented or only partially address the raised concern. These risks can then be placed upon the risk register. This initial complexity assessment is likely to generate high-level risks only due to the low amount of decomposition possible and available detail on the development process. Successive iterations of the complexity assessment
- 49. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 45 of 69 however will allow risks to be targeted and existing risks will be elucidated upon, confirmed or dismissed as not relevant. The planning process will identify still more risks and these will be focussed upon particular types of interactions or clusters of activities. Again these should be compared to risks identified earlier as a means to improving the quality of the information upon the risk register. A recognised and respected approach for managing risk is contained within the PRINCE2 project management methodology. PRINCE2 bases much of this approach on Management of Risk: Guidance for Practitioners as published by the UK Office for Government Commerce. Their guidance follows a number of important principles which are satisfied by risk management through the complexity management process [53]: Understand the project’s context – complexity assessment looks across the breadth of the technical development; Involve stakeholders – achieved at complexity assessment and particularly during DSM planning activities; Establish clear project objectives – as highlighted during the initial complexity assessment; Develop the project risk management approach – partially satisfied through complexity management process in conjunction with the general risk management process; Report on risk regularly – dependant on the frequency of complexity assessment and planning iterations (possibly partially satisfied); Define clear roles and responsibilities - as highlighted during the initial complexity assessment (possibly partially satisfied); Establish a support structure and a supportive culture for risk management – through an independent risk management strategy; Monitor for early warning indicators – through complexity management process monitoring activities; Establish a review cycle and look for continual improvement – partially satisfied by complexity assessment and planning iterations. Incorporation of the contribution of complexity management within the risk management strategy will provide additional support to the process and prevent duplicated effort. Complexity management will aid the early identification of risks. Conventional risk management techniques are well described in literature and risk and project management methodologies. Rather than define a particular approach to be used alongside complexity management only the few relevant points will be discussed. A template of the risk register, the artefact ultimately produced by any risk management process, is contained within Appendix M. 6.2. Important concepts The accommodation of risk management alongside complexity management should not present too much additional effort. Two of the components of the process are particularly important. The classification of risks can be undertaken using a pre-determined risk breakdown structure or RBS. The RBS is a hierarchical structure following the broad principles of other breakdown structures such as the WBS. The RBS contains successive levels of decomposition and there are as many levels to the RBS as is useful for the purposes of risk management. An example of high-level risk categories within a RBS are those of ‘schedule’, ‘resource’, ‘technical’ and ‘licensing’ [91]. Below this other risk category would decompose risks still further. This approach ensures consistency of risk category assignment throughout the project and allows the tracking of trends. These categories can then be further sub-divided as required. Other options include categorising by function or discipline. The goal of such an exercise is so ensure the consistency of categorisation and to allow trending of risks to show where risk particularly resides within the development. The artefact that is ultimately generated by the risk management process is the risk register. An example of the attributes that can be assigned to each risk is shown in Appendix N. The risk matrix; against which risk likelihood, impact and priority
- 50. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 46 of 69 are determined; is highly dependent on the magnitude and type of development as well as the organisation’s risk tolerance. As such this should be defined for each organisation or development process. 7. Complexity orientated development framework 7.1. Integrating the sub-processes together All the themes discussed in the previous sections can be gathered together into a single process. The process map, shown in Figure 18, represents interactions with the important process activities that are commonly undertaken in projects and technical development. It will also show the relationships between complexity assessment, determination of CSFs, detailed planning, definition of performance measures and risk. The complexity management framework fulfils and expands upon the high-level objectives described on page 8 in Section 1.6 in the following ways: 1. A top down assessment of complexity and measures to best manage it; a. Derive a strategy including proposed frequency and depth of detail during system development lifecycle; b. Undertake the complexity assessment at available or appropriate level of composition; c. Record areas where complexity assessment is considered unnecessary; d. For complexity ‘hotspots’ derive CSFs; e. Ensure that the relationship between complexity, the derived CSF and anticipated outcomes are articulated for later review; f. While considering CSFs in place review complexity to ensure that hotspots have not been simply displaced to another theme or aspect; i. Make appropriate changes to assessment and CSFs as necessary; ii. Repeat if absolutely necessary; iii. Determine frequency of review; 2. A Bottom-up assessment of complexity using information derived from planning exercises; a. Derive strategy, rules and conventions for planning through DSM; b. Apply dependency planning technique to areas of development as appropriate; c. Develop DSM and MDM to required level of detail; d. Assess complexity of DSM and compare with complexity assessment findings; 3. Application of dependency planning techniques against areas of particular complexity; a. Determine level of decomposition and type of DSM to be employed; b. Identify, analyse and display DSM, optimising the DSM as required; c. Feed outputs of DSM into creation of Gantt chart schedule; d. Determine frequency of review; 4. Application of performance measures according to type and depth of complexity; a. Determine ‘goal’ of performance measures and ‘questions’ to be asked and answered by them depending on complexity; b. Define metadata that is required to enable performance measures to be calculated in areas such as requirements management and development artefact production; c. Determine mechanisms to collect metadata including IT requirements d. Determine frequency of the collection of performance measures as dictated by nature of measure and requirements of development; 5. Contribute to risk management; a. Determine how risks identified by complexity management can be integrated into risk management strategy; b. Derive RBS to allow consistent categorisation of risks; c. Identify risks through determination of CSFs and planning activities; 6. Repeat activities at predetermined intervals.
- 51. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 47 of 69 The individual activities upon the process map will be colour coded to reflect whether they are those currently undertaken in technical development and unaffected by complexity management (grey) and those proposed within this framework (green). An additional coloured activity will show which activities currently exist but would be modified under this framework (amber). The first activities are the definition of strategy and conventions for the various sub-processes within the framework. This includes any criterion that is required to allow the Stage Gate to be passed. It is envisaged that the initial complexity assessment will be prompted by an early Stage Gate type decision point in the development lifecycle or the beginning of a development phase. A pre-requisite for the complexity assessment is the existence of a high-level WBS with higher levels of decomposition allowing more detailed analysis of complexity. Subsequent complexity assessments will be to a higher level of decomposition within the WBS. Each WBS element will be assessed against the following complexity themes within three categories: 1. Internal factors a. (Project) environmental constraints; b. Development process; c. Internal organisation; 2. External factors a. Contractual management; b. Stakeholders; c. Regulatory interfaces; 3. System development a. External (system) interfaces; b. Technology; c. Internal (system) interfaces. Each of the themes will be considered against each of five Complexity Criteria and each of these criteria will in turn be rated from very low to very high. This process was repeated for each of the WBS elements to the level of decomposition required. Any complexity theme that cannot be rated will be assigned not applicable. A template for the assessment is contained within Appendix B. The complexity assessment will be subject to additional iterations following the definition of CSFs until the WBS is fully assessed. This will include any changes prompted by identification of a CSF that may inadvertently and unknowingly increase complexity in any part of the development. The complexity assessment will also be subject to iteration at the beginning of each development phase or, for long development phases, at predetermined intervals of say six months. The identification of appropriate CSFs will be the next activity immediately after the complexity assessment is undertaken. The first consideration is the areas of the development that are likely to require more detailed planning and closer monitoring of progress and status. CSFs are then assigned according to areas of vulnerability due to particular complexity within the WBS. The list of CSFs is shown in Appendix C. This should be amended and updated over time according to lessons learned and the demands of the development type or organisation. This will be aided by reflection on the efficacy of the CSFs at the end of the development phase or before the next complexity assessment, as shown on the process map. Activities prompted by the generation of CSFs will include areas of the development that would benefit from the application of DSM and both the goals of monitoring and questions that should to be asked and answered by the performance measures. Consideration of CSF will facilitate the identification of risks. Development risks should be consistently categorised and a RBS will need to be produced to ensure this is done.
- 52. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 48 of 69 Figure 18. Complexity Management Framework.
- 53. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 49 of 69 7.2. Analysis of framework against criteria within existing literature In this section the complexity management framework is compared with the recommendations and criteria for successful projects as discussed within the literature survey and from a wide variety sources. Any obvious deficits are discussed and additional measures proposed that may benefit the future development of the complexity management framework. Other recommendation should be satisfied by activities outside the scope of this framework. The 2012 ‘thisiswhatgoodlookslike’ Gartner survey [92] made five high-level observations relating to high-level project strategies. The first concerned the size, complexity and duration of individual projects. Determination of size and duration, and to a lesser extent the complexity, of the overall project is very often outside the control of the technical development and indeed of the project itself. Complexity can be controlled once it is recognised and at the very least prevented from growing unduly. The survey describes that a project must ‘stay on top of costs’ and describes that there should be measures for early identification of cost overruns. This can be achieved through the adoption of leading performance measures that can identify delays to the development that inevitably lead to additional costs due to acceleration of the schedule. The potential for rework can also be highlighted which again can have significant associated costs. A realistic schedule can best be achieved through better planning. The targeted use of DSMs identifies coupling behaviours and dependencies to allow both optimisation of the schedule and improved representation of the schedule when used as an input in the production of Gantt charts. Using the measures previously described it does at the very least provide an early warning of discrepancies between early high-level schedule commitments and likely actual schedule. The fourth of the Gartner survey’s recommendation related to stakeholder and system requirements and the funding that the full realisation of these dictates. Requirements management is not within the scope of this dissertation. However, the assignment of requirements related performance measures and enhanced planning alongside project management of the budget should assist in this. Additionally, effective and rigorous requirements management would very likely be a development CSF which could then be monitored over time. The last recommendation refers to frequent ‘project status and review meetings’ and ensuring alignment of requirements with the business case. Better use of performance measures, as discussed within the dissertation, should facilitate this. Ultimately the identification of a failing project, unlikely to satisfy the business case, may lead to its cancellation. The Royal Academy of Engineering’s 2004 ‘The Challenges of Complex IT Projects’ [93] also had five high-level findings. ‘Lack of constraints’, relating to the definition of reasonable requirements, is outside the scope of this framework. Again this can be partially mitigated through effective planning and performance monitoring. Complexity assessment should highlight where complexity is highest which may indicate unrealistic expectations. This would apply especially to the aspects of environmental constraints and technology, especially where technological demands may be unduly constrained. Similarly, ‘Visualisation’ and ‘Flexibility’ will be again addressed by effective requirements management alongside the framework’s emphasis on performance monitoring. The impact of changes can also be readily assessed in portions of the development planned using DSM. High-impact change requests can then be identified and approvals made accordingly. Lastly both ‘complexity’ and ‘uncertainty’ can be assessed early using the top-down complexity assessment and later, using bottom-up as DSM planning is undertaken. CSFs and detailed planning can both reduce and manage both these properties along with the inevitable changes to development requirements and scope. Denker [94] cites the components of ‘Colossal Complexity’, ‘Invisibility’, ‘Over-Optimism’, ‘Extreme Uncertainties from the Kickoff’ and ‘Rework’. Mirroring the factors mentioned previously, these can be managed through the recognition of development complexity through complexity assessment, determination of ways to manage it through identification of CSF and through better planning. The paper specifically mentions the imbalance between complexity and imposed timescales. DSM should more completely identify coupling and emergence that are features of complex schedules. Other themes are the lack of timely feedback
- 54. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 50 of 69 mechanisms and the identification of cause and effect. Both of these are addressed through the targeted application of performance measures. Lastly the reuse of development patterns can be partially addressed through the complexity assessment, assignment of CSFs and the review of their efficacy for future use. While ‘control mechanisms’ and ‘feedback capabilities’ are directly addressed through the use of performance measures, the remainder Jiang’s 13 [95] should be addressed through general management arrangements. Deficits in Gantt chart schedules, as previously discussed, can be addressed through the use of targeted planning and specifically through the use of DSMs and appropriate performance measures. In it critical activities that are outside of the critical path can be identified and coupling behaviours better represented. The use of appropriate leading and lagging performance measures can measure the quality of processes and its outputs. The high degree of ‘information requirements’ associated with planning are a function of Program- size Complexity. Though it may be assessed it can only be managed, through appropriate resources and technology, and not reduced. Limited modelling of the ‘behavioural aspects of management’ may be represented using numerical type organisational-architecture DSM or process-organisation MDM. This does not preclude the use of other modelling techniques, such as OBM, when deemed appropriate. 8. Case study application of framework 8.1. Purpose In this section we evaluate the framework by using it to assess an existing complex project using publically information. The case study uses publically available information on the development of Boeing’s Dreamliner and will illustrate the use of the framework and identify areas of potential concern that require process improvement or additional guidance. The WBS will necessarily be at a high-level/strategy level due to constraints of information availability and the effort required to undertake analysis relating to detailed WBS structures. As such it will be assumed that the assessment is being undertaken during the concept design phase and used to analyse development up to the point of manufacture. The initial complexity analysis will be followed by suggested CSFs and recommendations requiring areas of planning, performance measurement and risk management. The assessment will be undertaken from the viewpoint of Boeing’s Dreamliner design team and was chosen due to the well published development and operational issues [96][97][98], sheer quantity and variety of literature, interesting characteristics and first-of-a-kind complexity. The findings will be compared to the outcomes and assessed for efficacy and it will demonstrate the process, albeit at a low level of detail. An additional case study, using the construction of a nuclear power plant, is contained in Appendix Q. This project is distinctly different from the Boeing Dreamliner and identifies other aspects of the framework requiring development but also demonstrates how the framework can be implemented. 8.2. Case study 1 – Boeing Dreamliner 8.2.1. Background In the early 1990’s Boeing required a new commercial aircraft to replace the aging 767. Two options were initially investigated; a sub-sonic cruiser with a speed of 0.98mach and similar fuel efficiency as the 767, and a design similar to the 747 with increased capacity, designed to directly compete with the Airbus A380. Neither was particularly well received by the commercial airline markets and when fuel costs escalated after 9/11 the requirement for fuel efficiency became paramount. This led to the concept of the 787 Dreamliner in 2003 [96]. The requirements for the Dreamliner were as follows: Lower fuel consumption; Lower maintenance costs; Longer range; Improved passenger comfort. This necessitated the use of technology not previously used extensively within commercial airliners. This included advanced composite use, eventually resulting in more than 50% by weight [97], and the replacement of pneumatic and hydraulic systems with electric architectures [98]. Additionally, the fuselage was to be
- 55. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 51 of 69 manufactured in ‘barrel sections’ and integrated later rather than being assembled by Boeing. These design decisions were undertaken early in the development cycle for reasons of overall weight and reduced system power requirements. Boeing’s initial estimate for the development was $6 billion over a four-year period, representing a significant investment Manufacturing costs for the Dreamliner were also a significant factor in order that it was to be both competitive and allow Boeing to make a return on its investment [99]. This prompted the adoption of a contracting strategy that was designed to make efficiencies in the development and manufacture of the aircraft but spread the commercial risks [100]. It also allowed market expansion by allowing the manufacturers in the countries that would buy the Dreamliner participate in the project in ‘offset deals’ [100]. Contractual arrangements allowed Boeing to delay payment to partners until aircraft were delivered [97], including risk-sharing partnerships, with payments depending on final outcome. Manufacturers were only paid when the plane was certificated and delivered, thus Boeing avoided bearing many of the up-front non-recurring research and development costs, encouraging efficiency in the supply chain and could delay investment until later in the project lifecycle. In addition to higher returns on successful deployment of aircraft the manufacturers also were allowed to retain intellectual property rights of their contribution to the aircraft [100]. ‘Partner councils’ were established to aid the creation of an ‘integrated, integrated, modularized supply-chain’ and mirror the practices used in military and business aircraft sectors [97]. The use of globally based suppliers ensured that the necessary expertise could be sourced and also allowed the development of many systems in parallel. 8.2.2. Work Breakdown Structure To meet the considerable technological challenges and cost efficiencies it was decided that design development and manufacturing, including research and development, of most of the aircraft would be outsourced across the globe, to Tier 1 suppliers. The contracts were designed to allow Final assembly would be undertaken by Boeing in the US. Ultimately this resulted in 70% of parts being manufactured by 50 companies in 28 countries [98]. The Tier 1 suppliers would in turn contract out manufacture and services to Tier 2 suppliers as appropriate. A WBS using system engineering and project activities has been produced decomposing the aircraft into systems and structures using the not-dissimilar Hyperion project [101] as a basis. Hyperon was chosen because it is within the aviation domain (unmanned aerial vehicle in this case) and has been developed internationally. This WBS has been adapted to include general commercial aircraft systems [102] the Dreamliner’s more detailed ‘Global WBS’ [103]. This gives an emphasis to the technology and contracting components of the Dreamliner project though common design phases have been included including a research and development phase that was undertaken by both Boeing and its Tier 1 partners. It is assumed that research and development would be undertaken after the concept had been agreed, allowing the development of a business case and the commencement of marketing and sales, but before scheme design. Based on the Dreamliner’s history the concept would have been available in late 2003/early 2004 before the first order with Nippon Airlines was agreed. An alternative would have been to decompose the project with a higher emphasis on functions and processes using a combination of system engineering and project management methodology. It should be noted that the contents of the WBS given below is not exclusive, though it is complete enough to allow early analysis. As can be seen the level of detail increases through the project’s phases as the aircraft’s systems are defined. Design Concept; o Aerodynamics; o Mass properties; o Composites; o Structures; o Fuselage; o Wings; o Doors; o Stabilisers;
- 56. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 52 of 69 o Landing gear; o Propulsion; o Controls; o Electrical systems; o Communications; o Navigation; Research and Development; o Structures; Centre wing box; Wing box; Wing to body fairing; Wing to body fairing components; Main landing gear well; Nacelles; Fin; o Fuselage; Forward fuselage; Mid fuselage; Rear fuselage; o Wings; Wing structures; Leading edges; Engine pylons; Fixed trailing edges; Moveable trailing edges; Fin leading edge; Rudder; o Doors; o Stabilisers; Horizontal stabilisers; o Landing gear Landing gear structures; o Propulsion; Engines; Gearboxes; Fuel; o Controls; Flight deck; Flight controls; Speed reference system (SRS); Autopilot; Electronics; Engine controls; o Electrical systems; o Communication; o Navigation; Scheme (similar to R&D and not expanded); Detailed (similar to R&D and not expanded). Manufacture (not expanded) Integration (not expanded) Certification (not expanded) Test and commission (not expanded) Management Project management; Financial management; Quality assurance; Legal; Shipping.
- 57. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 53 of 69 8.2.3. Complexity assessment The timing of assessments should be aligned with project phasing and project and development governance processes, including stage-gates and major design reviews. For the purposes of this case study a high-level assessment will be undertaken at the beginning and end of the design phase to highlight how the aspects of complexity evolve and illustrate some potential issues. An assessment was completed against the high-level WBS at concept, research and development and detailed design phases. These can be seen in Appendix O. Concept design During this phase it is assumed that the contracting strategy has been agreed in tandem with the development of the overall concept. Environmental constraints: Uncertainty and Ambiguity have been assigned as very high as contracts are still be let to Tier 1 partners under the pain/gain sharing arrangements and the effect on funding is uncertain. Emergence, Non-linearity and Program-size are low at this early point in the project. Development process: Both Uncertainty and Ambiguity have been assigned as very high, recognising the new way of working on this project, and thus immaturity of processes including the unexpected interactions with the Tier 1 suppliers. Emergence, Non-linearity and Program-size are low at this early point in the project. Organisation: Boeing’s organisation is likely to be smaller than on other comparable projects due to the Dreamliner’s outsourcing arrangements but as this is a novel development there will be a high level of Uncertainty and Ambiguity as to how integration will be achieved. Emergence, Non-linearity and Program-size are low at this early point in the project. Contractual management: There is considerable scope for error due to factors such as number of suppliers, use of Tier 2 and 3 suppliers outside direct control of Boeing, location and time zones and cultural and political factors. Uncertainty and ambiguity have been assigned very high due to the enhanced potential for change or conflict. Emergence, Non-linearity and Program-size are low at this early point in the project. Stakeholders: In this instance the stakeholders are largely the prospective customers. As such change relates to Uncertainty and Ambiguity in user requirements and these have been given a medium as there is scope for changes. Emergence, Non-linearity and Program-size are low at this early point in the project. Regulatory interfaces: Due to the technology levels the regulators’ responses may be highly uncertain and ambiguous and especially at the beginning of the development. In many areas the regulators may lack essential competencies to effectively review and agree key principles within the design. Emergence, Non-linearity and Program-size are low at this early point in the project. External Interfaces: This aspect relates to entities such as communication, maintenance and the individual airports at which the Dreamliner would land or take-off from. The use of relatively novel technology creates some uncertainty but changes should be relatively easily accommodated and interfaces relatively simple and understand. Technology: Both Uncertainty and Ambiguity with regards to technology is very high during the concept phase. Emergence, Non-linearity and Program-size are low at this early point in the project. System Integration: Changes to system integration could arise from both the technology characteristics and will be very high before the outsourcing arrangements are fully known. E mergence, Non-linearity and Program-size are low at this early point in the project. Research and Development At the beginning of the research and development phase it is assumed that all the design and manufacture packages have been let to the global Tier 1 partners. Environmental constraints - Boeing’s contractual arrangements with its suppliers were designed to share the pain/gain of the final outcome and shield Boeing from the potential variation in supplier costs. As such Uncertainty and Ambiguity have been assigned a medium rather than a high value which might have been
- 58. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 54 of 69 expected for such an early point in the project. Contracts have been let for the R&D effects of Emergence and Non-linearity resulting from imposed changes has been assigned medium rather than a low. Program- size Complexity is low in terms of environmental constraints. Development process - Both Uncertainty and Ambiguity have been assigned a high value, recognising the new way of working on this project, and thus immaturity of processed including the unexpected interactions with the Tier 1 suppliers. Communication with the Tier 1 suppliers will be especially critical to ensure requirements of the Dreamliner’s component parts and systems meet requirements and can be assembled. Emergence and Non-linearity have been assigned a medium value as there is potential that necessary changes to processes will be significant enough to compromise schedule and cost planning estimates. Program-size Complexity has been assigned high to reflect the processes interactions between the various Tier 1 suppliers. Organisation - Boeing’s organisation will be smaller than on other comparable projects due to the Dreamliner’s outsourcing arrangements, with an emphasis on integration and assembly over design and manufacture. Uncertainty and Ambiguity has been assigned a medium value to reflect the potential for changes that may result from changes in the development process described above. Emergence, non-linearity and Program-size Complexity have been assigned a low status due to the relatively low size of the design team. Contractual management - There is considerable scope for error due to factors such as number of suppliers, use of Tier 2 and 3 suppliers outside direct control of Boeing, location and time zones and cultural and political factors. Uncertainty and ambiguity have been assigned a very high value due to the enhanced potential for change or conflict while Emergence and Non-linearity have been given a medium to reflect the early stage in the development. Program-size Complexity has been assigned a high value due to the number of organisations and scope of their management. Stakeholders - In this instance the stakeholders are largely the customers as suppliers, project team and regulators are included within the other aspects. As such change relates to Uncertainty and Ambiguity of user requirements and these have been given a low value while Emergence and Non-linearity have been assigned a medium value. Changes in the enhanced requirement for fuel efficiency, range, maintenance cost and comfort are not thought likely but would have a significant impact if they came about. Program-size Complexity is seen to be low. Regulatory interfaces - Due to the technology levels the regulators’ responses may be highly uncertain and ambiguous and especially at the beginning of the development. In many areas the regulators may lack essential competencies to effectively review and agree key principles within the design. This will be exacerbated by the outsourcing arrangements. Emergence and Non-linearity has been assigned mediums while Program-size Complexity has been given a low to reflect the understanding relative simplicity of the regulator interfaces. External Interfaces - External interfaces relates to entities such as communication, maintenance and the individual airports at which the Dreamliner would land or take-off from. The use of relatively novel technology creates some uncertainty but changes should be relatively easily accommodated and interfaces relatively simple and understand. Technology - Both Uncertainty and Ambiguity with regards technology is naturally high at the beginning of an R&D phase. However much of this technology has beenpreviously used within other domains and applications. Emergence and Non-linearity have been assigned a medium as there is potential that changes will be significant enough to invalidate the concept design. Program-size Complexity has been given a low at this point in time. System Integration - Changes to system integration could arise from both the technology characteristics and the outsourcing arrangements that have been put in place. As such both Uncertainty and Ambiguity have been assigned a high status. Changes could have a significant effect on the concept design even at this early stage. Lastly the interfaces required for integration are onerous due to the outsourcing arrangements.
- 59. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 55 of 69 Scheme design At the beginning of the Scheme Design phase it is assumed that R&D is complete and the contracting arrangements are well embedded. Environmental constraints - These stay broadly similar to those identified in the R&D phase though with an increase in Program-size Complexity. Development process - Both Uncertainty and Ambiguity have been assigned a medium, recognising the embedding and increasing maturity of the project processes. Changes required to the development process will have a greater impact in terms of Emergence and Non-linearity so are given a high. Likewise, the process will become more complex to manage in terms of number of components. Organisation - This aspect will stay relatively stable with changes to the organisation having a greater impact as the Boeing design activities increase in scale. Contractual management - All factors are considered high due to the potential for change, the impact on other contractual arrangements and the number of contracts to be managed. Stakeholders - The impact of stakeholders stays relatively stable though changes in customer requirements or the markets could have an increased impact. Regulatory interfaces - Confidence in the technology and contractual approach should be increased though changes, brought about by regulatory changes and imposed constraints, will increase in impact. External Interfaces - The impact of external stakeholders stays relatively stable though changes could have an increased impact. Technology - Both Uncertainty and Ambiguity with regards technology will reduce and impact increase. Technology will take more management as its scope is defined in more detail. System Integration - In a similar fashion to contractual management, integration will increase in all areas. Detailed design At the beginning of the Detailed Design phase it is assumed that scheme design is complete and most issues regarding management of the design phase are becoming apparent. Environmental constraints - These stay broadly similar to the R&D phase though with an increase in Program- size Complexity. Development process - Both Uncertainty and Ambiguity have been assigned a medium, recognising the embedding and increasing maturity of the project processes. Changes required to the development process will have a greater impact in terms of Emergence and Non-linearity so are given a high. Likewise, the process will become more complex to manage in terms of number of components. Organisation - This aspect will stay relatively stable with changes to the organisation having a greater impact as the Boeing design activities increase in scale. Contractual management - All factors are considered high due to the potential for change, the impact on other contractual arrangements and the number of contracts to be managed. Stakeholders - The impact of stakeholders stays relatively stable though changes in customer requirements or the markets could have an increased impact. Regulatory interfaces - Confidence in the technology and contractual approach should be increased though changes brought about by regulatory changes and imposed constraints, will increase in impact. External Interfaces - The impact of external stakeholders stays relatively stable though changes could have an increased impact. Technology - Both Uncertainty and Ambiguity with regards technology will reduce and impact increase. Technology will take more management as its scope is defined in more detail.
- 60. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 56 of 69 System Integration - In a similar fashion to contractual management, integration will increase in all areas. 8.2.4. Critical Success Factors The CSFs will evolve throughout the development though thought should be given to putting them in place in advance of when they are particularly required and the phasing required. CSFs will be ranked and generally taken from ‘high to very high’ rated CSFs from the full survey dataset Appendix F. CSFs within the top three identified from each Complexity Theme will be shown in bold with the chosen CSFs in order of priority. The full dataset was chosen in preference to that from the aviation industry dataset due to its low sample size. Where the number of CSFs are overwhelming in number it is expected that these can be further ranked in importance. CSFs assessed as having the greatest impact will be chosen in preference to the others. Furthermore, it would also be expected that performance measures should be chosen to demonstrate the implementation and impact of CSFs. Examples of how these CSFs may be further tailored towards the Dreamliner project or otherwise supported are provided in italics with top three ranked CSFs identified in bold Concept design At this point in the development lifecycle the emphasis is on managing uncertainty. At this stage ambiguity will also be high to the technology and there may be constraints of funding prior to a commitment to long- term investment. Particular Environmental constraint related CSFs are as follows: I. Clear realistic project objectives – planning undertaken with minimum participation of a defined percent within the supply chain. Plan complexity measured and managed through use of DSMs; II. Composition of project team in terms of experience and capability – minimum contractual pre- qualification criteria for Tier 2 suppliers, Process-Organisation MDMs implemented. The development process will need to tailored to suit the particular challenges likely to be faced and also manage the uncertainty and ambiguity that will be present. There may be the opportunity to use methods from other domains or industries such as the design of business aircraft in with respect to the use of composite materials [104] or manufacture of electric cars with respect lithium ion batteries. Development CSFs may be: I. Critical activities are identified – Process DSM applied in areas of high complexity; II. Performance measures tailored to monitor areas of criticality and uncertainty – DSM related Performance Measures are implemented where required. The organisation would also need to cope with uncertainty and ambiguity relating to the new technology and could be assigned the following CSFs. i. Transparent definition of responsibilities - Process-Organisation MDMs implemented; ii. Composition of development team in terms of experience and capability – as above. Contractual management would be in its infancy at this point and would relate to the production of contractual documentation and selection of potential Tier 1 partners. In terms of uncertainty and ambiguity the following CSFs are important. i. Clearly understood contractual interfaces - Process-Organisation MDMs implemented; ii. Project breakdown into logical packages – complexity considered before allocation. However, this is one of the most critical aspects of the development and decisions made at this point and written into the contracts will have far reaching consequences. For this reason, CSFs need to be considered for the entire project lifecycle at this point in time. Neither the management of stakeholders or external interfaces are the project’s most challenging aspects and no CSFs should be emphasised. Regulatory interfaces are subject to some uncertainty due to the new technology and it treatment by the various certification authorities across the globe. Early and thorough engagement would be vitally important. i. Clearly identified and understood interfaces – MDM developed to model interfaces; ii. Clear lines of communication with regulators.
- 61. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 57 of 69 Technology is one of the most important aspects of the development and while choosing mature technologies is not an option the simplification of design should still be sought where possible to reduce uncertainty of outcome and understand it where at all possible. iii. Pursuing a simple as possible design - Requirements measurement is adopted; iv. Test early, test often philosophy is used during development – TRL measurement is adopted; System integration is the third important aspect, along with contractual management and technology, and should similarly be treated to reduce uncertainty and ambiguity. Some of the other CSFs should also be considered though these will not realise benefits until far into the development lifecycle. Those relating to Concept Design are: I. Test early, test often philosophy is used during development - IRL measurement is adopted; II. System element maturity is monitored - - IRL measurement is adopted. These two elements were taken from the ‘energy generation’ high influence CSFs as the full dataset did not provide any relevant CSFs for the issues encountered. This is possibly a weakness in this Complexity Theme’s CSF that could be bettered through increased sample size and increased representation from engineering personnel. Detailed Design As the development progresses the emphasis changes from Uncertainty and Ambiguity to the impact of change through Emergence and Non-linearity. While the reduction in the former are still important it is now the effect of Emergence and Non-linearity that will be felt most keenly. Alongside this, Program-size Complexity will also increase as the workload and number of parallel and dependant activities increases. As such the CSFs will become more about the plan and managing change to control impact. Important too is the managing the outputs of the design process and methods of working to manage the process as a whole. Considering Environmental Constraints first: i. Strong project sponsor/champion - use of a development health type Performance Measure; ii. Effective change management (project) – measurement of change control metadata is adopted; iii. Proactive risk management process – risk undertaken as a part of complexity management. For the Development aspects some if not all of the following would be applicable to ensure changes were recognised early and readily assessed for impact. Some obviously overlap in their scope. i. A well understood and mature design review process is in place; ii. Technical risks management process – risk management undertaken as a part of complexity management; iii. Effective technical change management processes – measurement of technical change control metadata is adopted; iv. Development and project plans are properly integrated; v. Effective monitoring/control of requirements and development deliverables. The organisation would need to consider the multi-organisational working methods and global span of the project to ensure timely and optimised decision making. i. Degree of collaboration – use of a development health type Performance Measure; ii. Coherent, self-organizing teamwork - use of a development health type Performance Measure. Similarly, contractual management should consider how the process and outputs can be best managed and as discussed above it is important that these are considered at the time of contract production. i. Clearly understood contractual interfaces – Organisational DSMs are adopted; ii. Good performance by suppliers/contractors/consultants; iii. Effective monitoring/control – Development Scorecard approach is adopted. Stakeholders have been considered as having a high impact on the development at this stage of the design.
- 62. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 58 of 69 Changes in requirements can have a serious effect on the project lifecycle. Specific CSFs surround the effective and timely communication and management of stakeholders. i. Client/user acceptance– Requirements measurement is adopted; ii. Decisions are agreed and documented; iii. Early identification and management of conflicting interests – Requirements measurement is adopted; iv. Active management of client/user integration – MDMs adopted for critical systems. External interfaces, while important, are not viewed as being as critical as other areas. Regulatory interfaces, technology and Integration have the potential to have an increased impact though a lower likelihood of change as the development proceeds and technology and design matures. As such the CSF will remain the same as there is little that can be done to reduce the impact of emergent change other than identify it early. 8.2.5. Planning technique The sheer size, scale and criticality of the project would lend itself to the use a combination of Design Structure Matrices. These could be targeted against particular areas of development and interfaces based on risks borne from the constraints of schedule and relative criticality, both a technically and in relation to the overall project. From the Criticality Assessment the three areas that deserve particular attention are management of Tier 1 suppliers, technology and integration. All have particular challenges arising from the project and development strategies: Organisation architecture DSM to consider how Boeing and the Tier 1 partners taking into account all interdependencies and constraints arising from their diverse geographical locations; Process architecture DSM to focus on particular groups of activities from the WBS. This might particularly look at integration or assembly type activities and dependencies between Tier 1 suppliers; MDM to look at particular parts of the development and be used to plan the interaction between organisations and show the impact on the wider schedule. Full integration of traditional scheduling and DSMs would allow a seamless blend of the two techniques to be used with areas of contractual interfaces, technology centric development and integration activities being prime candidates for binary MSMs and numerical DSMs. Further Complexity Assessment of the WBS could be taken at the next lowest level or levels to identify areas of particular complexity. From the project’s background information such candidates for enhanced planning include the following: Electrical systems, including the lithium-ion batteries [105]; Composite material development and especially construction of barrel sections of fuselage; Design integration activities; Aircraft assembly including Tier 1 partner dependencies; Certification related activities, such as quality assurance, across the entire development. Enhanced planning would be chosen in other areas, depending on the WBS and on the most recent Complexity Assessment findings. The scale and length of the development may justify specific tool support to be developed. 8.2.6. Performance measurement To meet the challenges of contractual management, technology and integration the following measures could be combined into a Development Scorecard across the entire project: Conformity with the plan consisting of: o Earned Value Management Cost Performance Index (lagging); Schedule Performance Index (lagging); System maturity consisting of:
- 63. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 59 of 69 o Technology Readiness level (lagging); o Integration Readiness level (lagging); Process Health to measure areas such as leadership and collaboration. The outsourced nature of the project makes collecting information challenging and quality assurance of Tier 1 supplied information would be required. Other measures could be applied, with discretion, in areas of particular concern and these include: Plan complexity consisting of: o Proportion of sequential, parallel, coupled and conditional activities within DSM (leading); Requirements consisting of: o Realisation – relating to MOE, MOS, MOP and TPM (lagging); o Execution – relating to selected requirements metadata (leading). Such data would need to be trended over time rather than compared with data over the entire project and would be used where there are particular technical risks or evidence of poor project performance. This would provide a mix of leading and lagging measures the purposes of monitoring, control and reporting. 8.2.7. Risk management The identification of development risks will not be undertaken in detail due to lack of data. It is not difficult to appreciate that there will be significant risks in the areas of contractual management, technology and integration. Detailed descriptions, risk responses and related information would naturally fall out of more detailed analysis and planning activities. The creation of a hierarchical Risk Breakdown Structure would be an important early activity in the formation of the risk management strategy. One suggestion for the high-level nodes (level 1) of the RBS could be those of ‘schedule’, ‘resource’,’ technical’ and ‘certification’. Level 2 nodes would include contracting and Boeing (below resource) and technology and integration (below technical). Certification could be divided amongst particular systems within the aircraft. With the high reliance on Tier 1 Partners it might be appropriate to also include this as a level 1 RBS node or else place it under resource risks alongside ‘Boeing organisation’. Similarly, the importance of technology and integration can be elevated to that of a level 1 node or else included under one or more of the other nodes. 8.2.8. Comparing framework findings with project outcomes Boeing’s Dreamliner has received a large amount of publicity due the prevalence of problems pre and post launch. A detailed time line of events can be found in Appendix P. Delays almost doubled the ‘launch-to-delivery’ time from 49 to 89 months [97] due to a variety of problems involving Tier 1 partners and the technology. Most notable were long standing issues with the supply of fasteners and a number of incidences involving the battery technology causing safety concerns [106]. Recognised risks included the management of the supply chain across different countries, such extensive use of outsourcing and assembly issues made worse by the extensive use of composite technology [97]. Costs spiralled to approximately $40billion, which was twice the original estimates and included costs due to defects, additional R&D, supplier support and buyout, customer contract penalties and cancelled orders [100]. This included bailing out Tier 1 partners such as Alenia and Vought who designed and manufactured the horizontal stabiliser and both central and rear fuselage. There has been much speculation, research and analysis as to the root causes for the issues described. Many relate to supply chain competence and Boeing’s own communication and procurement. Risk sharing neglected to hold individual suppliers to account for their actions and so encouraged them to benefit from savings on direct costs while the overall schedule was delayed at the same time Boeing placed a higher emphasis on costs over schedule. As such optimisation of aspects of development did not equate to overall optimisation, rather quite the opposite [100] Many of the additional costs thus related to Boeing requiring more effort than anticipated managing the supply chain. Post-launch problems were also significant in increasing costs but also
- 64. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 60 of 69 in damaging Boeings reputation and future Dreamliner sales. Significant safety events included a number of electrical fires relating to the batteries required for the aircraft’s electrical system. This has been attributed to quality control rather than strictly technology based issue [107] and as such reflects badly upon both Boeing’s control of its partners and the certification process [108] including the delegation for the provision of test data to the project’s suppliers [109]. Other operational safety issues included engine shutdowns, fuel leaks, loss of transponder, electrical faults, hydraulic failures, a cracked windscreen, cracks in wings discovered at the factory and a variety of on-board management systems [110]. All in all, there have been close to 100 reported incidents as of August 2016 [111]. The Complexity Assessment described particular areas of concern surrounding Contractual Management, Technology and Integration. It is fair to say that most of the issues encountered during the development of the Boeing Dreamliner fell into one of these categories. An additional case study describing the development and construction of AREVA’s Evolutionary Power Reactor in Olkiluoto, Finland (OL3) has been undertaken within Appendix Q. OL3, though very different to the Boeing Dreamliner, exhibits characteristics that are equally as complex. While it has similarities in the depth of technical complexity OL3 differs in Areva’s choice of procurement and contracting strategy. Whereas Boeing predominately contracted out design to Tier 1 partners, Areva undertook virtually all design in-house, creating its own unique issues. OL3 is again well documented and has a reputation well deserving of a case study. Both Boeing’s Dreamliner and Areva’s OL3 were briefly discussed within Section 1.5 of the Literature Survey [4]. 9. Results and Evaluation The dissertation has addressed the following objectives in an effort to develop a complexity management framework: Proposed a complexity assessment by decomposing a technical development by: o Discussing the rationale for complexity management alongside current project management methodologies and provide background to the complexity assessment proposal; o Decomposing development by the WBS; o Creating themes for each WBS element – (project) environmental constraints, development process, internal organisation, contractual management, stakeholders, regulatory interfaces, external (system) interfaces, technology and internal (system) interfaces; o Creating complexity criteria for each theme – uncertainty, ambiguity, emergence, non-linearity and programme-line complexity; o Rating of criteria between very low and very high; Describing how complexity may be profiled per criteria over the development lifecycle, how these criteria interacts and how change may affect this; Developing the concept of CSFs for application within the Complexity Framework; o To indicate where extra planning effort should be applied; o To direct where monitoring should be applied to a higher level of decomposition; o To align with areas of high complexity deserving interventions and management effort; o Testing and ranking the selection of CSFs through questionnaire using project and engineering professionals; Proposing the use of Design Structure Matrix to supplement and be used as an input into the production of Gantt chart schedules; o Describing the technique as it is already used for both process and organisational-architecture DSM and Multi Domain Matrix; o Proposing the rationale for the use of binary, numerical and MDM; o Aligning DSM with complexity criteria and use it to derive planning indicators and performance measures;
- 65. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 61 of 69 Propose the effective use of different types of performance measures to aid monitoring of development activities; o Discussing the varying types and purposes of performance measures to provide a way of accurately representing status to allow control and mitigating actions to be implemented; o Listing a number that have been previously proposed as suitable for use in this application; o Testing the proposed Performance Measures by questionnaire survey to indicate those that are considered most influential and to provide confidence in the measures selected; o Proposing a development scorecard to consist of a number of the performance measures to provide a balanced way of measuring status across the breadth and depth of a technical development; Describe how the entire process may inform risk identification as a part of the overall development risk management; Gather all these elements into the complete complexity management framework; Test the complexity management framework against two case studies. The dissertation has satisfied the objectives that were assigned within the introduction with varying degrees of success. The individual components of the Complexity Management Framework will be discussed for their strengths and weaknesses. Finally, areas of further research will be identified where they are appropriate. Complexity assessment The rationale for the assessment was derived from a combination of literature and the author’s personal experience. The use of the WBS as a focus provides a convenient way of decomposing the activities while the complexity themes allow the assessor to look at the WBS elements from a variety of important viewpoints. The complexity criteria decompose the actual complexity further and introduce uncertainty and ambiguity as discreet elements, which are very often barriers to good decision making. The criteria broadly follow the principles the business assessment methodology called VUCA [37], using classical definitions of complexity [37] and adopting with similarities to 5DPM [31]. The regime for scoring individual criteria is as precise as such a heuristic can possibly be though of course it is still open to a high degree of subjectivity. The assessment provides little in isolation but does prompt qualitative analysis of the development and the identification of areas of concern for assignment of CSFs. Together they show potential areas of complexity related risk and a description of the issues within the technical development. Difficulties arise in assigning criteria ratings that relative to the entire development lifecycle. For example, providing a ‘very high’ rating early in the development allows no scope for increasing the rating later on. This can be at least partially cured through the development of assessment guidance. Another area requiring guidance is that pertaining to System Development Themes, of which there can be seen to be some overlap. Practical application would be preferred method of refining analysis and improving consistency of application. Focussed further research could be used to aid lessons learned type exercises. The nature of Complexity Criteria was discussed and how these evolve during a project. Uncertainty/Ambiguity closely equates to likelihood of change while Emergence/Non-linearity related more closely to impact. In this way it can be seen that Uncertainty/Ambiguity is most influential in early phases of development while Emergence/Non-linearity becomes increasingly predominant as the development matures. Complexity Profiles could be developed to allow baselines against which similar technical development could be assessed. The use of case studies high-lighted a number of potential shortfalls in the Complexity Assessment. The WBS is an important consideration during the Complexity Assessment as inconsistent or a poorly conceived structure will prove problematic in the application of Complexity Assessment. Issues arise where the WBS uses inconsistent methodology and omits development phases at a high level as can be seen in the OL3 case study contained in Appendix Q. This makes Complexity Assessment more of a challenge at each point that it is done. In the example of OL3 the WBS will have been created early in the project lifecycle and presumably by a variety of business entities in parallel. In its favour it would have stayed largely unchanged throughout as it would
- 66. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 62 of 69 have been largely uninfluenced by contracting decisions, relying instead on established business functions for its structure. In contrast the Boeing Dreamliner WBS would have been dependant on the ‘Global WBS’ that was created as a consequence of how the aircraft was contracted out for development and manufacture. Later interventions by Boeing to support the supply chain and provide additional supervision and management would have also affected the WBS. Ideally early input into the WBS during initial complexity assessment would alleviate these issues. The case studies allowed an assessment of the technique but with the inevitable bias relating to knowledge of actual real-life outcomes and limitations of high-level application. However, they were able to demonstrate its use against a high-level WBS. Development of assessment against a detailed WBS would benefit from practical application to identify further areas for improvement. Critical Success Factors This is a technique already established in industry [5][7][45] though without any defined methods to define and implement. This dissertation was able to link it into a process designed to better guide its application and also relate the realisation of CSFs to Performance Measures. The literature survey was effective in providing a comprehensive list of CSFs. The validity of these CSFs was confirmed following the questionnaire survey with only a single CSF from the 141 being identified as having low influence. The ranking of CSFs allows them to be both prioritised when chosen for use and when applied in practice. Further development of CSFs will again rely on project data and particularly the review of past development performance. The questionnaire has a reasonable sample size that provides a good starting point for their application. Another area of development that became obvious while undertaking the case studies is the need to categorise the CSFs to match the Complexity Criteria to inform their selection. This would move the selection of suitable CSFs from being highly subjective to that of a semi-formal method. Each CSF would be considered against its potential to reduce or allow better management of Ambiguity, Uncertainty, Non-Linearity, Emergence and Program-size Complexity. Initially this may take the form of having have simple categorisations of positive, neutral or negative influence. There would then be the opportunity to develop this further using lessons learned from the use of CSFs. Foremost the application of CSFs would benefit from a commentary being given to each, proposing how it could be best implemented and under which circumstances. Ultimately the CSFs could be given a more complicated rating against Complexity Criteria. Candidates for this include a scale from high to low (with none and negative to denote where the CSF has no benefit) or a numerical scale with positive and negative integers. Identification of CSFs should not be based purely on the analysis of the current stage of development as omission of CSFs for particular aspects can have far reaching consequences. Examples include Contractual Management where the cognisance of CSFs at the time of contract documentation writing can influence strategy and content. Another example may be the consideration of verification and validation strategy early in the design where the management of uncertainty and ambiguity early in the design may overshadow that of managing the same later on. The application of these generic CSFs to make them effective within particular projects was not developed beyond some examples within the case studies. The application of CSFs could be applied in isolation of the other sub-processes described within this dissertation. As such there is much value in the findings of the questionnaire survey and their application even if complexity management is not adopted as a whole. There is however a significant benefit of the selection of CSFs being informed by a process such as Complexity Assessment. Planning using Design Structure Matrix While there is substantial literature on DSM and related techniques its use has yet to become commonplace. Exceptions include a few specialised application examples, such as Boeing’s Unmanned combat aerial vehicle (UCAV), Intel’s microprocessor development and BMW’s hybrid vehicle architecture concepts [67]. Evidently its use does not yet approach the level that will enable tool support and training to be made widely available.
- 67. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 63 of 69 DSMs can provide substantial value within planning development activities with complex relationships and it should be used to supplement, rather than a replace, currently used techniques. It is supposed that a barrier to its use is that of cost as an additional overhead, requiring additional resource and specialised skills to implement. Therefore, the benefits of DSM need to be demonstrated for industry to justify investment to develop and implement it. The section on DSM clarified the similarities between planning relationship descriptions between DSM and traditional techniques. It showed how the creation of DSMs could be used as an input into Gantt chart development. Finally, it proposed a number of Performance Measures relating to plan complexity. The need for such Performance Measures was confirmed by the results of the questionnaire survey which indicated that this was the third most influential, therefore desirable Performance Measure for use. The activity relationships sequential and parallel are commonly used within planning, the DSM process activity types of coupled and conditional are not. There is an opportunity to better define the interfaces between Gantt charts and both DSMs and MDMs. This would potentially lead to better tool support, including an interface with the most popular planning software such as Oracle’s Primavera [69]. Such tool support for transposition of activity relationships information from numerical DSMs into a Gantt chart format would improve the popularity of DSM as a planning tool. Tool development should not be confined purely to that of Process DSMs. Of equal benefit is the development of techniques and tooling to allow Organisational DSMs, and ultimately Process-Organisational MDMs, to support resource loaded Gantt charts. This would benefit the application of Earned Value management by improving the quality of the plan against which activities are to be monitored and controlled against. As well as providing information and data to support the implementation of EVM these techniques would support the implementation of plan complexity Performance Measures. As the third ranked most influential Performance Measure, behind both Requirements related measures, it should be considered as an important method of monitoring and controlling activities. Indeed, its value is that it is very much a leading measure of project performance. The benefit of using DSM and appropriate tooling is that the collection of these plan complexity performance measures can be largely automated. This allows these measures to influence the planning process itself in reducing complexity. It should influence the ultimate forming of the Gantt chart. It is also a leading measure that can be used to inform management of areas of particular concern against which resources and expertise should be expended. In common with CSFs DSM can be applied outside complexity management and without the proceeding Complexity Assessment and CSF identification. These sub-processes do facilitate the targeted and semi-formal method of applying DSM against areas of particular complexity. Again there is additional value in the application of DSM within this framework in that areas of low concern can be disregarded and planned using more traditional techniques. Performance measurement From the questionnaire it is obvious that there is a strong need for techniques to supplement EVM. EVM was used as a benchmark and rated relatively poorly and even amongst project personnel. This suggests there is considerable room for improvement in this area. Requirements management figured most strongly overall reinforcing the author’s view that the monitoring and control of requirements deserves greater emphasis within project management methodologies. The measurement of plan complexity, which could be satisfied using a number of DSM related measures, also figured strongly as third most influential. Requirements management relates closely to that of scope management since all scope should be derived from development and project requirements. This demonstrates there should be a lot stronger emphasis on the area of scope within the traditional project management model of the ‘Iron Triangle’ [32]. The measurement of schedule and cost, other than through EVM, received barely any attention within the questionnaire comments. In contrast, scope, both directly through plan complexity and indirectly through requirements was mentioned in the top three measures.
- 68. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 64 of 69 The use of TRL and IRL was regarded as being more influential in some industries than others and unsurprisingly was considered more important by engineering personnel. Incorporation of these methodologies would be of benefit in developments where novelty of technology or integration were identified as being of concern only. Both the measurement of sub-processes and project documentation, design and drawings should be considered for any project. Relating to the control and monitoring of scope, rather than the requirements, these are important lagging Performance Measures. Other measures scored poorly for reasons previously described. The adoption of a Development Scorecard type approach should eliminate biases that are inherent in any Performance Measure though the scope of performance measurement should be commensurate with the risks and the actual benefits. The adoption of these Performance Measures and the Development Scorecard can be readily applied in isolation of other supporting sub-processes. What Complexity Assessment and application of DSM do allow is the identification of where particular effort can be applied against the WBS. In this way particular areas of requirements and scope can measured. Risk management This is an area of management that is relatively well developed though its application can often be less than effective. The application of a Risk Breakdown Structure should aid consistency and application of the proceeding activities within this framework will aid the identification of risks. 10.Conclusion There is a general need to develop guidance and the formality of application across all of the sub-processes. This can be achieved through a blend of further research and real-world application. These areas for development will be discussed within this section. It is felt that application of Complexity Assessment against the WBS would benefit from further practical application. Further work should be undertaken to provide additional guidance to allow the application of the scoring of criteria and develop typical complexity profiles across the development lifecycle. These are outside the scope of this dissertation due to their reliance on actual project data and the sample sizes required. Application across successive developments may provide general trends which would be beneficial to allow the assessment of a development against norms. It may also support types of responses to complexity and inform high-level risk management exercises. The analysis of high-level trends to group Uncertainty/Ambiguity and Emergence/Non-linearity together would allow the formation of standard complexity profiles. This could then be used to guide initial project strategy and formation of the WBS as well as comparing the results of the complexity assessment against norms. This could then be used to assess where there may be particular risks, i.e. areas where complexity is excess or higher than expected. Only application of the Complexity Framework against different types of WBS and to varying degrees of detail can fully demonstrate its value and allow it to be improved upon. An iterative approach to developing the framework, and its constituent parts, will allow it to be applied more effectively over subsequent projects. Development of guidance would be of some importance to guide its use and attempt to drive consistency of application. Further research through larger survey sample sizes would improve the applicability of CSFs and Performance Measures within individual domains though this of course does not preclude their immediate use from the conclusions based on the survey data within this dissertation. Improvements could be made through increases in sample sizes across a range of industries, ensuring representation across age groups and role descriptions. In this way differences could be better seen between different industries and the ranking of relevant CSFs better tailored to the requirements of the particular industry. Furthermore, while the use of CSFs is
- 69. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 65 of 69 documented a process to aid actual identification, management and the review of their impact is not. Analysis of CSF efficacy would be highly useful and this sub-process provides a basis for such an activity that is not discussed elsewhere in literature. It is also felt more work is required in the Complexity Themes of ‘technology development management’ and ‘system integration management’. An increased sample size, including more engineering personnel, would improve the findings in these areas There is a deficit in the number of CSFs relating to the Criticality Themes under the External factors and System Development groupings. Neither research of literature or the questionnaire survey identified any additional CSFs so again it is felt only practical application of the framework would yield progress in this area. Guidance should be developed for the application of CSFs against particular Complexity Criteria. This could take the form of a simple categorisation of positive, neutral or negative influence to begin with. Additional instruction could be developed for the use of CSFs in the form of a commentary for each one. Guidance should also be developed in applying CSFs in specific technical developments. This should include appropriate wording and content but should also relate to the use of Performance Measures. Further work is required to develop suitable Performance Measures that will support the implementation of CSFs with maximum effort applied to the most influential CSFs. Many of these CSFs are intangible and subjective, though nonetheless important, making this task difficult. For some CSFs objective measurement will be impossible with examples including ‘good leadership’ and ‘degree of collaboration’. However, the identification of which CSFs can and cannot be measured would be useful in itself. Methods to assess those that cannot be measured could be undertaken through appropriate review at Stage Gates or similar development decision points. Examples of those that are readily measured include CSFs such as ‘critical activities are identified’ and ‘clearly understood contractual interfaces’. In these examples the use of DSM methodologies would be of assistance. The application of DSMs has been developed over several decades though commercially available tooling is weak. Development of tooling to allow DSMs to be used and be used as both an input into the development of Gantt charts and Performance Measures would be useful. Other areas of research include the development of numerical DSM methodologies, developed and improved over several development lifecycles, and its use in combination with other modelling techniques. OBM [41] has been referenced earlier in the dissertation but there are many other modelling techniques that may be more appropriate to integrate with DSM and traditional planning techniques. The practical application of complexity related performance measures against DSM would allow the proposed technique to be developed further. There is considerable benefit in such a leading measure and, considering its ranking in the questionnaire survey, its use may well be an attractive proposition within technical development. It is suggested that trialling of this, in conjunction with the application of DSM, is the next step in its development. The development of Performance Measures to support the use of CSFs deserves further research. The measurement of CSF effectiveness would encourage their use and be useful as an input into lessons learned exercises. This can be undertaken alongside general development of Performance Measures. Another aim is that of automation of the collection of data for the use in Performance Measures using existing a new computer tooling. This would reduce the manual overhead and encourage their greater use. The survey findings suggested significant benefits in further development of requirements and scope based Performance Measures. Database and tooling support would allow some degree of automation. Further research should be undertaken in the development of additional measures and those measures that did not receive particularly good rankings in the survey. It is felt the application of these measures may be appropriate in some technical developments. Overall there are opportunities to apply several of the individual sub-processes in isolation to better develop them, despite the benefits of combining them. This may be more viable for their development as the overhead
- 70. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 66 of 69 in terms of resources will be considerable and especially if they are applied in tandem with conventional and proven project monitoring and control techniques. Project management has failed to learn the lessons over successive high-profile project failures. Many of these failings were discussed within the literature survey [4] and so it may be surmised the discipline has been resistant to change for some considerable time. Without sufficient data, preferably corroborated, the benefits of Complexity Management via this framework will be intangible as best. This combined with project management’s inherent conservatism may be the greatest barrier to the acceptance to the tools and techniques described within this paper. An iterative and considered approach to the management of complexity will be required to gain its support. What is clear is that the current project management and systems engineering methodologies will need to improve their efficacy in the face of increasing levels of complexity. The question that has to be asked before commencing any complex project is what sets this undertaking apart from the high percentage of unsuccessful projects? The recognition of complexity and tailoring of the technical development process to suit a project’s unique complexity traits is one way this question may be answered. References [1] M Lee (2003). HRD in a Complex World, (Ed), London: Routledge. [2] C E Shannon and W Weaver (1998). The Mathematical Theory of Communication, University of Illinois Press, Champaign. [3] Australian Public Service Commission (2012). Tackling wicked problems: A public policy perspective, Last updated: 31 May 2012, Available at: http://www.apsc.gov.au/publications-and-media/archive/publications-archive/tackling-wicked-problems, Accessed on: 25th October, 2016. [4] N Brook (2015). Integration of technical development within complex project environments Literature Survey, MSc, Safety Critical Systems Engineering, University of York. [5] Mindtools (2016). CSFs; Identifying the Things That Really Matter for Success, Available at: https://www.mindtools.com/pages/article/newLDR_80.htm, Accessed on: 28th October, 2016. [6] L S Pheng and Q T Chuan [2006]. Environmental factors and work performance of project managers in the construction industry, International Journal of Project Management, Volume 24, Issue 1, January 2006, Pages 24–37, Available at: http://www.sciencedirect.com/science/article/pii/S0263786305000633, Accessed on: 25th October, 2016. [7] BMG Research (2014). Factors in project success; Research Report, Prepared for: The Association for Project Management (APM), November 2014, Available at: https://www.apm.org.uk/sites/default/files/APM%20Success%20report_NOV%2014.pdf, Accessed on: 25th October, 2016. [8] OGC (2012). PRINCE2; Directing Successful Projects, 5th Ed, TSO, Norwich. [9] aceproject (2014). Project management and firefighting, Available at: http://www.aceproject.com/blog/2009/04/29/project- management-and-firefighting/, Accessed on: 25th October, 2016. [10] M Ramsden (2013). Ten rules for smart bowtie analysis, 31 October 2013, ERM, Available at: http://www.erm.com/en/news- events/platform/ten-rules-for-smart-bowtie-analysis/, Access on: 8th November 2016. [11] N Brook (2015). Presentation: Failure curves and bowtie analysis (adapted from ERM (no date). Introduction to Bow-tie Diagrams, Available at: http://events.r20.constantcontact.com/register/event?llr=6quxcycab&oeidk=a07e5zzlwto9ff6b679), Northumbria Water [12] B Juttehttps (no date). 10 Golden Rules of Project Risk Management, Project Smart, Available at: http://www.projectsmart.co.uk/10- golden-rules-of-project-risk-management.php, Accessed on: 25th October, 2016. [13] K Jackson (no date). Plan-Do-Check-Act (PDCA), Mindtools, Available at: http://www.mindtools.com/pages/article/newPPM_89.htm, Accessed on: 25th October, 2016. [14] N Settle-Murphy (no date). 10 Top Tips for Leading Great Lessons Learned Reviews in a Virtual World, Guided Insights, Available at: http://www.guidedinsights.com/10-top-tips-for-leading-great-lessons-learned-reviews-in-a-virtual-world/, Accessed on: 25th October, 2016. [15] J Shane et al (2015). Guide to Project Management Strategies for Complex Projects, SHRP 2 Report S2-R10-RW-2, Institute for Transportation, Iowa State University, Available at: http://www.trb.org/Main/Blurbs/167482.aspx, Accessed on: 26th October, 2016. [16] Helmsman Institute (2012). Why Complexity Matters, Available at: http://www.apmg- international.com/nmsruntime/saveasdialog.aspx?lID=5479&sID=6815, Accessed on: 26th October, 2016. [17] N Goldenfeld and L P Kadanoff (1999). Simple Lessons from Complexity, Science 02 Apr 1999: Vol. 284, Issue 5411, pp. 87-89, Available at: http://science.sciencemag.org/content/284/5411/87, Accessed on: 25th October, 2016. [18] D W Oliver et al (1997). Engineering Complex Systems with Models and Objects, McGraw-Hill Inc.,US, Available at: https://oldsite.incose.org/ProductsPubs/DOC/EngComplexSys.pdf, Accessed on: 26th October, 2016. [19] D Rind (1999). Viewpoint: Complexity and climate. Science, 284, 105-107, Available at: http://pubs.giss.nasa.gov/docs/1999/1999_Rind_ri04100j.pdf, Accessed on: 25th October, 2016. [20] S A Kauffman (1993). The Origins of Order: Self-organization and Selection in Evolution, Oxford University Press. [21] J H Holland (1996). Hidden Order: How Adaptation Builds Complexity, Addison-Wesley, Reading, Massachusetts. [22] Y Bar-Yam (1997). Dynamics of Complex Systems, Addison-Wesley, Reading, Massachusetts, Available at: https://fernandonogueiracosta.files.wordpress.com/2015/08/yaneer-bar-yam-dynamics-of-complex-systems.pdf, Accessed on: 25th October, 2016. [23] G Weng et al. (1999). Complexity in biological signaling systems, Science. 1999 Apr 2;284(5411):92-6, Available at: https://www.ncbi.nlm.nih.gov/pubmed/10102825, Accessed on: 26th October, 2016.
- 71. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 67 of 69 [24] M Brugnach (2008). Complexity and Uncertainty: Rethinking The Modelling Activity, Published in Environmental Modelling, Software and Decision Support: State of the Art and New Perspectives, Amsterdam et al.: Elsevier, 2008.Available at: http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1071&context=usepapapers, Accessed on: 26th October, 2016. [25] M T Pich (2000). On Uncertainty, Ambiguity, and Complexity in Project Management, Published Online: August 1, 2002, Management Science, Available at: http://dx.doi.org/10.1287/mnsc.48.8.1008.163, Accessed on: 26th October, 2016. [26] The British Computer Society (2006). Case Study of Successful Complex IT Projects, Lancaster University, August 2006, Available at: http://www.bcs.org/upload/pdf/casestudy2.pdf, Accessed on: 26th October, 2016. [27] SEBoK (2016). Guide to the Systems Engineering Body of Knowledge, (SEBoK), Available at: http://sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_Body_of_Knowledge_(SEBoK), Accessed on: 26th October, 2016. [28] INCOSE (2015). Systems Engineering Handbook; A Guide for System Life Cycle Process and Activities, 4th Ed, INCOSE‐TP‐2003‐002‐04, January 2015, John Wiley & Sons, Hoboken, New Jersey. [29] Project Management Institute. (2004). A guide to the project management body of knowledge (PMBOK guide). Newtown Square, Pa: Project Management Institute. [30] D A Buchanan and A Huczynski (1997). Organisational Behaviour, 3rd Edition, Hemel Hampstead, Prentice Hall. [31] D D Gransberg and H D Jeong (2015). Managing Mega-Project Complexity in Five Dimensions, Conference: The 6th International Conference on Construction Engineering and Project Management (ICCEPM 2015), At Busan, Korea, October 2015, Available at: https://www.researchgate.net/publication/284533244_Managing_Mega-Project_Complexity_in_Five_Dimensions, Accessed on: 26th October, 2016. [32] Mindtools (2016). The Iron Triangle of Project Management, Balancing Your Budget, Scope, and Schedule, Available at: https://www.mindtools.com/pages/article/newPPM_54.htm, Accessed on: 26th October, 2016. [33] PCubed (2016). Nuclear Industry Sector Challenges, Available at: http://www.pcubed.com/bulletins/nuclear-industry-sector- challenges, Accessed on: 26th October, 2016. [34] ICCPM (no date). About ICCPM, Available at: https://iccpm.com/content/about-iccpm, Accessed on: 26th October, 2016. [35] A R McGowana et al (2013). A Socio-Technical Perspective on Interdisciplinary Interactions during the Development of Complex Engineered Systems, 2013 Conference on Systems Engineering Research, Volume 16, 2013, Pages 1142–1151, Available at: http://www.sciencedirect.com/science/article/pii/S187705091300121X, Accessed on: 26th October, 2016. [36] N Bennett and G J Lemoine. (2014). What VUCA Really Means for You, Harvard Business Review, January–February 2014 Issue, Available at: https://hbr.org/2014/01/what-vuca-really-means-for-you, Accessed on: 27th October, 2016. [37] G Satell (2013). Management Has to Change In An Increasingly Complex World, June 9th 2013, Available at: http://www.businessinsider.com/how-to-manage-complexity-2013-6?IR=T, Accessed on: 27th October, 2016. [38] L Fortnow (no date). Kolmogorov Complexity, Available at: http://people.cs.uchicago.edu/~fortnow/papers/kaikoura.pdf, Accessed on: 27th October, 2016. [39] P Holman (2010). Engaging Emergence: Turning Upheaval into Opportunity, Chapter 1. What Is Emergence? Berrett-Koehler Publishers, San Francisco. Available at: http://peggyholman.com/papers/engaging-emergence/engaging-emergence-table-of- contents/part-i-the-nature-of-emergence/chapter-1-what-is-emergence/, Accessed on: 27th October, 2016. [40] P Reason (1991). Human Error, Cambridge University Press, Cambridge. [41] J S Warboys and J S Keane (1993). OBM: A Specification Method for Modelling Organisational Process, Published in the Proceedings of the Workshop on Constraint Processing at CSAM’93, St. Petersburg, 1993, Available at: [42] ITIL (2016). Back to basics (People, Process and Technology), Available at: http://www.itilnews.com/index.php?pagename=ITIL__Back_to_basics_People_Process_and_Technology, Accessed on: 27th October, 2016. [43] M Frank et al (2011). The Relationship among Systems Engineers’ Capacity for Engineering Systems Thinking, Project Types, and Project Success, Project Management Journal, Volume 42, Issue 5, pages 31–41, September 2011, Available at: http://onlinelibrary.wiley.com/doi/10.1002/pmj.20252/abstract,, Accessed on: 26th October, 2016. [44] Techopedia (2016). Scope Creep; Definition - What does Scope Creep mean? Available at: https://www.techopedia.com/definition/24779/scope-creep, Accessed on: 26th October, 2016. [45] Association for Project Management. (2012). APM body of knowledge, 6th ed., High Wycombe. [46] A Bassiouny (2008). UAV Stability Augmentation System, Published on Nov 16, 2008, Available at: http://www.slideshare.net/ahmad1957/uav-stability-augmentation-system-usas-presentation, Accessed on: 27th October, 2016. [47] S Denning (2013). What Went Wrong at Boeing? Jan 21, 2013 Forbes, Available at: http://www.forbes.com/sites/stevedenning/2013/01/21/what-went-wrong-at-boeing/#30d2a3465aad, Accessed on: 27th October, 2016. [48] J Laaksonen (2010). Lessons Learned from Olkiluoto 3 Plant, 9th October, 2010, Power Engineering, Available at: http://www.power-eng.com/articles/npi/print/volume-3/issue-3/nucleus/lessons-learned-from-olkiluoto-3-plant.html, Accessed on: 27th October, 2016. [49] L Jun-yan (2012). Schedule Uncertainty Control: A Literature Review, 2012 International Conference on Medical Physics and Biomedical Engineering (ICMPBE2012), Available at: http://www.sciencedirect.com/science/article/pii/S1875389212016069, Accessed on: 27th October, 2016. [50] L P Leach (1999). Critical Chain Project Management Improves Project Performance‖, Project Management Journal, vol. 30, no. 2, pp. 39-51, June 1999, Available at: https://www.pmi.org/learning/library/critical-chain-pm-improves-performance-5305, Accessed on: 27th October, 2016. [51] R de Neufville and S Scholtes (2011). Flexibility in Engineering Design, The MIT Press, Cambridge, Massachusetts, London, England. [52] D Cleland and W R King (1997). Project Management Handbook, Chapter 20, Critical Success Factors in Effective Project implementation by J K Pinto and D P. Slevin, 2nd Edition, Wiley. [53] OGC (2012). PRINCE2; Directing Successful Projects, 5th Ed, TSO, Norwich. [54] G J Roedler and C Jones (2005). Technical Measurement, A Collaborative Project of PSM, INCOSE, and Industry, 27th December 2005, INCOSE-TP-2003-020-01, http://www.incose.org/docs/default-source/ProductsPublications/technical-measurement-guide---dec- 2005.pdf?sfvrsn=4, Accessed on: 28th October, 2016. [55] D Dvir et al. (1998). In search of project classification: a non-universal approach to
- 72. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 68 of 69 project success factors, Research Policy, Volume 27, Issue 9, December 1998, Pages 915–935, Available at: http://www.sciencedirect.com/science/article/pii/S0048733398000857, Accessed on: 12th November 2016. [56] T Chow and D-B Cai (2007). A survey study of critical success factors in agile software projects, Journal of Systems and Software, Volume 81, Issue 6, June 2008, Pages 961–971, Available at: http://www.sciencedirect.com/science/article/pii/S0164121207002208, Accessed on: 12th November 2016. [57] G L Ragatz et al. (1997). Success Factors for Integrating Suppliers into New Product Development, Product Innovation Management, Volume 14, Issue 3, May 1997, Pages 190–202, Available at: http://onlinelibrary.wiley.com/doi/10.1111/1540-5885.1430190/abstract, Accessed on: 12th November 2016. [58] Koutsikouri et al (2008). Critical success factors in collaborative multi-disciplinary design projects, Journal of Engineering, Design and Technology, Available at: http://www.emeraldinsight.com/doi/abs/10.1108/17260530810918243, Accessed on: 12th November 2016. [59] J Fortune and D White (2006). Framing of project critical success factors by a systems model, International Journal of Project Management, Volume 24, Issue 1, January 2006, Pages 53–65, Available at: http://www.sciencedirect.com/science/article/pii/S0263786305000876, Accessed on: 12th November 2016. [60] SurveyMonkey (2016). Home, Available at: https://www.surveymonkey.co.uk/, Accessed on: 27th October, 2016. [61] Dr. B Mehlenbacher (2002). Communication Disasters, ENG 421: Computer Documentation Design, Available at: http://www4.ncsu.edu/~brad_m/teaching/eng%20331/Lessons/communication.html, Accessed on: 28th October, 2016. [62] M V Malko (no date). The Chernobyl Reactor: Design Features and Reasons for Accident, Joint Institute of Power and Nuclear Research, National Academy of Sciences of Belarus, Available at: http://www.rri.kyoto-u.ac.jp/NSRG/reports/kr79/kr79pdf/Malko1.pdf, Accessed on: 28th October, 2016. [63] D Haughey (2014). A Brief History of Project Management, Project Smart, Available at: https://www.projectsmart.co.uk/brief-history- of-project-management.php, Accessed on: 28th October, 2016. [64] C Borysowich (2008). Pros & Cons of Gantt Charts, Toolbox.com, Feb 2, 2008, Available at: http://it.toolbox.com/blogs/enterprise- solutions/pros-cons-of-gantt-charts-22233, Accessed on: 6th November 2016. [65] T Browning (2012). The Design Structure Matrix: A Tool for Managing Complexity, Scientific American, 15th September 2012, Available at: http://blogs.scientificamerican.com/guest-blog/the-design-structure-matrix-a-tool-for-managing-complexity/, Accessed on: 28th October, 2016. [66] S D Eppinger et al (1992). Organising the Tasks in Complex Design Projects: Development of Tools to Represent Design Procedures, NSF Design and Manufacturing Systems Conferences, Atlanta, Georgia, January 1992, Available at: http://web.mit.edu/people/eppinger/pdf/Gebala_NSF1992.pdf, Accessed on: 28th October, 2016. [67] S D Eppinger and T R Browning (2014). Design Structure Matrix Methods and Applications, The MIT Press, Cambridge, Massachusetts, London, England. [68] DSMweb.org (2009). The Design Structure Matrix (DSM), Available at: http://www.dsmweb.org/, Accessed on: 28th October, 2016. [69] Oracle (2016). Primavera Enterprise Project Portfolio Management, Available at: https://www.oracle.com/uk/applications/primavera/index.html, Accessed on: 31st October 2016. [70] Planning Planet (2009). Resource Loading, Posted Thu, 2009-02-05, Available at: http://www.planningplanet.com/wiki/422401/resource-loading, Accessed on: 28th October, 2016. [71] GanttChartExample.com (2016). Gantt Chart Example, Available at: http://www.ganttchartexample.com/, Accessed on: 28th October, 2016. [72] S Vajna et al (2010). Designing the Solution Space for the Autogenetic Design Theory (ADT), International Design Conference – Design 2010, Dubrovnik, Croatia, May 17-20, 2010, Available at: https://www.designsociety.org/download- publication/29490/designing_the_solution_space_for_the_autogenetic_design_theory_adt, Accessed on: 29th October, 2016. [73] M Kreimeyer and U Lindemann (2011). Complexity Metrics in Engineering Design; Managing the Structure of Design Process, 1st Ed, Springer-verlag Berlin Heidelberg. [74] D Simon and F Simon (2005). Das Wundersame Verhalten von Entwicklern beim Einsatz von Quellcode-Metriken. Proccedings in DASMA Software Metrik Kongress: Metrikon 2005, Shaker Verlag, Aachen, 2005, pages 263-272, Available at: http://www.softwarekompetenz.de/servlet/is/28003/?print=true, Accessed on: 29th October, 2016. [75] M Bremer and B McKibben (2011). Escape the Improvement Trap: Five Ingredients Missing in Most Improvement Recipes. Boca Raton, London, New York. [76] A Kline et al (2003). Creating and Using a Performance Measure for the Engineering Design Process, American Society for Engineering Education, Available at: http://www.webpages.uidaho.edu/ele/scholars/Results/Publications/asee/ASEE_2003_Creating_Performance_Measure.doc, Accessed on: 29th October, 2016. [77] Balanced Scorecard Institute (2016). Balanced Scorecard Basics, Available at: http://balancedscorecard.org/Resources/About-the- Balanced-Scorecard, Accessed on: 1st November, 2016. [78] D Huether (2010). How Do You Know Your Metrics Are Worth It, Make Things Better, Available at: http://www.derekhuether.com/2010/02/11/how-do-you-know-your-metrics-are-worth-it/, Accessed on: 2nd December 2016. [79] R Navon (2005). Automated Project Performance Control (APPC) of construction resources, Automation in Construction 14(4):467-476 · August 2005, Available at: https://www.researchgate.net/publication/223091830_Automated_project_performance_control_of_construction_projects, Accessed on: 29th October, 2016. [80] A Yassine et al (2003). Information hiding in product development: the design churn effect, Research in Engineering Design 14 (2003) 145–161, DOI 10.1007/s00163-003-0036-2, Available at: http://necsi.edu/affiliates/braha/RED03_Info.pdf, Accessed on: 28th October, 2016. [81] N Brook (2016). Guidelines for Requirements Engineering and Management, Tractebel Engineering. [82] R Torbett et al (2001). Design Performance Measurement in the Construction Sector: A Pilot Study, Science and Technology Policy Research, Sussex University, Available at: http://www.sussex.ac.uk/spru/documents/sewp66, Accessed on: 29th October, 2016. [83] US Department of Energy (2010). DOE G 413.3-12, U.S. Department of Energy Project Definition Rating Index Guide, Available at: https://www.directives.doe.gov/directives-documents/400-series/0413.3-EGuide-12, Accessed on: 29th October, 2016. [84] NASA (2010). PDRI; Project Definition Rating Index Use on NASA Facilities, April 2000, Available at: http://www.hq.nasa.gov/office/codej/codejx/Assets/Docs/ProjectDefinitionRatingIndex.pdf, Accessed on: 29th October, 2016
- 73. Nick Brook MSc Safety-Critical Systems Engineering 9th January 2017 Page 69 of 69 [85] Construction Industry Institute (2015). Project Definition Rating Index (PDRI), Available at: https://www.construction- institute.org/scriptcontent/more/rr113_11_more.cfm, Accessed on: 29th October, 2016 [86] Department of Defense (2010). Technology Readiness Levels in the Department of Defense (DoD), Defense Acquisition Guidebook), Available at: https://www.army.mil/e2/c/downloads/404585.pdf, Accessed on: 31st October. [87] NASA (2007). Systems Engineering Handbook, NASA/SP-2007-6105 Rev1, December 2007, Available at : http://foiaelibrary.gsfc.nasa.gov/_assets/doclibBidder/tech_docs/5.%20NASA%20SP- 6105%20Rev%201%20(Sys%20Eng%20Handbook).pdf, Accessed on: 31st October, 2016. [88] NASA (2012). Technology Readiness Level, Oct. 28, 2012, Available at: http://www.nasa.gov/directorates/heo/scan/engineering/technology/txt_accordion1.html, Accessed on: 31st October, 2016 [89] M F Austin and D M York (2015). System Readiness Assessment (SRA) An Illustrative Example, 2015 Conference on Systems Engineering Research, Volume 44, 2015, Pages 486–496, http//www.sciencedirect.com/science/article/pii/S1877050915002677, Accessed on: 31st October, 2016. [90] M Westcott et al (2013). The DMI Design Value Scorecard: A New Design Measurement and Management Model, Design Management Review, Volume 24, Issue 4, pages 10–16, Winter 2013, Available at:.http://onlinelibrary.wiley.com/wol1/doi/10.1111/drev.10257/full, Accessed on: 7th November 2016 [91] N Brook (2016). Risk Management – Management Procedure, ref 3887; PALLAS Project, Tractebel Engineering. [92] L Mieritz (2012). Thisiswhatgoodlookslike; Gartner Survey Shows Why Projects Fail, Gartner, 1th June 2012, Available at: http://thisiswhatgoodlookslike.com/2012/06/10/gartner-survey-shows-why-projects-fail/, Accessed on: 29th October, 2016 [93] Royal Academy of Engineering (2004). The Challenges of Complex IT Projects, , Available at: http://bcs.org/upload/pdf/complexity.pdf, Accessed on: 29th October, 2016 [94] A Denker (2007). The Challenge of Large-Scale IT Projects, World Academy of Science, Engineering and Technology, International Journal of Social, Education, Economics and Management Engineering Vol:1, No:9, 2007, , Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.193.3858&rep=rep1&type=pdf, Accessed on: 29th October, 2016 [95] R Frese (2003). Project success and failure: What is success, what is failure, and how can you improve your odds for Success, 16th December 2003, , Available at: http://www.umsl.edu/~sauterv/analysis/6840_f03_papers/frese/, Accessed on: 29th October, 2016 [96] Modern Airliners (no date). Boeing 787 History, the creation of the Dreamliner, Available at: http://modernairliners.com/boeing-787-dreamliner/boeing-787-dreamliner-history, Accessed on: 29th October, 2016. [97] M Mecham (2011). 787: The Century’s First Jet to Fly. Sep 26, 2011, Aviation Week & Space Technology, Available at: http://aviationweek.com/awin/787-century-s-first-jet-fly, Accessed on: 29th October, 2016. [98] A Casey (2012). Boeing 7E7 case study, 21 November 2012, Available at: https://prezi.com/64hzwunhvlap/boeing-7e7-case-study/, Accessed on: 29th October, 2016. [99] P Ausick (2014). Why a Boeing 787-9 Dreamliner Costs $250 Million, June 17, 2014, 24/7 Wall St, Available at: http://247wallst.com/aerospace-defense/2014/06/17/why-a-boeing-787-9-dreamliner-costs-250-million/, Accessed on: 29th October, 2016. [100] Y Zhao, PhD (2013). Why 787 Slips Were Inevitable? Available at: http://zhao.rutgers.edu/787-paper-12-02-2013.pdf, Accessed on: 29th October, 2016. [101] J Koster et al (2012). Hyperion UAV: An International Collaboration, Conference: AIAA-ASM, Available at: https://www.researchgate.net/publication/263045116_Hyperion_UAV_An_International_Collaboration, Accessed on: 31st October 2016. [102] I Moir and A Seabridge (2008). Aircraft Systems Aircraft Systems: Mechanical, electrical, and avionics subsystems integration, 3rd Ed, John Wiley & Sons, Ltd. [103] J Koster et al (2011). Workforce Development for Global Aircraft Design , Full-text available · Conference Paper · Jan 2011, Available at: https://www.researchgate.net/figure/267593377_fig1_Figure-1-Boeing-787-Global-Work-Breakdown-Structure-1, Accessed on: 31st October 2016. [104] M Thurber (2009). An All-composites Learjet, Business Jet Traveler, Wednesday, April 1, 2009, Available at: http://www.bjtonline.com/business-jet-news/an-all-composites-learjet, Accessed on: 31st October 2016. [105] U Irfan (2014). How Lithium Ion Batteries Grounded the Dreamliner: Official report on Boeing 787 fires tells a cautionary tale about advanced batteries, ClimateWire on December 18, 2014, Scientific American, Available at: https://www.scientificamerican.com/article/how-lithium-ion-batteries-grounded-the-dreamliner/, Accessed on: 31st October 2016. [106] Telegraph Travel (2013). Boeing 787 Dreamliner: a timeline of problems, Telegraph Travel, 28 July 2013, Available at: http://www.telegraph.co.uk/travel/comment/Boeing-787-Dreamliner-a-timeline-of-problems/, Accessed on: 29th October, 2016. [107] C Negroni (2015). FAA Is Doing Nothing About Continued Boeing Dreamliner Battery Failures, 29 October 2015, Gizmodo, Available at: http://www.gizmodo.com.au/2015/10/faa-is-doing-nothing-about-continued-boeing-dreamliner-battery-failures/, Accessed on: 29th October, 2016. [108] P Marks (2013). Grounded: Where the Boeing Dreamliner went wrong, 6 February 2013, New Scientist Available at: https://www.newscientist.com/article/mg21729036-700-grounded-where-the-boeing-dreamliner-went-wrong/, Accessed on: 29th October, 2016. [109] W Kaufman (2013). Dreamliner Woes Expose FAA's Potential Weak Spots, January 23, 2013, npr Business Available at: http://www.npr.org/2013/01/23/170096977/dreamliner-woes-expose-faas-potential-weak-spots, Accessed on: 29th October, 2016. [110] D Rushe (2013). Why Boeing's 787 Dreamliner was a nightmare waiting to happen, 18 January 2013, The Guardian, Available at: https://www.theguardian.com/business/2013/jan/18/boeing-787-dreamliner-grounded, Accessed on: 8th November 2016. [111] AeroInside (no date). Airline Incidents for aircraft type Boeing 787-8 Dreamliner, Available at: https://www.aeroinside.com/incidents/type/b788/boeing-787-8-dreamliner, Accessed on: 31st October 2016.