#ISSS2015 Berlin - Gilbert et al - Understanding Systems Engineering Project Development

Dawn Gilbert – dawn.gilbert@bristol.ac.uk
Mike Mertens – Mike.Mertens@uk.thalesgroup.com
Mike Yearworth – mike.yearworth@bristol.ac.uk
Understanding Systems
Engineering Project
Development – A Traditional
and Complex Adaptive Systems
View

Question
How does complexity in the organization affect the ability of
systems engineers to meet delivery expectations in terms of cost
and time?

The Research
• Critical realist exploratory empirical case study
• Participant observation
• Data supported reflection in real-time and analysis at later stage
• Case study design informed by Yin (2014) and Palmberg (2009)
Novelty
• ‘An Investigator has the opportunity to observe and analyse a
phenomenon previously inaccessible to scientific investigation’
(Yin, 2014)
• Empirical CAS research of Systems Engineering Development (The
Evidence Centre, 2010)
• Logically coherent case study research in Systems Engineering
(Martin and Davidz, 2007), (Valerdi et al., 2010)

Organizational PM
• ‘a field dominated by
reductionist paradigms
and perspectives’ (Chia,
2013)
Chia, R. (2013) Paradigms and perspectives in organizational project management research: implications for knowledge-creation. In: Drouin, N., Müller, R. and Shankar, S.
(eds.) Novel Approaches to Organizational Project Management Research: Translational and Transformational. Series: Advances in organisation studies. Copenhagen Business
Press: Copenhagen, Denmark, pp. 33-55. ISBN 9788763002493

Complex Adaptive Systems
Holland, J. (2014) Complexity – A Very Short Introduction, Oxford University Press, Oxford,
UK
• Comprised of elements called
agents
• Agents learn or adapt in response
to interactions with other agents
• Agents exhibit bounded rationality
• Non-linearity
• Self-organizing
• Agents have 3 levels of activity:
– Performance (moment-to-moment
capabilities)
– Credit-Assignment (rating the usefulness
of available capabilities)
– Rule-Discovery (generating new
capabilities)

Source: Hobday, M. (2000) The
Project-Based Organisation: An
Ideal Form for Managing Complex
Products and Systems? Research
Policy, Vol 29, pp871-893

SE Function of a Business Line

Analysis
• Key Stakeholder, plus two steps away in org chart
• Three themes
– Software sprints
– Staff Circulation and Restructure
– Management Tools and Databases
• Does a ‘traditional’ view explain what is observed
– Are we surprised at what we find?
• Does a CAS view explain what is observed
– Are we surprised at what we find?

#ISSS2015 Berlin - Gilbert et al - Understanding Systems Engineering Project Development

Scrum Framework
"Scrum Framework" by Source (WP:NFCC#4). Licensed under Fair use via Wikipedia -
https://en.wikipedia.org/wiki/File:Scrum_Framework.png#/media/File:Scrum_Framework.png

"SampleBurndownChart" by Pablo Straub - Own work. Licensed under Public Domain via Wikimedia Commons -
https://commons.wikimedia.org/wiki/File:SampleBurndownChart.png#/media/File:SampleBurndownChart.png

Committed (x) vs Completed (y)
y = 0.6813x
R² = 0.9281
0
200
400
600
800
1000
1200
0 200 400 600 800 1000 1200 1400 1600
All Data Points
Completed
Linear (Completed)

Committed (x) vs Completed (y)
y = 0.5354x
R² = 0.1673
0
50
100
150
200
250
0 50 100 150 200 250
Data Points Under 200
Completed
Linear (Completed)

What’s happening?
• Semi-structured interview – Scrum Master, Product_1

• Staff_2 (software head of function):
• Programme Review Meetings review the health of the project, we want to underpin that with numbers….. 90% of
info of what’s going on on projects comes from walking the halls, the Programme Review presents the same info,
and lands surprises, metrics aim to balance the Programme Review to 50/50 coherent data. We are doing agile
across everything…We are using jira out of the box for tasking and defects…The burndown chart and velocities
we get from jira…. We have a 3 month sprint look ahead, where we can compare the backlog versus the team
size, but not all the work is in the backlog… I already look at software burndown daily.
• Staff_3 (programme management head of function):
• The main risk is in software development. We look at the Programme Review pack and we don’t get good info on
the state of the project…. The business objective here, I want quantitative and predictable software performance,
delivery against budget.
• Staff_7 (Product_2 programme manager):
• I want to know, what is our capability? What we struggle to do is estimate and stick to it. Across a number of
projects it is pretty much the same thing year after year in the software domain. … I look at Jira almost daily.
One cause of problems is other activities, like business winning that’s over and above, but knowing that, we said
we do the planned work.
• Staff_5 (Product_1 Scrum Master):
• If issues are not authorised (by the CCB – Configuration Control Board) they don’t get on the backlog…. .stories
as soon as you create them appear on the backlog Backlog isn’t a bucket of all the work left to do – it a ‘no
worse than’view... Reports are frozen with history, containing planned and actuals…The asterisk means added to
the sprint after the start, if it goes in, its urgent, it can come out if its not needed, something else changes, or
there’s no time… Velocity doesn’t work for us, it assumes we have a constant team, and the team doesn’t work on
the sprint work 100% of the time… It would be interesting to know how much of the Primavera and Business
Planning backlog is in the jira.

Staff_5: (Product_1 Software Scrum Master)
‘The programme review packs still don’t get looked at, they don’t look at my
supplementary pack [which reports sprint performance, lessons learned, engineering
metrics, context, achievements, and upcoming milestones] it’s all the SOFT, finance, QA
stuff, it makes it soul destroying…I strive to give full transparency and highlighted where
there have been issues elsewhere’.
Staff_1: (Acting Operations Director, and Head of Systems Engineering Function)
I think from a business perspective we need, what we needed was some relatively simple
metrics that could be collected easily
Researcher:
I think on Product_1 you’ve been seeing some software metrics the last couple of sessions
in there [referring to Programme Review Meetings]
It might do if I’d been in the last session with Product_1, I don’t think I was in the last
session with Product_1 so, no if I’m honest, no it doesn’t.

Analysis 2 – Staff Circulation and Restructure
• Business Line Performance Plan
– Staff rotation target 10%/yr.
– This underpins a business challenge to develop a high performance team
culture by optimizing and enhancing capability.
– Feedback would be gathered from managers and staff, quarter by quarter
to continuously improve the approach

Analysis 2 – Staff Circulation and Restructure
Staff_5 (Product_1 Software Scrum Master)
We have 1000-1100 hours of work per sprint. What it is doesn’t really matter, we know we
need a good range. When we change people, we change velocity. The team productivity is the
velocity, velocity changes from sprint to sprint. There is an administrative overload related to
juggling tasks so people who have fewer skills are occupied, so there are pick-ups and drop-
offs of work. If you look at capacity in advance, it reduces the capacity! We have varying
utilization a rule of thumb is that we have about 10 percent waste, someone will be off, and
chew up several days. A 2-day task for handholding is non-specific, you have to add in the
time, it looks like the sprint got bigger but it was expected to increase anyway. I looked at
work pick ups and drop offs versus focussed depth and asked why? We got support interrupts
(ie tech support from users in the field), we hit a blocker like we were waiting for SE/PDA who
needed something, or we are juggling skills of task to the skills of the people…. We’ve got 2
new staff, they are struggling to get their machines up to date….. People moved off the
project…’
Staff_6: (Product_2 Software Scrum Master)
‘We are rotating staff left right and centre so we have programme mobility we need a
consistent approach … A restructure is afoot…. The last 2 sprints had to pay significantly for
new people. You don’t get it and you won’t get it for several months, this is the rotation plan,
One or two people are so heavily relied upon, like the tech lead, its like fighting a losing
battle….’

https://forio.com/simulate/k.e.v.oorschot/get-fat-fast-hiring-chains-in-projects/simulation/#p=page3
‘ah, it might be that we’ve had a worse impact that we thought, that’s enlightening, we
want to cope with the perturbation…maybe we are a more competent engineering
organisation than we thought’

After conceptualizing discrete event modelling
• Staff_1
‘I like that, I like that…you know what, I’d be quite keen to try and
do something like this on something like Product_4’
Two weeks later I heard Staff_1 had submitted his resignation

Analysis 3 – Management Databases

• Staff_1: (Acting Operations Director, and Head of Systems Engineering Function)
• We have an operations meeting, the outputs of project review meetings don’t go in to it, it’s a supply and
capability review for Business Planning…..the demand on the business human resources now and out to 2 years
looking at potentials and set demand…. We have a heatmap for 12 months, Business Planning hasn’t changed the
tactical planning over the next 3 months, we’re not very good at 3-12 months, 12-24 month Business Planning is
better, it used to be a 12 month rolling wave, now we have reliable data for 15 months, 18-24 months is not so
reliable. I think we make decisions on up to 15 months, this is what we can reliably believe, we keep watch on 5-
18 months. Staff_18 [Thales UK Corporate Business Planning] was hard on prescriptive direction, now it’s ‘you
tune the process for what works for you’, we’ve gone full circle back with everyone being in the room at the same
time…..You’ve got to have human beings in the loop.
• Staff_2: (software head of function
• There is general support for jira, we’ve transitioned to tool up for agile. We want to integrate to primavera for
work package management and earned value management. We have a 3 month sprint look ahead, where we can
compare the backlog versus the team size, but not all the work is in the backlog… Business Planning deals with
resourcing… I already look at software burndown daily.
• Staff_3: (programme management head of function):
• We’ve been trying to sort out how we do agile, using Primavera for schedules and reviewing in project
reviews...If we have our bow-wave of known work and forecast wave beyond that, we can match it to
resources…Primavera has always been a bit of a dark art. Last year we had a workshop on linking primavera to
jira...Primareva tells you how many people you need in the sprint to do it. The actuals should be in Jira and
Primavera. The Primavera month is a financial month, we export from Jira back to Primavera as target EVM.
The Functional Manager View

• Staff_5: (Product_1 Software Scrum Master)
• It would be interesting to know how much of the % primavera Business Planning backlog is in the Jira backlog.
[When looking in Jira] make sure what you are looking at is what you are thinking you are looking at.
• Staff_6: (Product_2 Software Scrum Master)
• Jira is the backlog manager…. We don’t have backlog reviews yet, which will be weekly with programme
managers and the integrated product team (we have Jira drive this). We have only got so much capacity, what
you do is what you do…. Software estimating is crap… we are underperforming, we are incapable, and you can’t
ascertain that from a sprint burndown. Jira will help us in time track how much time we spend, but we haven’t
met one sprint yet, we are on sprint 5 at the moment….. in Jira we know day by day we are ‘not performing’, but
even when we know we don’t use open language in stand ups
• Staff_7: (Product_2 programme manager):
• I look at Jira almost daily…. Demand is now run from Primavera, which has problems, its driven by Business
Planning. We use the same tool for different purposes.
• Staff_9:Systems Engineering Manager – Product_3
• Jira is used to monitor, which is the best tool seen so far…. I monitor the burndown in jira, it gives KPIs which
are tweaked and added to…
• Staff_10: ‘Front-End’Systems Engineer – Product_3
• The project has three pulls, the PM reports to the PM line…the SEM reports up through to OPS about cost,
money, time, the PDA reports to the design criteria – there is a question of whether this reflects the customer…
we don’t have a sound approach to estimating. I’ve made an estimate, what happens to that info? When it comes
to lessons learned, we cut and paste the last lot. In actuals, is overtime visible or not? – it’s not visible on actuals.
We never see positive methods of capturing, we are mixing apples and pears, we are mixing the contents….
The Project Staff View

• Staff_14: Project Controller – multiple projects
• The risk management pack is in the project management pack and is referred to in the Organisational
Management System. There is a monthly update to the risk register during each project, there is an independent
assessor at the meeting along with the PM and senior technical team people – the PDA… You enter the risk and
its weight in line with Organisational Management System criteria….Risk happens, it is built in to the project
schedule, then budget and resource allocated. We should then maintain a log of what happened. In the perfect
world we wouldn’t be surprised by late work and cost overruns, they will be in the PMR, the top 5 risks or maybe
10. I wouldn’t be surprised, but Staff_16 (Business Line General Manager) might be. We are not running risk
through Primavera. Horse-trading of scope, time, and budget is down to commercial, Staff_15 (Business Line
Technical Director) is an independent assessor. The [Organisational Management System] instruction is clear, it
works, project managers should be aware of the instruction, I can only assume project managers get it…. I don’t
know who makes the decision on what the risks are, I’d say the risk is our estimates are bad. The pressure is
coming from above, not from projects….Over a 3 year project we forecast finishing to one day, but we don’t know
if that’s a 95% chance of getting there, and we don’t know how long it will be until we get to 100% done – is it 2
months past the end date? If we show a SPI or CPI of 50% we get questions from above, so here we massage it so
we show no variance and don’t get questions…. The Work Breakdown Structures are defined prior to Primavera,
the project managers say to software, “I don’t care how you do it, just do it”. The cost collection is actually at
different levels, for example, from subcontractors and from Jira. The lowest level we go should be the level you
get actuals against. Primavera breakdown was mapped against delivery structure routed through work packages
by customer, but we estimate and collect cost through product and function. I’m restructuring Primavera to
match against what we measure…. I’m resorting and simplifying Primavera to reduce dependencies that are
likely to cause errors in updating that are also unnecessary levels of detail for one-person staffing
The Project Staff View

Discussion
• Analysis 1 – Software Sprints
– A traditional view would not expect to see this sprint performance
(expected vs completed), and could only explain it as poor performance.
Continuous improvement-type approaches within that paradigm could
explore why this is, but they weren’t undertaken
• (‘its pretty much the same thing year after year’)
– A CAS perspective shows bounded rationality, and agents (staff) adapting in
response to others
– The traditional view can’t explain why the scrum metrics data isn’t
reviewed by Staff_1 in the Programme Review Meetings that his presence is
‘mandatory’ at (according to the organisational management system). A
CAS view understands that a staff member in 2 roles will personally
prioritise, and that staff in 2-day-long review meetings may also choose not
to review everything’

Discussion
• Analysis 2 – Staff Circulation
– The traditional view produced the Business Line Performance Plan which
called for staff rotation, outlined it’s aims and described it’s monitoring and
improvement approach.
– Staff Circulation was implemented (25% staff rotation during the 6 months
field work, a further 25% in the subsequent 6 months was observed)
– Prior to this research, monitoring had not taken place. The traditional view
would not expect this, and can’t explain this.
– A CAS perspective explains staff circulation and restructuring inherently.
– A CAS perspective also explains a lack of monitoring, as Staff_1 was acting
in two roles, overburdened with work, and made personal decisions about
how to prioritize work (agent performance).
– In presenting the system dynamics model to Staff_1 the researcher
supported Staff_1 in rule discovery, generating new capabilities, in seeing
the impacts of staff rotations differently (i.e. supported by evidence from
the business line). Since his resignation followed shortly after, we can’t say
whether this new capability would have been assigned sufficient credit to
impact the performance (chosen actions) of Staff_1.

Discussion
• Analysis 3 – Management Databases Contd.
– The Software and Project Management functional managers show strong
reductionist preferences, believing the data in the databases are explicit
and objective, and can be, or are linked in a sensible, valuable way.
• This can drive a lack of questioning of the data.
– The staff see value, and use the databases with caution, or interpretation.
In the risk example, the databases are explicitly manipulated to avoid
questions from ‘above’. The prevalent reductionist approach of senior
management understands deviation from plan (or expected) only in terms
that means something is at fault.

Reflection
• The ‘traditional’ approaches don’t explain or expect what is
evident
• Staff see the effects of complexity
– They don’t necessarily have frameworks to understand them
– May instinctively navigate them well
• Do they have targets on their backs?
• An unwillingness to be open to ‘other ways’ of viewing the
organization can be explained, but not overcome by the CAS
view
– Performance (moment-by-moment capabilities)
– Credit-assignment (rating the usefulness of capabilities)
– Rule discovery (generating new capabilities)

Reflection
• Thales UK View
– The data is probably familiar to other Tier 1 Systems Engineering
Contractors
• The case study is expected to resonate with others, and provide an
illustrative example of the value of novel research approaches in systems
engineering management
– The methodology was richly revealing about the state and nature of the
Business Line.
– The BL was so dynamic, more standard methods are immediately obsolete.

#ISSS2015 Berlin - Gilbert et al - Understanding Systems Engineering Project Development

More Related Content

What's hot (13)

Viewers also liked (17)

Similar to #ISSS2015 Berlin - Gilbert et al - Understanding Systems Engineering Project Development (20)

More from International Society for the Systems Sciences (ISSS) (6)

Recently uploaded (20)

#ISSS2015 Berlin - Gilbert et al - Understanding Systems Engineering Project Development

Editor's Notes