Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key
1 like1,038 views
This document discusses challenges with effectively leveraging lessons learned from past projects. It notes that while trillions are spent annually on projects globally, lessons learned systems often do not work well to avoid repeating mistakes. The document analyzes lessons learned data from over 50 organizations and 1,000 projects, finding issues like a lack of context, root cause analysis, and measurable actions. It argues for a more data-centric approach to link lessons to risks and enable different perspectives to utilize the information. The document provides an example of how Crossrail leveraged experience across similar projects.
![Copyright ProjectingSuccess2018
The failure of the FiReControl project
The Stark Reality of Not Learning Lessons
Common Agricultural Policy Futures
programme
We do not expectthe programme to delivervaluefor money. The cost of
£178 millionis 74percent morethan theoriginalbusiness case,and the
programmewill not fully deliver theanticipatedbenefits. Farmers reported
they have been adversely affectedby the programmedelays, highlighting cash
flow problems that haveknock-on effects for thewiderruraleconomy.
http://www.audit-scotland.gov.uk/uploads/docs/report/2016/nr_160519_cap_futures.pdf
‘911 Upgrade’
(Emergency Communications
Transformation Program)
The project started in2004with an estimated costof$1.3billion and a three
year timeframe.
According to an audit fromCity Comptroller,released 21March2012, the
project is now $1billion over budget andsevenyears behind schedule.
The Comptroller blamed the cost overruns on inadequate project
management within thecity.
http://sa.acs.org.au/gallery/l ead ers _lunch/20120516_Common_causes _of_failur e_in _m ajor_ICT- enabl ed_proj ects.pdf
NHS 24
The total cost oftheFuture Programmehadrisen by 55 per centto £117.4
million,compared toa business casecostof£75.8million covering theten-
year contractperiod. Theincreasewas dueto changes inthecontract
specification and costs associatedwith the delay in implementing theIT
system. [2016]
http://www.audit-scotland.gov.uk/uploads/docs/report/2016/s22_161006_nhs_24.pdf
Disclosure Scotland
The whole programme was estimated to cost £31millionand included a
£29 million projectto designandbuild a new ICTsystem tosupport thenew
arrangements and replacetheexisting system, along with otherinformation
management projects, training and guidance.
The programme experienced a number ofsignificantproblems whenit went
live in February 2011. DisclosureScotland and the supplier continuetowork
on these problems and corefunctionality is now expectedto beestablishedby
autumn 2012.
Crown Office and Procurator Fiscal
Service (COPFS)
The Phoenixprogrammewas initiated inearly 2009, at an estimated cost
of£10 million, becausea number ofsoftware elements ofthe existing case
management system used tohelp prepare criminal cases for prosecution were
outdated and lackedreadily availabletechnicalsupport.
Managementofthe programmeexhibiteda number ofweaknesses butit was
cancelled in November 2010before construction started dueto increased
costs and a significantlyreduced capital budget.
National Polar-orbiting Operational
Environmental Satellite System
Department ofVeterans Affairs’ Scheduling Replacement Project was
terminated inSeptember2009afterspending an estimated $127millionover
9 years.
http://www.gao.gov/assets/690/681420.pdf
Secure Border Initiative Network program
The Departmentof Homeland Security’s Secure Border InitiativeNetwork program
was ended inJanuary 2011, after the departmentobligatedmorethan$1billion to
the program, becauseitdidnot meet cost-effectiveness andviability standards.
http://www.gao.gov/assets/690/681420.pdf
Financial and Logistics Integrated Technology
Enterprise program
The DepartmentofVeterans Affairs’ Financial and Logistics IntegratedTechnology Enterprise program
was intended to bedelivered by2014ata totalestimatedcostof$609million,butwas terminated in
October 2011duetochallenges in managing theprogram
http://www.gao.gov/assets/690/681420.pdf
eCare
eCare programme was supposedto solve allourdata communication
problems in thehealth andsocialcareservice, but itwas abandoned
completely after sevenor eightyears at—wearetold—a cost of£56million,
although I suspectthatthecostwas substantially more.
http://www.parliament.scot/S4_PublicAuditCommittee/20_January_2016.pdf
Registers of Scotland (RoS)
Responsiblefor compiling and maintaining registers relating to
property and other legal documents.Its annualoperating costs were
£78 million in 2010/11. While bothRoS and BT consider the ongoing
ICT serviceprovision has largelybeen successful,RoSnow considers
that the partnership outsourcing ofallICT was inappropriate and it
has terminated theSPA.
Two individualprojects within theprogrammenowcancelled,with
costs of£6.7millionwritten offin March2011.
Expeditionary Combat Support System
The DepartmentofDefense’s Expeditionary CombatSupportSystem was
canceled in December2012afterspending more than a billion dollars and
failing to deploy within 5 years ofinitiallyobligating funds.
http://www.gao.gov/assets/690/681420.pdf
Retirement Systems Modernization program
The Office ofPersonnel Management’s Retirement Systems Modernization program was canceled in
February 2011, afterspending approximately $231millionon theagency’s third attemptto automate
the processing of federal employeeretirement claims.
http://www.gao.gov/assets/690/681420.pdf
>£20,000,000,000](https://image.slidesharecdn.com/presentationtoapmnov18v2compressed-181205115737/85/Lessons-Learned-Don-t-Work-Why-a-Data-Centric-Approach-May-Hold-the-Key-4-320.jpg)
Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key
- 1. Copyright ProjectingSuccess2018 Martin Paver CEO / Founder www.projectingsuccess.co.uk Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key APM South Wales and West of England Branch Event 22 Nov 2018
- 2. Copyright ProjectingSuccess2018 Why ‘Lessons Learned’? Project Lead for $Bn Nuclear New Build Programme Manager for £400m portfolio of 60 projects PMO roles in multi $Bn portfolios I’ve alwaysstruggled to leverage the broad experience that I know exists. "Those that fail to learn from history are doomed to repeat it."
- 3. Copyright ProjectingSuccess2018 Leverage Experience – A Wicked Problem? Small Margins Source:APM Project Success Factors https://commons.wikimedia.org/wiki/File:Tour_de_France_2016,_team_sky_(28595456475).jpghttps://upload.wikimedia.org/wikipedia/commons/7/76/Mclaren_racing_usgp_2004.jpg Avoiding the avoidable Certainty in delivery Reduced outliers Strategic focus Reputation Reduce incidents Increased profit Cost avoidance Sector 1 InternationalA Sector 2 ScheduleRiskLessons Cost Project Delivery Experience Company A
- 4. Copyright ProjectingSuccess2018 The failure of the FiReControl project The Stark Reality of Not Learning Lessons Common Agricultural Policy Futures programme We do not expectthe programme to delivervaluefor money. The cost of £178 millionis 74percent morethan theoriginalbusiness case,and the programmewill not fully deliver theanticipatedbenefits. Farmers reported they have been adversely affectedby the programmedelays, highlighting cash flow problems that haveknock-on effects for thewiderruraleconomy. http://www.audit-scotland.gov.uk/uploads/docs/report/2016/nr_160519_cap_futures.pdf ‘911 Upgrade’ (Emergency Communications Transformation Program) The project started in2004with an estimated costof$1.3billion and a three year timeframe. According to an audit fromCity Comptroller,released 21March2012, the project is now $1billion over budget andsevenyears behind schedule. The Comptroller blamed the cost overruns on inadequate project management within thecity. http://sa.acs.org.au/gallery/l ead ers _lunch/20120516_Common_causes _of_failur e_in _m ajor_ICT- enabl ed_proj ects.pdf NHS 24 The total cost oftheFuture Programmehadrisen by 55 per centto £117.4 million,compared toa business casecostof£75.8million covering theten- year contractperiod. Theincreasewas dueto changes inthecontract specification and costs associatedwith the delay in implementing theIT system. [2016] http://www.audit-scotland.gov.uk/uploads/docs/report/2016/s22_161006_nhs_24.pdf Disclosure Scotland The whole programme was estimated to cost £31millionand included a £29 million projectto designandbuild a new ICTsystem tosupport thenew arrangements and replacetheexisting system, along with otherinformation management projects, training and guidance. The programme experienced a number ofsignificantproblems whenit went live in February 2011. DisclosureScotland and the supplier continuetowork on these problems and corefunctionality is now expectedto beestablishedby autumn 2012. Crown Office and Procurator Fiscal Service (COPFS) The Phoenixprogrammewas initiated inearly 2009, at an estimated cost of£10 million, becausea number ofsoftware elements ofthe existing case management system used tohelp prepare criminal cases for prosecution were outdated and lackedreadily availabletechnicalsupport. Managementofthe programmeexhibiteda number ofweaknesses butit was cancelled in November 2010before construction started dueto increased costs and a significantlyreduced capital budget. National Polar-orbiting Operational Environmental Satellite System Department ofVeterans Affairs’ Scheduling Replacement Project was terminated inSeptember2009afterspending an estimated $127millionover 9 years. http://www.gao.gov/assets/690/681420.pdf Secure Border Initiative Network program The Departmentof Homeland Security’s Secure Border InitiativeNetwork program was ended inJanuary 2011, after the departmentobligatedmorethan$1billion to the program, becauseitdidnot meet cost-effectiveness andviability standards. http://www.gao.gov/assets/690/681420.pdf Financial and Logistics Integrated Technology Enterprise program The DepartmentofVeterans Affairs’ Financial and Logistics IntegratedTechnology Enterprise program was intended to bedelivered by2014ata totalestimatedcostof$609million,butwas terminated in October 2011duetochallenges in managing theprogram http://www.gao.gov/assets/690/681420.pdf eCare eCare programme was supposedto solve allourdata communication problems in thehealth andsocialcareservice, but itwas abandoned completely after sevenor eightyears at—wearetold—a cost of£56million, although I suspectthatthecostwas substantially more. http://www.parliament.scot/S4_PublicAuditCommittee/20_January_2016.pdf Registers of Scotland (RoS) Responsiblefor compiling and maintaining registers relating to property and other legal documents.Its annualoperating costs were £78 million in 2010/11. While bothRoS and BT consider the ongoing ICT serviceprovision has largelybeen successful,RoSnow considers that the partnership outsourcing ofallICT was inappropriate and it has terminated theSPA. Two individualprojects within theprogrammenowcancelled,with costs of£6.7millionwritten offin March2011. Expeditionary Combat Support System The DepartmentofDefense’s Expeditionary CombatSupportSystem was canceled in December2012afterspending more than a billion dollars and failing to deploy within 5 years ofinitiallyobligating funds. http://www.gao.gov/assets/690/681420.pdf Retirement Systems Modernization program The Office ofPersonnel Management’s Retirement Systems Modernization program was canceled in February 2011, afterspending approximately $231millionon theagency’s third attemptto automate the processing of federal employeeretirement claims. http://www.gao.gov/assets/690/681420.pdf >£20,000,000,000
- 6. Copyright ProjectingSuccess2018 But does it really matter? Within the next 10 years…. How much could be saved? The analysis of National Infrastructure and Construction Pipeline contains projections of £600 billion of public and private investment in infrastructure over the next 10 years. McKinsey analysis estimates that global capital spending will total $77 trillion between 2018 and 2023, which places the annual value of that spending at more than $10 trillion. …. >$100 trillion over 10 years
- 7. Copyright ProjectingSuccess2018 Pursuing the Case for Change: PMI
- 8. Copyright ProjectingSuccess2018 Programme Manager for £400m portfolio of 60 projects Sept 2018 Blog Pursuing the Case for Change: APM
- 9. Copyright ProjectingSuccess2018 Papers and ArticlesPapers and Articles
- 10. Copyright ProjectingSuccess2018 The Journey Lessons learned systems do not work Instinctively,we should be able to leverage experience Why don’t the methods work? What can we do to fix it? What have others done? Is it fixable? • NAO Reports • GAO Reports • Public hearings • Public Accounts Committees • Lessons learned data (Aus, Sco, NI) • Audit Reports (Can, Aus, NZ, Holland) • FOI requests • Commercially sensitive data
- 11. Copyright ProjectingSuccess2018 Capacity to Learn Lessons Home Office Government Major Projects Portfolio 2016 Ministry of Justice Government MajorProjects Portfolio 2016 With so many projects not delivering to plan there should be a rich opportunity for leveraging experience for the collectivebenefit of UK PLC…. …..but there is limitedevidence of this happening. Dept of Health Government Major Projects Portfolio 2016 Ministry of Defence Government MajorProjects Portfolio 2016
- 12. Copyright ProjectingSuccess2018 Digging a Little Deeper Year RAG Forecast End Forecast Whole Life Costs £m 2012 31/12/2014 138 2013 01/04/2016 1375 2014 01/09/2016 1205 2015 30/11/2017 1706 2016 30/11/2017 2879 Source: GMPP Reports
- 14. Copyright ProjectingSuccess2018 Reasons for Not Recording Lessons? Capitalizing from past projects the value of lessons learned. PMI CONFERENCE PAPER 2008 By Trevino, Stephanie A. | Anantatmula, Vittal S. Its a matter of priorities for: • Senior Management • Project Leadership Priority not given because: • Benefits are not measurable • Lessons recorded aren’t used • Process driven approach Therefore, effort is prioritised on delivery. With exceptions for: • Projects in heavilyregulated industries • Public sector projects with learning legacy as a key factor • Projects in consultancieswhere learning provides clear competitive advantage.
- 15. Copyright ProjectingSuccess2018 Lessons Learned Processes DH does not have any policies or processes on lessons learned. Lesson learned methodologyis standardProgramme and Project Managementpractice which programmes and projects deploy as they see fit.
- 16. Copyright ProjectingSuccess2018 But aren’t the NAO fixing the problem? https://www.nao.org.uk/naoblog/a-systematic-look-at-major-programmes/ It is the role of Departments to leverage the experience, not the NAO
- 17. Copyright ProjectingSuccess2018 Nearly 20,000 Lessons….. Overview >50 Organisations >1,000 Projects >2 Years of effort to collate Covering 3Continents Estimated Portfolio ~$30 bn From …. ”Benefits clearer” to …”There was insufficient clarity about accountability for each budget line, reflecting a centralised approach to budget management. Poor links between the Finance Team and key programmes compounded this lack of accountability, which meant that forecasting, monitoring and budget discussions, as well as early planning of procurement strategies, did not take place in a sufficiently timely manner to identify and address the key risks”.
- 18. Copyright ProjectingSuccess2018 Existing Lessons Learned Analysis http://www.treasury.govt.nz
- 19. Copyright ProjectingSuccess2018 Is This A Knowledge Management Problem? Project, Programme, Portfolio Management TechnicalDelivery I don’t know I do know Knowable Unknowable
- 20. Copyright ProjectingSuccess2018 Every Lesson has An Action… Or does it? Lessons learned….. You experienced XYZ and if you do ABC in the future then the lesson won’t be repeated. Observation 1: We should be getting better at this… but the list of lessons has been similarfor the last 30 years. Observation 2: An effective project manager needs to know every lesson, in every context about every sort of project. Observation 3: There are organisational factorsthat inhibitlearning
- 21. Copyright ProjectingSuccess2018 Findings From the Lessons • Lack of forensicinsight • ‘The number of technical reviews increased during this time and were discussed at the project board but there is limited evidence that the risks highlighted in the reviews were acted upon or resolved’. • ‘Lack of robust business requirements at the outset, leading to unrealistic IT project budgets and timescales’. • Lots of lessons on reporting, but few define the problem or what to do about it. • Very variable. • Processrather than outcome focus. • Repetition. Some lessons repeated up to x10. • Often lack context. • Lack consequence. Difficult to understandwhat was a big problem. • Lacks root cause analysis. • Difficult to identify correlationsbetween lessons. The best lessons address the above.
- 22. Copyright ProjectingSuccess2018 Nature & Quality of the Lessons The SRO and ProjectTeam should develop a robust and comprehensive Stakeholder Management and CommunicationPlan. The projectshould considera workshop to capture all LessonsLearned to date and ensure that the learning is embeddedwithin all future development Review and implementproject governance arrangements. Projectreporting formatshould be reviewed. Observation? Something to consider in the next phase? Missing the basics and don’t know better Should know better Fundamentalshortfallwith minor impact Fundamentalshortfallwith major impact
- 23. Copyright ProjectingSuccess2018 ‘Avoidability’ of lessons •Act of God, environmental factors •Unknown/unknowableexternal event •Unknown/knowableexternal event •Known external event •Internal decision or organisationalconstraint •Lesson identifiedor known but deemed irrelevant •Lesson identifiedor known but ignored due to other priorities/resourcing Unavoidable lesson But may be aware of emerging environmental conditions Avoidable lesson Priority, resourcing, SQEP or judgmentcall Strategic lesson Organisation proceeded knowing thatstrategic decisions may/willlikely impact the project. Lesson learned wasforeseeable and wasn’tmitigated.
- 24. Copyright ProjectingSuccess2018 The Link Between Risk and Lessons Risk IssueLesson If the lesson is not acted upon it becomes a risk for the next project How many organisations link lessons learned and risk management? What is the correlationbetween risk drawdown and lessons learned? Tolerate Treat Transfer Terminate
- 25. Copyright ProjectingSuccess2018 Projects:An Enabler or Primary Purpose? ProjectDuration Similarity Short Long SimilarDissimilar Major IT upgrades Business transformation Enabling Activity Office Move New office Call Centre ProjectDuration Similarity Short Long SimilarDissimilar Event management Major IT upgrades Business transformation Office Move Crossrail New office Primary Activity HS2 Call Centre Station refit Station build 1, 2, 3
- 26. Copyright ProjectingSuccess2018 Leveraging Experience: Events Scotland Security Transport Ticketing Funding Marketing Infrastructure ……. Opportunityto recycle experience: • Capturing schedules, assumptions, risks and budgets. • Identifying areas of stability and uncertainty and which are likely to change. Management effortis then focused on areas of instability. • Understanding the effectiveness of management action associated with risks, assumptions and delivery. • Embedding members of a team into an existing project so that the learning can be recycled into the next project. “A 6-7year journey” Immovable deadlines • Context is critical in understanding application of experience • Cost over-runs tend to be associated with infrastructure • Organisational Capacityto leverage experience • International sharingof experience
- 27. Copyright ProjectingSuccess2018 One Dataset. Different Perspectives Assurance reviewer Governance Supply chain Project team Next project Otherorganisations Process of personal promotion,personal reflection & learning for the next phase Holding to account Preparationfor key gatewayreviews Enablingthe supply chain and improving efficiency Leveraging experience for aligned projects Leveraging experience for other organisationsor other parts of ‘the group’
- 28. Copyright ProjectingSuccess2018 A Tailored Approach Any approach to improvingproject delivery productivitymust be tailored to the organisation: Stable workforce & suppliers Experience is ‘recycled’ Transient workforce. Churn in supply base Experience is fragmented Multiple and parallel implementations of similar projects Bespoke, unique projects delivered infrequently Experience is primary guide Process is primary guide Highly competitive where productivity is a major differentiator Public sector or monopoly, incremental improvement Cross fertilisation between departments Experience is typically siloed within the organisation Internal governance External regulation/governance Workflows, Wikis, Processes, Assurance Key considerations Knowledge sharing, project controls, lean Spectrum of approaches
- 29. Copyright ProjectingSuccess2018 An Example: Crossrail
- 30. Copyright ProjectingSuccess2018 Exhaust plume fromproject delivery An Example: Crossrail
- 31. Copyright ProjectingSuccess2018 What Happens to the Data?
- 33. Copyright ProjectingSuccess2018 Instinctive Behaviours System 1 System 1 is an automatic, fast and often unconscious way of thinking. •It is autonomous and efficient, requiring little energy or attention, •Is susceptical to biases and systematic errors. System 2 is an effortful, slow and controlled way of thinking. It requires energy and can’t work without attention but, once engaged, has the ability to filter the instincts of System 1 System 2 Nudge Theory We adjust our driving to the conditions of the road We adjust our speed if we know that others have experienced costly accidents VS System 1 and 2 thinking
- 34. Copyright ProjectingSuccess2018 Leveraging the Connected Data Projects LessonsRisks$ Graph Data Stored in Silos LessonX Draw down Cost impact Time impact Mitigate Cost Mitigate effective -ness Project 1 Project 2 Taxon- omy TechnicalSafetySecurity Technical issue Security Issue Safety Issue
- 35. Copyright ProjectingSuccess2018 Exploitation of Data Actors Views User Stories Degree of fit ‘Perfect Data’ ‘Typical Data’ For most organisations,the ‘lessons learned’ process is only solving 25% of the requirement, at best. “As a Project Manager I want to be able to drill into lessons that I am interested in so that I can gain a better understanding of context, relevance and how the lesson may impact my own project”
- 36. Copyright ProjectingSuccess2018 EVM data Resourcing Weather Supplier performance Dependencies Risks etc Real time update of assigned tasks WBS Elements Scheduling Corpus and Context Extract Triples Benchmarking Adaptive SchedulingRecommendations Scheduling
- 37. Copyright ProjectingSuccess2018 A once through process Risk lifecycle Leveraging Risk Experience Connected risks Risks-Issues-Lessons Informed risk registers Risk trends Risk mitigations Risk budget Systemic Risk Risks
- 39. Copyright ProjectingSuccess2018 So... What is Advanced Data Analytics
- 40. Copyright ProjectingSuccess2018 Icons made byFreepik fromwww.flaticon.com Machine Learning: Bid Data – a worked example
- 41. Copyright ProjectingSuccess2018 Bent Flyvbjerg’s work on Major Projects
- 42. Copyright ProjectingSuccess2018 Bent Flyvbjerg’s work on Major Projects
- 43. Copyright ProjectingSuccess2018 Bent Flyvbjerg’s work on Major Projects
- 44. Copyright ProjectingSuccess2018 Counteracting Bias • Consistency/desirability bias • Recency effects • Human memory bias • Optimism bias • ‘Strategic Deception’ Data
- 45. Copyright ProjectingSuccess2018 Fundamentally: • What is the predisposition of the work to variance? • Can we predict it? • How do we test for it? • How do we treat it and change the future? Evidence based, tempering against bias. Project DNA
- 46. Copyright ProjectingSuccess2018 Barriers to Adoption Its not on the corporate ‘to do’ list • Lack of a shared vision • Lack of evidence to support the vision • Lack of skilled horsepower • Lack of data • Siloed • Poor quality • Understanding the investment case
- 47. Copyright ProjectingSuccess2018 Summary • The majority of organisationsin the sample are not leveraging experience • The opportunityto leverage experience is significant • Potentialimpact on project delivery productivityis huge (up to 30%) • The approachesdo not differentiate between P3M and technicallessons. They are different! • Does your organisation believethere is value in leveragingpast investments? • STOP boxticking processes and START to link valueto your particularchallenge • Pivot towardsa data driven insights approach where evidence shapes future decisions • Deliver information and insights when required, in the right form, which are consumable • Supported by wikis, peer assists, communities of practice etc. • Machine learning is marching towardsus which will be transformational. • We need to developan approachto support it. "Those that fail to learn from history are doomed to repeat it."
- 48. Copyright ProjectingSuccess2018 • Free to join • Free to attend • Crosses professional boundaries • Developing a community • Developing into a force for good Developing a Community
- 49. Copyright ProjectingSuccess2018 Any questions, comments…. …. Or could you share your own experience?