IT Architecture’s Role In Solving Technical Debt.pdf

IT Architecture’s Role In
Solving Technical Debt
Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney
https://www.amazon.com/dp/1797567616

Topics
• Technical Debt Complete View
• Technical Debt As A Security Risk
• Is Technical Debt Inevitable?
• Sources And Causes Of Technical Debt
• Technical Debt Within New Solutions
• Shadow IT And Technical Debt
• Technical Debt Assessment And Resolution
• Approaches To Solving Technical Debt
March 29, 2022 2

Definition
• Technical debt is an overworked term without an effective and
common agreed understanding of what exactly it is, what
causes it, what are its consequences, how to assess it and what
to do about it
• Technical debt is the sum of additional direct and indirect
implementation and operational costs incurred and risks and
vulnerabilities created because of sub-optimal solution design
and delivery decisions
• Technical debt is the sum of all the consequences of all the
circumventions, budget reduction, time pressure, lack of
knowledge, manual workarounds, short-cuts, avoidance, poor
design and delivery quality and decisions to remove elements
from solution scope and failure to provide foundational and
backbone solution infrastructure
March 29, 2022 3

The Cycle Of Technical Debt
March 29, 2022 4
Circumventions, budget reduction,
time pressure, lack of knowledge,
manual workarounds, short-cuts,
avoidance, poor design and delivery
quality and decisions to remove
elements from solution scope and
failure to provide foundational and
backbone solution infrastructure
Enduring excess cost, resources,
effort, inefficiency, inflexibility, risks
and vulnerabilities
Sub-optimal solutions,
workarounds, shadow IT, unfulfilled
later demand for IT solutions
Negative feedback cycle - short
solution lifespan, earlier solution
replacement, short-term tactical
remedial actions

Technical Debt Complete View
• Failing to take a complete view of the technical debt within
the organisation means problems and risks remained
unrecognised and unaddressed
− The real scope of the problem is substantially underestimated
• Failure to take account of all aspects of technical debt in
turns leads to more technical debt in the form of Shadow
IT
• The first lesson is to understand the entirety of the
organisation’s technical debt
− This will enable informed decisions to be made on what to do
March 29, 2022 5

Technical Debt
• A complete view of technical debt includes:
− The enduring consequence of the sub-optimal implementation of the components of a
solution that cause persisting inefficiencies, additional costs and/or risks other than those
expected
− Deficiencies in the implementation and operation of IT solutions due to the absence of
necessary common infrastructural components or where those available components are of
a poor quality – Structural And Inherited Technical Debt
− The continued use of and the systematic failure to fix older solutions that no longer fully
meet solution consumer needs, that have to be supplemented with bridging solutions and/or
that have increasing operational costs – Zombie Solutions
− The retention of older solutions that require the use of old unsupported solution
components and the systematic failure to fix them that give rise to additional costs and/or
increase risks – Ailing Solutions
− The failure or the deferral of a decision to implement solutions that would meet valid
consumer needs or that would solve business problems or that would deliver real benefits
leading to continued inefficiencies or opportunities not exploited – the breeding ground for
Shadow IT – Unfulfilled Latent Demand
− The direct acquisition of technology solutions by business solution consumers without the
knowledge of or the involvement of the IT function, giving rise to unknown costs and risks –
Shadow IT
• The complete solution is always much more than software
• Technical debt is always much more than poorly written software (“code smells”)
March 29, 2022 6

True Scope of Technical Debt
March 29, 2022 7
Focussing on
software
aspects of
technical debt
means looking
into a small
hole while
ignoring the
chasm

Zombie
Solutions
Technical Debt Landscape
March 29, 2022 8
New Sub-
Optimally
Implemented
Solutions
Ailing
Solutions
Unfulfilled Latent
Demand
Shadow IT Solutions
Active
Technical Debt
Passive and Hidden
Technical Debt

Technical Debt Traditional Focus
March 29, 2022 9
Traditional
Narrow Focus of
What Constitutes
Technical Debt
Traditional Focus of Technical Debt Discussions
And Analyses Is Development And
Infrastructure, Ignoring The Broader Scope Of
The Problem
Unfulfilled Latent
Demand
Shadow IT Solutions
Zombie
Solutions
New Sub-
Optimally
Implemented
Solutions
Ailing
Solutions

Technical Debt Landscape
• Active Technical Debt – problems in this domain are
understood but perhaps not quantified and technical debt
arises from, possibly poor, decisions made, perhaps out of
necessity, or deliberately not made or accidental or
inadvertent consequences of actions and decisions
• Passive and Hidden Technical Debt – this is all too
frequently ignored and little, if any, effort is allocated to
understanding its size and impact
− Unfulfilled Latent Demand is the breeding ground for Shadow IT
March 29, 2022 10

Shining A Light Into The Dark Areas Of Technical
Debt
• An effective
approach to
technical debt
discovery will
shine a light to
the darker areas
of shadow IT use
within the
organisation
March 29, 2022 11

Errors Leading To Technical Debt
Error Types
Principle/
Fundamental
Knowledge
Gaps in Knowledge About the Technology
or Business Need
Design Failures in the Design Process
Inclusion Too Much
Excess and Unnecessary Functionality
Leading to Excess Cost, Elongated Schedule
Omission
Accidental
Required Functionality Accidentally
Omitted
Deliberate Strategic Misrepresentation
Commission
Poor Quality Lack of Quality in the Delivery Process
Removal
Functionality Removed During Delivery
Due to Budget and Schedule Exigencies
March 29, 2022 12

Errors Leading To Technical Debt
• There are many different types of errors that can give rise
to technical debt
March 29, 2022 13

Technical Debt As A Security Risk
March 29, 2022 14

Technical Debt As A Security Risk
• Technical debt can introduce security risks and
vulnerabilities into the organisation’s solution landscape
• Failure to address technical debt leaves exploitable
security risks and vulnerabilities in place
March 29, 2022 15

Solutions Need To Incorporate Protection
• Unauthorised access to solution functionality and its data
involves some or all of:
March 29, 2022 16
• Getting the solution to do something it
should not
• Stopping the solution from working as it
should or enabling it to be bypassed
• Getting consumers of the solution to perform
actions they should not
• Gaining unauthorised access to the solution
as a solution consumer
• Getting the data held in the solution
• Damaging the solution to prevent its use
• Denying access to the solution
• Using the solution as a gateway to other
organisation solution and data assets
• Stealing data to sell
or holding for
ransom
• Collecting ransom
before application or
data restored
• Using the application
to steal money
• Causing reputational
damage
• Stealing intellectual
property
• Putting the company
out of business
With
The
Aims Of

Solution Inheritance Of Security Infrastructure
• Individual solutions must be able to inherit security controls, facilities and
standards from common enterprise-level controls, standards, toolsets and
frameworks.
• Individual solutions must not be forced to implement individual
infrastructural security facilities and controls
− This is wasteful of solution implementation resources, results in multiple non-
standard approaches to security and represents a security risk to the organisation
• Solution architects must be aware of the need for solution security and of
the need to have enterprise-level controls that solutions can adopt.
• The extended solution landscape potentially consists of a large number of
interacting components and entities located in different zones, each with
different security profiles, requirements and concerns
− Different security concerns and therefore controls apply to each of these
components
• Solution security is not covered by a single control
− It involves multiple overlapping sets of controls providing layers of security
March 29, 2022 17

Is Technical Debt Inevitable?
March 29, 2022 18

In the red
corner we
have Gall’s
Law:
A complex
system that
works is
invariably
found to
have
evolved
from a
simple
system that
works
March 29, 2022 19
In the blue
corner we have
Ashby’ Law Of
Requisite
Variety:
The solution
must as a
complex as the
streams of
information,
inputs and
actions it is
designed to
handle

• Gall’s Law 1
• A Complex System That Works Is Invariably Found To Have Evolved From A
Simple System That Works
− The parallel proposition is also asserted to be true:
• A Complex System Designed From Scratch Never Works And Cannot Be
Patched Up To Make It Work. You Have To Start Over, Beginning With A
Working Simple System
• So, in order to guarantee that you have a solution that
works and is used, you have to start with a simple sub-
optimal solution that can evolve the necessary complexity
over time as the work is funded and as the true nature of
the solution requirements become known through use and
feedback
March 29, 2022 20
1 SYSTEMANTICS: How Systems Really Work and How They Fail John Gall ISBN 978-0812906745

• Ashby’s Law Of Requisite Variety 1
− Essentially the system as a regulator requires an equivalent amount of
variety (processing functionality) as the information or signals it must
handle. That is, the solution must as a complex as the streams of
information, inputs and actions it is designed to handle.
• This is further elaborated by Roger C. Conant and W. Ross
Ashby in their Good Regulator Theorem 2
− "every good regulator of a system must be a model of that system“
• So, in order to usable and operable the solution must
incorporate the necessary complexity
• Where the solution does not include sufficient complexity and
functionality, it will not work as needed and will contains gaps
March 29, 2022 21
1 An Introduction to Cybernetics W Ross Ashby ISBN 978-1614277651
2 Int. J. Systems Sci., 1970, vol. 1, No. 2, 89-97

March 29, 2022 22
Galls’s Law – Start Small and Simple and Add
Complexity Incrementally
Ashby’s Law – Necessary Complexity Is
Required From the Start in Order to for
the Solution to Work as Needed
• Both approaches give rise to technical debt
But All the Requirements Can Never Be
Exactly Known in Advance and the
Inevitability of Components Being Dropped
Because of Delivery Exigencies

Technical Debt Account/Register
• A technical debt account/register should be maintained by
solution design and delivery activities explicitly listing
decisions and actions that will give rise to technical debt,
similar to a risk register
• Recognition that technical debt is the consequence of a
pragmatic reality
• Allows informed decisions to be made on resolution
actions
March 29, 2022 23

Sources And Causes Of Technical Debt
March 29, 2022 24

Sources And Causes Of Technical Debt
• Technical debt arises from many sources and for many
reasons
− Short-cuts and other scope-related decisions taken during
solution design and delivery
− Strategic misrepresentation
− Enterprise architecture failures to have full set of infrastructural
foundational solutions
− Suppliers and other third-parties
− Shadow IT and business bypassing IT solution acquisition
processes
− Use of new unfamiliar and unproven technologies
March 29, 2022 25

What Gives Rise To Technical Debt In New Solutions
March 29, 2022 26
Strategic
Misrepresentation
Business Requests
for IT Solutions
Ignored or Rejected
Solution Scope Reduced
Due to Budget or Time
Pressures
Solution Design Does
Not Capture Necessary
Complexity
Solution Design
Based on Unproven
Technology
Solution Design
Based on Unproven
Technology
What Is
Needed
What Gets
Delivered
Inadequate
Budget

What Gives Rise To Technical Debt In New Solutions
– Another View
• Another view is that pressures to deliver a basic working solution within
the allocated and available time and budget causes functionality to be
removed/deferred leading to an initial poor/quality/sub-optimal solution
that can be remediated later
March 29, 2022 27
What Is Originally
Planned
What Gets
Delivered
Functionality Dropped Due to
Time and Budget Pressures

What Should Happen During Subsequent Delivery
Stages and Phases
• What should happed, but rarely does, is that removed/deferred
functionality should be added during later stages and phases
• What was dropped typically remains dropped
• There is no record or register of these deferral decisions that
would enable the technical debt to be repaid
March 29, 2022 28

Strategic Misrepresentation
• This is the deliberate, systematic distortion or
misstatement of facts such as costs, time, complexity or
risk in order to get a solution delivery project approved
initiated
− Subsequent solution scope changes are not really changes due to
new functionality requirements being uncovered but knowable
requirements and needs ignored
• Do not underestimate the contribution of strategic
misrepresentation to solution scope, budget, functionality
and timeline problems
March 29, 2022 29

Suppliers As A Source Of Technical Debt
• Organisations outsource IT solutions and services to third-parties in many
ways:
− IT outsourcing arrangements
− Development of solutions assigned to third-parties
− Use of hosted/cloud solutions
− Use of external service providers
− Process outsourcing
− Resource outsourcing
− Use of hosted/cloud platforms or infrastructure
− Acquisition of products developed by third-parties
− Incorporation of third-party components into solutions
• When you outsource you inherit supplier technical debt
• Third-parties externalise their technical debt to their customers
• The impact of weaknesses in third-parties will be felt by their customers
• How often do you audit the products and services from suppliers for
technical debt?
March 29, 2022 30

Suppliers As A Source Of Technical Debt
March 29, 2022 31
Resource
Outsourcing
Process
Outsourcing
Cloud
Solutions
Development
Outsourcing
Service
Outsourcing
Data Centre
Outsourcing
Product
Acquisition
Component
Acquisition
Flow of Technical Debt

Common Solution Delivery And Operation
Framework
March 29, 2022 32
Common Service Management
Processes and Standards – solution
support, service level management
Common Financial
Management Processes and
Standards – solution cost and
asset management
Common Enterprise
Architecture Standards –
compliance with organisation
technology standards and
principles
Common Security
and Regulatory
Compliance
Architecture –
integration of
solution into overall
security standards
and operations
Common Data Architecture –
integration of solution data
into the organisation data
model and access to solution
data, compliance with data
regulations and standards
Business Process and
Organisation
Structure – business
processes and
organisation
functions that use the
solution
Extended Solution
Landscape With
Integration With Other
Solutions – solution
support, service level
management, integration,
data exchange
Individual Solution
Landscape – set of
components that comprise
the overall solution
Common Data Integration
Architecture – common data
integration, exchange, access
platform and standards

Common Solution Delivery And Operation
Framework
• Individual solutions should ideally be delivered and
operate in a common framework where infrastructural
facilities are provided by common cross-functional
components defined, implemented and maintained by
Enterprise Architecture
• Solutions should be able to inherit these common,
enterprise-wide, enterprise-level foundational facilities
• Where those common infrastructural components do not
exist, individual solutions have to implement them,
generally in tactical and sub-optimal ways
March 29, 2022 33

Gaps In Common Solution Delivery And Operation
Framework
• Gaps in common
backbone facilities
means individual
localised tactical
and expedient
approaches are
frequently taken
during solution
delivery leading to
technical debt
• Individual solutions
must fill the gaps
as best they can
March 29, 2022 34

Monolithic Solutions And Technical Debt
• Large monolithic solutions subject to lengthy initial designs
effort and duration frequently give rise to technical debt
• Elements of the design are invariably dropped during
implementation
March 29, 2022 35

Replacing The Solution Monolith
• Traditional solution delivery journey can
feel like pushing a monolithic solution
design up a hill
− Inflexible and resource-intensive
• A more agile approach replaces this
monolithic straight-line journey with a
faster but more circuitous journey
March 29, 2022 36
• The Solution Monolith is the large up-
front solution design with too much
unnecessary functionality handed over
to solution delivery and pushed
through to implementation

Monolithic Solution Design And Delivery And
Solution Shrinkage
• An almost unvarying characteristic of monolithic
solution design and delivery is that the monolith is
chipped away and is reduced as functionality and
quality are removed to meet time, cost and risk
constraints, as both schedule and budget overrun
• Dropped functional elements are a frequent source of
technical debt
March 29, 2022 37

New Technologies And New Solution Models As
Sources Of Technical Debt
• New solution development, deployment and operating options
give rise to the opportunity for additional technical debt
− New solution design, deployment and operating models
− Distributed solution components, distributed solution consumer base,
distributed access with many interfaces, integration points and data
flows
− Complexity with multiple handoffs gives rise to gaps in end-to-end view
and knowledge leading to risks
• New technologies introduce new risks, direct and indirect
− Lack of familiarity of and experience with technology increases the
likelihood of technical debt
− Underlying technology components subject to rapid change
− Multiple overlapping configuration options
− New technology is less proven and contains more errors
• Solution risk and security status is becoming harder to track
March 29, 2022 38

New Technology And Technical Debt
March 29, 2022 39
Solution
Technical
Debt
Dispersed
Operational
Solution
Landscape
New
Unfamiliar
Technologies
Error Prone
Technology
Deployment
And
Operation
Rapidly
Changing
Underlying
Technologies
And
Platform
More And
Diverse
Solution
Components
Lack Of
Skills And
Experience In
New
Technologies
Multiple
Different
Technologies
Greater
Complexity
And
Fragility

Technical Debt Within New Solutions
March 29, 2022 40

Scope Of Complete Solution
March 29, 2022 41
Changes to Existing Systems
New Custom Developed Applications
Information Storage Facilities
Acquired and Customised Software Products
System Integrations/Data Transfers/Exchanges
New Business Processes
Organisational Changes
Reporting and Analysis Facilities
Existing Data Conversions/Migrations
Changes to Existing Business Processes
New Data Loads
Training and Documentation
Central, Distributed and Communications Infrastructure
Application Hosting and Management Services
Cutover/Transfer to Production
Parallel Runs
Enhanced Support/Hypercare
Sets of Maintenance, Service Management and Support Services
Operational Functions and Processes
Sets of Installation and Implementation Services
Component 1
Component 2
Component 3
Component 1
Component 2
Component 1
Component 2
Component 3
Component 1
Component 2
Component 1
Component 2
Component 1
Component 2
Component 1
Component 2
Component 3
Component 1
Component 2
Component 3
Component 1
Component 2
Component 1
Component 2
Component 1
Component 2
Component 3
Component 1
Component 2
Component 1
Component 2
Component 3
Component 1
Component 2
Component 1
Component 2
Component 1
Component 2
Component 1
Component 2
Component 3
Component 1
Component 2
Component 3
Component 1
Component 2
Component 1
Component 2
Component Types
Delivery Of Solution Components Of Component Type
The Complete Solution Is
Considered Delivered And
Operational When All
Components Of Each
Component Type Have Been
Implemented
Component
Delivery Occurs
Throughout The
Solution Delivery
Timeline

Solution Implementation And Operation
March 29, 2022 42
Solution As Designed And
Implemented Consists Of A Number
Of Components Of Different
Component Types
Operational Solution Consists
Of Those Solution
Components That Endure
After Delivery
Sub-Optimal Operational
Solution Components Give
Rise The Consequences Of
Technical Debt
Sub-Optimal Solution Design or
Dropping of Necessary Solution
Components Give Rise To Technical
Debt

Scope Of Complete Solution And What Gets
Implemented
• Active technical debt arises
from decisions made or
deliberately not made
regarding solution
components
• Technical debt arises from
many actions, activities and
decisions made during
solution design and
delivery
March 29, 2022 43
Implemented
As Designed
and Needed
Partially or
Inadequately
Implemented
Not
Implemented
Sources
Of
Technical
Debt

Consequences Of Technical Debt
• Direct or Consequential Additional Financial Cost – initial
and recurring additional unplanned or unbudgeted costs
or required unplanned activities that ultimately have a cost
• Indirect Risks – additional risks and security deficiencies
introduced
− Risks and vulnerabilities are potential rather than actual – they
only become actual when the risk occurs and the security
vulnerability is exploited with negative outcomes
March 29, 2022 44

Consequences Of Technical Debt
Consequences
Of Technical
Debt
Inefficiencies in solution use and operation
Additional costs to complete gaps
Significant rework or replacement required
Poor usability
Solution partial or complete rejection by consumers
Increased administration, maintenance and support cost/effort
Solution can not evolve easily to meet new and changed requirements
Requirements and business needs change during the solution delivery
timeline but the solution design does not change
Uncosted/unknown peer support required or arises
Security vulnerabilities
Increased operational and other risks
Reduced functionality requiring workarounds
Shorter solution life and premature replacement
Additional solution components needed
Performance and/or operational problems
March 29, 2022 45

Technical Debt Over Time
• The
accumulation of
technical debt
over time is the
sum of the
consequences
over time
• The profile of
technical debt
accumulation
will vary for
each affected
solution
March 29, 2022 46

Organisation Technical Debt
• The sum of an organisation’s technical debt is the sum of the debt over all
types of consequence and over all categories of debt for all solutions
• Only if you include all technical debt categories do you get a true
understanding of the size of the problem
March 29, 2022 47
New Sub-
Optimally
Implemented
Solutions
Zombie
Solutions
Ailing
Solutions
Unfulfilled
Latent
Demand
Shadow IT
Solutions

Actions, Activities And Decisions Taken And Not Taken
Leading To Solution Component Technical Debt – 1
March 29, 2022 48
Changes to
Existing Systems
Existing systems
cannot be
changed to meet
the complete
needs of the
target solution so
workarounds
needed
Changes are
poorly designed
and developed
Required changes
are omitted
Required
functionality
dropped from
changes
implemented
New Custom
Developed
Applications
Custom solution
components
poorly designed
and developed
Required
applications are
omitted
Required
functionality
dropped from
applications
developed
Acquired and
Customised
Software
Products
Incorrect or
inadequate
products selected
Products dropped
from solution
scope and not
acquired
Products not
configured
correctly
Products require
customisation to
operate correctly
System
Integrations/
Data Transfers/
Exchanges
No integration
platform and
individual point-
to-point
integrations
implemented
Integrations
unreliable and
prone to failure
Integrations not
fully automated
Reporting and
Analysis Facilities
Functionality
removed from
solution scope
and not
implemented
Inadequate
reporting and
analysis facilities
implemented
Sets of
Installation and
Implementation
Services
Services not
acquired and
scope or services
insufficient
Services not
performed
correctly
Information
Storage Facilities
Services not
acquired and
scope or services
insufficient
Services not
performed
correctly
Existing Data
Conversions/
Migrations
Data not or only
partially
converted
Poor quality data
conversion
Target data
model cannot
accommodate the
required data
New Data Loads
Data not or only
partially loaded
Poor quality data
load
Target data
model cannot
accommodate the
required data
Central,
Distributed and
Communications
Infrastructure
Inadequate or
insufficient
infrastructure
acquired
Unreliable
infrastructure
acquired
Inadequate or
insufficient
infrastructure
capacity planning

Actions, Activities And Decisions Taken And Not Taken
Leading To Solution Component Technical Debt – 2
March 29, 2022 49
Cutover/ Transfer
to Production
And Support
Cutover to
production not
planned
Cutover to
production not
performed
correctly
Operational
Functions and
Processes
Required support,
housekeeping
and operational
processes not
defined
Processes poorly
implemented
Processes do not
cover the
required scope
Parallel Runs
Parallel runs not
performed
Inadequate or
insufficient
parallel runs
Enhanced
Support/
Hypercare
No or insufficient
enhanced support
acquired
Sets of
Maintenance,
Service
Management and
Support Services
Inadequate
support leading
to unrecognised
peer support
effort
Services not
performed
correctly
Application
Hosting and
Management
Services
Selected platform
is incomplete and
has gaps and
flaws
Platform is new
so skills and
experience in its
optimal and
secure use are
not available
Platform has
insufficient
capacity
Changes to
Existing Business
Processes
Business
processes not
changed to take
advantage of new
solution
Existing business
processes not
defined
New Business
Processes
New business
processes not
defined to take
advantage of new
solution
Inadequate or
insufficient
inventory of new
business
processes created
Target data
model cannot
accommodate the
required data
Organisational
Changes,
Knowledge
Management
New or changed
organisation
operating model
not defined
Organisation
changes not
planned or
implemented
No or insufficient
knowledge
management
Sufficient
personnel with
necessary skills
and experience
not allocated
Training and
Documentation
No or insufficient
training at all
required levels
No or insufficient
system and user
documentation
Inadequate or
insufficient
infrastructure
capacity planning

Pre-Solution Implementation Causes Of Technical
Debt
• The solution may be poorly designed or may not be linked to correctly
identified business need
− Poor initial requirement definition
− Poor requirements validation
− Poor management of requirements
− Requirements not linked to business benefits
− Solution design not validated
− Solution design not linked to business needs
− Solution design too complex
− Solution design does not capture necessary complexity
− Solution design based on unproven technology
− Solution not implementable
− Underlying business processes not defined adequately
− Errors due to limitations in estimating procedures
− Failure to understand and account for technical risks
− Deliberate underestimation/misrepresentation of costs
− Poor inflation estimates
− Top down pressure to reduce estimates
− Lack of valid independent cost estimates
March 29, 2022 50

Shadow IT And Technical Debt
March 29, 2022 51

Shadow IT And Technical Debt
• Shadow IT or ghost IT is a largely unrecognised source of
technical debt including security risks and vulnerabilities
• Shadow IT is the consequence of a set of reactions by business
functions to an actual or perceived inability or unwillingness of
the IT function to respond to business needs for IT solutions
− End User Computing – the business develop the solution themselves
− Direct Business Sourcing of IT Solutions – the business sources the IT
solution from a service provider in an uncontrolled manner, either as a
product installed within the organisation or as a service delivered
through a hosted product
− Outsourcing Of IT Services – the business takes a strategic decision to
outsource elements of the internal IT service as a way of dispensing
with the need for the internal IT function
− Abandonment Of Solution Need – the business need remains latent,
unfulfilled and in the shadows
March 29, 2022 52

The Source Of Shadow IT
March 29, 2022 53
Business
Objectives
Business
Operational
Model
Solution
Portfolio
Realisation
And
Delivery
Solution
Usage,
Management ,
Support
And
Operations
Business
Strategy
Business
IT
Strategy
Solution
Portfolio
Design And
Specification
Business
shadow IT
expenditure External
Suppliers and
Service
Providers
External
Suppliers and
Service
Providers
Business-perceived or actual
barriers to solution delivery by
internal IT organisation
Shadow IT solutions
ultimately may be
passed to the
support function
At least 40% of technology
spending is diverted from IT
Over 30% of CIOs
routinely not consulted
on IT solution acquisition
and expenditure
Them Us
Them and
Us Mentality

Core Solution Business Processing Stages And
Shadow IT
March 29, 2022 54
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Step 12
Extract
Data and
Analyse
Outside
Solution
Extract
and
Exchange
Data With
Other
Party
Reporting
Using
Separate
Solution
Use
Separate
Tool To
Perform
Work
Extract
and Send
Data
Outside
Party
Manually
Enter
Output
from
External
Solution
Perform
Additional
Steps
Using
Separate
Solution
Reporting
and
Analysis
• Shadow IT is frequently needed to make up for gaps in core business solutions,
supplementing incomplete solutions and providing omitted functionality
• These components are needed to link business solutions together into an
operational reality
• These Shadow IT components represent potential or actual technical debt
Functionality Of Delivered Business Solution
Shadow
IT

Core Solution Business Processing Stages And
Shadow IT
• Use of shadow IT solutions occurs routinely at multiple stages
throughout the use of business systems, extending and enhancing
their functionality or providing features not available or that area
easier to use
March 29, 2022 55
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Step 12
Extract
Data and
Analyse
Outside
Solution
Extract
and
Exchange
Data With
Other
Party
Reporting
Using
Separate
Solution
Use
Separate
Tool To
Perform
Work
Extract
and Send
Data
Outside
Party
Manually
Enter
Output
from
External
Solution
Perform
Additional
Steps
Using
Separate
Solution
Reporting
and
Analysis
Shadow IT Occurs Pervasively Throughout the Use of Core IT Solutions

The Long Long Shadow Of Shadow IT
March 29, 2022 56
Shadow IT
Shadow
Projects
Shadow
Sourcing
Shadow
Development
Shadow
Solutions
Shadow
Support
Arrangements
May Give
Rise To
May
Involve
May
Involve
May Be Included In
Projects that have never been
subject to a formal evaluation
and approval process, formally
managed and tracked and who
success or failure is not
recorded
Unapproved usage of third party
services or product and service suppliers
in the business not subject to format
evaluation and approval process
including costing and quality and that
not is formally recorded and tracked
Custom development of
solutions performed by
business personnel or
contracted to third-parties not
subject to formal design and
delivery approaches including
testing and quality
Solution that comprises an
information technology system
that is developed or sourced and
implemented by business users
that is not approved by the IT
function and is not
part of the organisation's
accepted, documented and
supported information
technology infrastructure
portfolio
Shadow
Data
Gives
Rise
To
May
Involve
Which
Require
Informal, undocumented,
unrecorded, uncosted and
untracked arrangements to
provide support for a shadow
IT solution that typically
involves effort by unapproved
third parties or by business
personnel for whom providing
support is not their formal
role
Uncontrolled copies or
extracts of data from
formal IT solutions stored
outside formal data
structures or data
generated by shadow IT
solutions that may be
held separately from
formal data structures or
that may be partially of
completely entered into
formal data structures
Use And/Or
Generate

Types Of Shadow IT Solution
• Shadow IT takes many forms and types
1. CUST – customised solution developed by a third-party
2. DEV – personal devices used to access business systems or
authenticate access to hosted solutions used for business
3. DIY – end-user computing application developed by the business
4. HOME – organisation data sent to home devices to be worked on
5. MSG – public messaging and data exchange platforms
6. OPEN – open-source software used as a stand-alone solution or
incorporated into other solutions
7. OUT – outsourced service solution
8. PROD – software product acquired by the business and implemented
on organisation infrastructure
9. PUB – accessing organisation applications and data using public
devices or networks
10. STOR – public data storage and exchange platforms
11. SVC – hosted software solution
March 29, 2022 57

Shadow IT Landscape
March 29, 2022 58
Core IT Solutions
On
Premises
EUC/DIY
Solutions
On Premises
Product Solutions
Hosted
Product
Solutions
Outsourced
Service
Solutions
Personal
Devices
DEV
SVC
OUT
PROD
DIY
On
Premises
Third-Party
Custom
Solutions
CUST
Open
Source
Software
OPEN
Use of
Public
Networks
or Devices
PUB
Send
Data to
Home
Devices
HOME
Public
Messaging
Platforms
MSG
Within The Organisation Outside The Organisation
Public Data
Storage and
Exchange
Platforms
STOR

State Of Shadow IT – It’s Not Pretty
March 29, 2022 59
Spending
Decision
Making
Cloud and
Data
Knowledge
Estimated Spending on
Shadow IT:
2013 – 40% of Total1
2017 – 50% of Total2
76% of CIOs Do
Not Know
Spending on
Cloud3
54% of CIOs Do Not Know The Number of
Cloud Services Being Used4
The Business Uses 15 Times The
Number of Cloud Applications
IT Believe They Use12
90% of CIOs Are Bypassed
Sometimes By Business in IT
Spending7
31% of CIOs Are
Routinely
Bypassed By
Business in IT
Spending8
86% of Cloud Applications
Represent Unsanctioned
Shadow IT9
Only 8 % of Companies
Know the Scope of
Shadow IT10
58% of CIOs are Worried
About the Spiralling Cost
of Cloud Sprawl5
1 https://www.forbes.com/sites/tomgroenfeldt/2013/12/02/40-percent-of-it-spending-is-outside-cio-control/
2 https://www.everestgrp.com/2017-04-eliminate-enterprise-shadow-sherpas-blue-shirts-39459.html/
3,4,5 https://www.trustmarque.com/wp-content/uploads/2018/03/Cloud_Sprawl_and_Shadow_IT_Trustmarque.pdf
6,11 https://go.nttict.com/the-growth-of-shadow-IT-and-why-many-enterprises-are-now-dependent-on-it.html
7,8 https://www.logicalis.com/news/cios-line-up-to-transform-it-in-response-to-the-shadow-it-phenomenon/
9 http://pages.ciphercloud.com/rs/ciphercloud/images/CipherCloud-Cloud-Adoption-and-Risk-Report.pdf
10 https://downloads.cloudsecurityalliance.org/initiatives/surveys/capp/Cloud_Adoption_Practices_Priorities_Survey_Final.pdf
12 https://blogs.cisco.com/cloud/shadow-it-rampant-pervasive-and-explosive
80% of Business Decision Makers
Believe that Data Stored in
Shadow IT is Critical to their
Departments6
80% of Business Decision Makers
Admit that Employees in their
Department Were Using Cloud Services
Without the IT Department’s
Knowledge11

Technical Debt Assessment And Resolution
March 29, 2022 60

Technical Debt Resolution Approaches
March 29, 2022 61
Severity
Of
Solution
Problems
Solution
Replaceability
Strategic Importance Of Solution
Net Solution Finances
Low High
Low High
Low Low
High
High
Retain
Repair
Retire
Reserve
Reduce
Refactor
Reassemble
Redevelop
Replace
Rehost
Replatform

Technical Debt Resolution Approaches
• Reassemble – combine functionality of solution with other solutions to create new combined
solution
• Redevelop – redevelop the custom application and retain its functionality
• Reduce – stop using some functional elements of the solution while retaining it
• Refactor – change the internal solution structure, design and implementation without changing
the external appearance and functionality
• Rehost – move solution to new platform without change
• Repair – resolve problems and issues with the current application while retaining it on the same
platform
• Replace – replace the solution with a functionally similar one
• Replatform – move solution to new platform with some limited changes to enable the solution
run on the new target platform
• Reserve – retain the solution but encapsulate access to its functionality via some form of
interface
• Retain – retain the solution entirely in its current form
• Retire – stop using an obsolete solution without explicitly replacing it
• Each organisation should agree the spectrum of resolution approaches and where the lie on the
four axes.
March 29, 2022 62

Technical Debt Evaluation
• Four axes of evaluation
− Severity Of Solution Problems
• Factors including what is the current negative impact of the solution, manual processing, inefficiencies and
gaps, use of bridging solutions to link solution components, administration effort, support effort, existence
and scale of peer support, what is the impact of gaps with the current solutions, introduction of and
measure of severity of operational and security risks and vulnerabilities, existence of and severity of
performance problems, difficulty of evolving to meet known new needs, degree to which solution has been
rejected by target consumers, age and supportability of underlying technology components, age and
supportability of underlying infrastructure components, difficulty of data extraction for reporting and
analysis, non-compliance with current organisation IT standards
− Solution Replaceability
• Factors including what options to replace – custom or package, how replaceable, should it be replaced, what
would it take to replace, how long, what resources, what impact, can it be consolidated with other
applications, does replacement approach fit into overall solution acquisition approach, how complex would
the replacement be, complexity of replacement project, number of solutions replaced/complexity reduced
− Strategic Importance Of Solution
• Factors including how many internal and external consumers, measure of importance of solution to business
operations, volume of data being process, measure of business and time criticality, privacy and
confidentiality of data, performance, capacity and throughput considerations, number of business functions
using solution, deployed externally, complexity of underlying business processes
− Net Solution Finances
• Factors including internal cost and effort to support, operate, maintain, support and administer the
application, estimated infrastructure costs, estimated supplier costs, estimated excess costs, estimated cost
of bridging solutions, estimated replacement cost, estimated replacement cost/savings
March 29, 2022 63

Technical Debt Resolution Evaluation
• Evaluate the solution across the four axes
− Severity Of Solution Problems
− Solution Replaceability
− Strategic Importance Of Solution
− Net Solution Finance
• Assign a weight or relative importance to each axis
• Respond to questions in each axis
• Assign an importance and a rating to each question – score
= importance x rating
• Generate a recommendation based on the response
March 29, 2022 64

Technical Debt Resolution Evaluation – Severity Of
Solution Problems
SEV-1 Assessment of Negative Technical Impact of Solution
SEV-2 Assessment of Negative Business Impact of Solution
SEV-3 Assessment of Negative Operations and Support and Administration Impact of Solution
SEV-4 Assessment of Negative Maintenance Impact of Solution
SEV-5 Assessment of Additional and Unnecessary Manual Processing Created as a Result of Deficiencies
SEV-6 Assessment of the Inefficiencies and Gaps in the Solution
SEV-7 Assessment of the Scale of End-User Developed Bridging Solutions Used
SEV-8 Assessment of the Scale of Peer Support Effort
SEV-9 Assessment of the Severity of Risks Introduced by the Solution
SEV-10 Assessment of the Severity of Security Vulnerabilities Introduced by the Solution
SEV-11 Assessment of Performance Problems with the Solution
SEV-12 Assessment of Unnecessary Complexity in or Gaps with Solution Integration
SEV-13 Assessment of Unnecessary Complexity in or Gaps with Solution Data Model, Data Architecture, Data
Management
SEV-14 Assessment of the Need to and Difficulty in Evolving the Solution to Meet Changed Business Needs
SEV-15 Assessment of the Degree of Rejection of the Solution by its Target Consumers
SEV-16 Assessment of the Age and Support Problems with Underlying Technology Components
SEV-17 Assessment of the Age and Support Problems with Underlying Infrastructure Components
SEV-18 Assessment of the Difficulty of Extracting Data for Reporting and Analysis
SEV-19 Assessment of Solution's Non-Compliance with Organisation Technology Standards
March 29, 2022 65

Technical Debt Resolution Evaluation – Solution
Replaceability
March 29, 2022 66
REP-1 Assessment of the Feasibility of Replacing the Solution by Existing Solutions, Possibly with Some Enhancements
REP-2 Assessment of the Option to Stop Using the Solution
REP-3 Assessment of the Existence of Packaged On-Premises Options to Replace Solution
REP-4 Assessment of the Existence of Packaged Hosted Options to Replace Solution
REP-5 Assessment of the Existence of Service Options to Replace Solution
REP-6 Assessment of the Length of Any Replacement Project
REP-7 Assessment of the Resources Required for Any Replacement Project
REP-8 Assessment of the Complexity of Any Replacement Project
REP-9 Assessment of the Organisational Impact of Any Replacement Project
REP-10 Assessment of the Number of Solutions Replaced and Reduction in Complexity by a Replacement Solution

Technical Debt Resolution Evaluation – Strategic
Importance Of Solution
March 29, 2022 67
STR-1 Assessment of the Importance of the Solutions to Business Operations
STR-2 Assessment of the Number of Internal Solution Consumers
STR-3 Assessment of the Number of External Solution Consumers
STR-4 Assessment of Number of Business Functions Using Solution
STR-5 Assessment of the Time Criticality of Processing by the Solution
STR-6 Assessment of Degree of Processing of Confidential Data
STR-7 Assessment of Degree of Processing of Personal Data
STR-8 Assessment of the Time To Recover the Solution in the Event of Failure
STR-9 Assessment of Complexity of Underlying Business Processes
STR-10 Assessment of the Volume of Data Being Processed
STR-11 Assessment of the Remaining Life of the Solution

Technical Debt Resolution Evaluation – Net Solution
Finance
March 29, 2022 68
FIN-1 Assessment of the Excess Internal Cost to Support, Maintain, Operate and Administer the Solution
FIN-2 Assessment of the Excess External Cost to Support, Maintain, Operate and Administer the Solution
FIN-3 Assessment of the Excess On-Premises Infrastructure Costs
FIN-4 Assessment of the Excess Hosting/Service Costs
FIN-5 Assessment of the Cost of Bridging Components
FIN-6 Assessment of Estimated Replacement Cost
FIN-7 Assessment of Estimated Replacement Savings

Technical Debt Resolution Evaluation
• Questions with an Importance or Applicability of zero are
ignored when calculating axis score
• Axis score = sum of ratings divided by number of questions
where Importance or Applicability in greater than zero
• Axis score is multiplied by its relative importance
March 29, 2022 69

Technical Debt Resolution Evaluation Dashboard
March 29, 2022 70
1
2
3 4 5 6 7 8 9
10
11

Technical Debt Resolution Evaluation Dashboard -
Elements
1. Evaluation Axis
2. Assessment questions within evaluation axis
3. Importance / Applicability setting for assessment question
4. Assessment score for each question
5. Calculated overall score for each question
6. Relative importance setting for each evaluation axis
7. Evaluation axis raw score – sum of overall scores for each question
8. Average score for each evaluation axis
9. Weighted score for each evaluation axis
10. Graphic representation of weighted scores for evaluation axes
11. Recommendation based on weighted scores for evaluation axes
March 29, 2022 71

Approaches To Solving Technical Debt
March 29, 2022 72

IT Architecture’s Role In Solving Technical Debt
• All the disciplines within IT architecture have a role to play
in promoting an understanding of and in the identification
of how to resolve technical debt
• IT architecture can provide the leadership in both
remediating existing technical debt and preventing future
debt
March 29, 2022 73
IT Architecture
Enterprise
Architecture
Application
Architecture
Business
Architecture
Solution
Architecture
Information
and
Data
Architecture
Security
Architecture
Technical
Architecture
Infrastructure
Architecture
Service
Architecture

IT Architecture’s Role In Solving Technical Debt
March 29, 2022 74
• Promote understanding of technical debt
• Lead initiative to resolve problems
• Define debt-free/low-debt cross-platform
architecture
• Ensure complete set of foundational
solutions are available
• Maintain a
registry of
decisions
that
generate
technical
debt
• Promote business and
IT understanding
• Prevent shadow IT
• Offer consulting service
• Understand design and delivery decisions that cause technical debt
• Create debt-free/low-debt solution designs
• Work on technical debt remediation
• Maintain a registry of decisions that generate technical
• Understand and manage supplier generated technical debt
IT Architecture
Enterprise
Architecture
Application
Architecture
Business
Architecture
Solution
Architecture
Information
and
Data
Architecture
Security
Architecture
Technical
Architecture
Infrastructure
Architecture
Service
Architecture
• Create reusable common data
debt-free/low-debt architecture
• Create and
manage
practical
cross-
platform
security
architecture
• Create debt-free/low-
debt technical designs
• Define debt-
free/low-debt
cross-
platform
infrastructure
architecture
• Maintain a
registry of
decisions that
generate
technical debt

Hgh Level Technical Debt Remediation Activities
• Determine and agree the set of remediation options to be
considered
− The organisation’s IT strategy and in particular the organisation’s digital
strategy and cloud adoption stance will govern the remediation option
spectrum
• Define the relative importance of each assessment axis
• Define the set of solution assessment questions and their
importance and weight
• Conduct solution technical debt assessment and create
technical debt inventory across the solution landscape,
especially any gaps in foundational/backbone solutions
• Define and agree the technical debt remediation approach
• Mobile the technical debt project
March 29, 2022 75

More Information
Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney
https://www.amazon.com/dp/1797567616
29 March 2022 76

IT Architecture’s Role In Solving Technical Debt.pdf

More Related Content

What's hot (20)

Similar to IT Architecture’s Role In Solving Technical Debt.pdf (20)

More from Alan McSweeney (20)

Recently uploaded (20)

IT Architecture’s Role In Solving Technical Debt.pdf