Introduction to Software Evolution: The Software Volcano

Introduction to
Software Evolution:

“The Software Volcano”

Paul Klint

Introduction to Software Evolution 1

Software Volcano

Mt. Etna, Sicily, Italy


The Software Volcano: Languages
Distribution of languages in use, worldwide
Language Used in % of total
COBOL 30
Assembler 10
C 10
C++ 10
550 other languages 40

● For mainframe applications 80% is COBOL!
● Figures taken from Capers Jones (Software
Productivity Research)


Software Volcano: Volume
● The total volume of software is estimated at
7 * 109 function points
● 1 FP = 128 lines of C or 107 lines of COBOL
● The volume of the volcano is
– 750 Giga-lines of COBOL code, or
– 900 Giga-lines of C code

Printed on paper we can wrap planet Earth
9 times!


Software Volcano: Defects
● Observation:
– on average 5 errors (bugs) per function point
– includes errors in requirements, design, coding,
documentation and bad fixes
● The software volcano, world-wide, contains
5 * 7 * 109 Bugs = 35 Giga Bugs

This means 6 bugs per human being on
planet Earth!


Work distribution of programmers
Year New projects Enhancements Repairs Total
1950 90 3 7 100
1960 8,500 500 1,000 10,000
1970 65,000 15,000 20,000 100,000
1980 1,200,000 600,000 200,000 2,000,000
1990 3,000,000 3,000,000 1,000,000 7,000,000
2000 4,000,000 4,500,000 1,500,000 10,000,000
2010 5,000,000 7,000,000 2,000,000 14,000,000
2020 7,000,000 11,000,000 3,000,000 21,000,000

Now: 60% of the programmers work on enhancement and repair

In 2020: only 30% of all programmers will work on new software


Message
● When an industry approaches 50 years of age it
takes more workers to perform maintenance than
to build new products (ex: automobile industry)
● Maintenance and renovation of existing software
become more and more important: avoid that the
software volcano explodes


Software Maintenance & Evolution
● What is Software Maintenance?
● Why evolves software?
● Problems in Software Maintenance
● Solutions


What is Software Maintenance?
● Modification of a software product after delivery
to correct faults, to improve performance or other
attributes, or to adapt the product to a modified
environment (IEEE 1219, 1993)
● Observe that:
– maintenance is seen as after-the-fact activity
– no integration with software development process


A

K
N
O
W
N

P
I
T
F
A
L
L Introduction to Software Evolution 10

Another Classification
● Software maintenance
– Changes are made in response to changed requirements
– The fundamental software structure is stable
● Architectural transformation
– The architecture of the system is modified
– Generally from a centralised to a distributed architecture
● Software re-engineering
– No new functionality is added to the system but it is
restructured and reorganised to facilitate future changes


Why systems change
● Correct errors
● Business pull:
– Business / IT alignment
– Requirements change (legislation, new insights, efficiency)
– Re-organization
– Mergers / take-overs
– New products, marketing actions
– Market hypes (CRM, ERP, BPR, STP)
● Technology push:
– Internet
– Mobile
– Updates of operating system, development environment, databases
– Hardware

Categories of Maintenance
● Corrective: needed to correct actual errors
● Adaptive: result from changes in the environment
● Perfective: modifications to meet the expanding
needs of the user
● Enhancement = Adaptive + Perfective


Cost Distribution per Category

Corrective
20%

Perfective
55%
Adaptive
25%


Costs of Maintenance
● Usually greater than development costs
– 2* to 100* depending on the application
● Affected by both technical and non-technical factors
● Increases as software is maintained
– Maintenance corrupts the software structure, making further
maintenance more difficult
● Ageing software can have high support costs
–old languages, compilers etc.
● Think of your software as continuously evolving


Cost distribution of two systems

System 1

System 2

450 $
0 50 1 00 15 0 200 2 50 3 00 35 0 400 500

Development costs Maintenance costs


Costs and Complexity
● Predictions of maintainability costs can be made by
assessing the complexity of system components.
● Most maintenance effort is spent on a relatively small
number of system components.
● Complexity depends on
– Complexity of control structures;
– Complexity of data structures;
– Object, method (procedure) and module size
– Dependencies


Lehman’s Laws for
Software Evolution
● Lehman observed that software evolves
● Law of Continuing Change: software needs to
change in order to stay usefull
● Law of Increasing Complexity: the structure of a
program deteriorates as it evolves
– the structure of a program degrades until it becomes
more cost effective to rewrite it


Why Systems Survive
● Organisations have huge investments in their
software systems
● Systems are critical business assets
● Organizations depend on the system
● Organizations know how to use their systems
● (Re) building systems is high risk


Business versus IT in
Software Maintenance
• Low costs • Need for adequate resources

• Opportunistic / flexible • Requires planning / choices
• Quick decision making • Hard to predict costs, impact

• Time to deliver quality
• Reliability in short time
• Business should understand IT
• IT should understand business
• Want something new
• Protect initial investment
• Why all these procedures?
• Problems with IT systems
make companies careful


Major problems in
Software Maintenance
● Inadequate testing methods
● Performance measurement difficulties
● Knowledge management / documentation
● Adapting to the rapidly changing business
environment
● Large backlog


Major problems in
Software Management
● Lack of skilled staff
● Lack of managerial understanding and support
● Lack of maintenance methodology, standards,
procedures & tools
● Program code is complex and unstructured
● Integration of overlapping/incompatible systems


Solutions
● Architecture
● Refactoring
● Automated regression testing
● Knowledge management
● Automated code inspection
● Better organization -> ITIL / CMM


What About
Software Renovation?
● Legacy system:
– (information) system that defeats further maintenance,
adjustment or renewal due to its size and age
– requires increasing maintenance costs
● System renovation:
– understanding and improvement of legacy systems
– by means of reverse engineering, program
understanding, design recovery, transformation, ...


Forward Engineering

Goals
Goals

Requirements
Requirements

Specification
Specification

Implementation
Implementation


Reverse Engineering
Legacy system Renovated system

Goals
Goals Goals
Goals

Requirements
Requirements Requirements
Requirements

Specification
Specification Specification
Specification

Implementation
Implementation Implementation
Implementation


A Typical Legacy System

~ 1-100 MLOC

● Different implementation languages
● Job Control Language scripts serve as glue
● Part of programs/databases are obsolete
● Some source text lost or incomplete; version
unknown
● Documentation is incomplete or obsolete

Typical Renovation Questions
● What is the architecture of this system
● Can we improve its structure?
● Can we generate documentation for it?
● Can we migrate it from COBOL 74 to
COBOL85?
● Can we connect it to Internet?
● Can we migrate it to a client/server architecture?


Synergy between Renovated and New
Components

Legacy code New business requirements
New business requirements
Legacy code

Extracted New
components components


Renovation =
Analysis + Transformation
Human
Legacy code
Legacy code insight +
tools

Analysis
Analysis
Documentation, object model, types,
metrics, visualization, components, ...

Transformation
Transformation Transformation Renovated system
rules Transformation Renovated system
rules

Software Renovation
● Analysis (partly supported by tools):
– architecture recovery
– system understanding
● Transformation (mostly supported by tools):
– systematic repairs
– code improvement/dialect conversion/translation
– architecture improvement/change


Software Renovation: Analysis
● Extraction of procedure calls and call graph
● Database usage between programs
● Dataflow analysis (at program and system level)
● Type analysis
● Cluster and concept analysis
● Metrics
● Visualization


Software Renovation: analysis
Legacy code
Legacy code
Elementary
facts Extract Enriched by
semantic
queries
Facts
Facts

Abstract

View


The Analysis Funnel

Legacy code
Legacy code

Volume inhibits
detailed analysis Lexical analysis
Lexical analysis
of all code
Syntactic analysis
Syntactic analysis

Semantic analysis
Semantic analysis

Facts
Facts

Example: DocGen
● Given the sources of a legacy system, web-based
documentation is generated containing
– overall architecture
– module dependencies & internal structure modules
– database usage
– simple metrics
● Fact: code reading finds two times more defects
than testing
● Commercialized by Software Improvement
Group

Software Renovation: Transformation

Legacy code
Legacy code

Facts
Facts

Transformation
Transformation Transformation
rules Transformation Renovated system
Renovated system
rules


Typical Transformations
● Year 2000
● Euro
● Extending bank account numbers to 10 digits
● Goto elimination
● OO restructuring
● Dialect translation (Cobol 74 → Cobol 85)
● Language conversion (Cobol → Java, VB →
Java, ...)

Observations
● Most legacy systems are multi-lingual
● A generic approach is needed to describe all
forms of analysis and transformations for all
required languages
● Languages like COBOL and PL/I are big:
– getting the right grammar is difficult
– many parsing techniques break down
● Meta-Programming can help


N
e
e
d get
e
o to
w ammi ng
d H gr
-p ro or
Tm eta ols f s ing?
o to roces
o
agep
l
l an gu
s Introduction to Software Evolution 39

Meta-Programming in Rascal

Software Analysis
Software Transformation
DSL Design & Implementation


Some Problems We Address
● How to parse source code/data files/models
● How to extract facts from them
● How to perform computations on these facts
● How to generate new source code (trafo, refactor,
compile)
● How to synthesize other information

EASY: Extract-Analyze-SYnthesize Paradigm
EASY: Extract-Analyze-SYnthesize Paradigm


System

?
?
Under
Investigation
(SUI)
EASY
Paradigm
Extract
Extract

Internal Representation Analyze
Analyze
Internal Representation

Synthesize
Synthesize

Results
Results

Here comes Rascal to the Rescue
● Lab infrastructure
● Test bed
● Social infrastructure
● Transfer medium
– Academia
– Industry (IBM, Eclipse)


Rascal Highlights
● Sophisticated built-in ● Concrete syntax
data types (immutable) ● Visiting/traversal
● Static safety ● Comprehensions
● Local type inference ● Higher-order
● Pattern Matching ● Familiar syntax
● Pattern-directed ● Java and Eclipse
invocation integration
● Syntax definitions and ● Read-Eval-Print
parsing (REPL)

The Rascal Standard Library
● Benchmark/test ● Exception
● Boolean, Number, ● IO (text, HTML, XML,
String, Node, Location, ATERM, RSF, AUC, ...)
DateTime ● JDT (Eclipse only)
● List, Set, Tuple, ● Resource (Eclipse only)
Relation, Map
● Subversion/CVS
● (Labelled) Graph,
PriorityQueue ● Figures (declarative
graphics)


Applications
● Several refactorings ● Analysis/transformation
of service composition
● Typechecking Rascal automata
● Analysis SVN ● Assertion checking
repositories JML
● Perf. Analysis Hibernate ● Various DSLs
code (KPMG)
– Auditing (PWC)
● Generic metrics – Forensics (NFI)
● ECORE models ● UvA Master's courses
● Magnolia compiler ● OU course

ile
nc es in f
o ccurre
View


Main Research Topics
Methods Methods for Methods for software
Specific DSLs
for DSL design software maintenance improvement
Applications

Constraint IDE
Parsing Matching Rewriting Traversing solving Visualisation support
Rascal
Technology

Grammar theory
Language Compilation Software
and
semantics techniques Engineering
Engineering

Foundations

Take Home Messages


Take Home Messages
● Software evolves in order to stay usefull
● Maintenance enables this evolution
● Software renovation is needed to extend the life
cycle of a system
● Rascal is a meta-programming language for
– software analysis & transformation
– DSL design and implementation


www.studeren.uva.nl/master-software-engineering-nl

www.cwi.nl/sen1 www.rascal-mpl.org

Introduction to Software Evolution: The Software Volcano

More Related Content

What's hot (20)

Similar to Introduction to Software Evolution: The Software Volcano (20)

More from Devnology (20)

Recently uploaded (20)

Introduction to Software Evolution: The Software Volcano