The Metrology Journey towards an 8th Base Quantity for Software: How Far or How Close Are We?

Metrology Journey towards an
8th Base Quantity for Software:
How Far or How Close
Are We?
Alain Abran
École de Technologie Supérieure – Université du Québec (Canada)
alain.abran@etsmtl.ca
1© Copyrights Abran 2016

Alain Abran
20 years 20 years
 Development
 Maintenance
 Process Improvement
ISO: 19761,
9216, 25000,
15939, 14143,
19759
© Copyrights Abran 2016
+ 35 PhD

Software is different!
• It is an intellectual product:
 It cannot be measured like physical
objects!
• Let us revisit this issue….
© Copyrights Abran 2016 3

List of topics
1. Measurement Lessons from the Masters
2. Metrology: Concepts & Terminology issues
3. Software Measurements:
 Diversity & Immaturity
 Key Advances in Functional Size Measurement
4. Software Measurement & Metrology: COSMIC –
ISO 19761
5. The future…..?

Measurement
of Time
Evolution of Time Measurement:
Concepts & Tools
Time Museum in Besancon (France)

The public sun dial
Sun shadow
The communal local time

Communal Time : The “horloge”
XVIII Century

Mechanical
Personal Time: Watches
Quartz
Atomic

Measurement
of distance
Geneva Museum of Science &
Technology:
Lack of universally
accepted references
(1802)

Architect KhaAmenhotec II
Dynasty XVIII-XX
(1550-1070 B.C.)
Egyptian Measurement &
Tools
Egyptian Museum – Torino (Italy)

Architect Kha
Measurement
Tools
© Fondazione Museo Antichità Egizie di Tonino – used with permission
Golden étalon
Work étalon

Masters from
the Past
The meter - universal étalon:
 A product of the French Revolution!
• A consensual definition: A fraction of the Meridian at the Equator
• A practical étalon: it took 7 years to measure the Meridian at the Equator..!
 And since then, the physical representation of the meter as an
étalon has changed a few times....

Key Lessons
from
the Masters
1. Evolutionary societal & consensual understanding of
measurement concepts
– Perfection is not expected first: the search for precision –
when necessary only!
2. Development of Measuring Instruments
3. Establishment of measurement ‘etalons’
4. Specialized measurement training & certification:
– Land surveyors
– Accountants,
– Engineers,
– Testing labs,
– etc.

List of topics
ISO 19761
5. The future…..?

Counting Measuring Quantitative models
of relationships
Quantifying?

Counting Measuring Quantitative models
of relationships
Quantification
& Metrology

Metrology
Key Metrology Concepts embedded within the International Vocabulary
of Metrology – VIM (& GUM)
Widely adopted by Masters in measurements:
 International Bureau of Weights & Measures,
 International Electrotechnical Commission,
 International Federation of Clinical Chemistry & Laboratory
Medicine,
 International Organization for Standardization (ISO),
 International Union of Pure & Applied Chemistry,
 International Union of Pure & Applied Physics,
 International Organization of Legal Metrology,
 International Laboratory Accreditation Cooperation
• + long list of organizations in Sciences & Engineering

Metrology
Metrology components (& articles for the law):
1. Definitions
2. National metrology
3. Traceability & uncertainties
4. Legal units of measurement
5. Transparency of metrological information.
6. Legal metrology
7. Application of the Law
8. Offences
9. Responsibilities & duties
10. Conformity assessment procedures

Criteria for a good design of measurement -
already defined in metrology:
1- components of a measurement system
2- quality criteria

Ref.: Sellami PhD thesis & Abran (2010)
Input quantity in a
measurement model
Output quantity in a
measurement model
Measurement
Function
Measurement Model
Measured
quantity
value
Measurement Procedure
Measurand
Operator
Measurement
Principle
Measurement
Method
Influence
Quantity
Measurement
Result

Table 3: Description of the measurement elements
Measurement
Measurement
result
Measurement
procedure
Measuring
device
Measurement
conditions
Measurement
error
Measured
quantity value
True quantity
value
Conventional
quantity value
Reference
measurement
procedure
Primary
reference
measurement
procedure
Measuring
system
Repeatability
condition of
measurement
Intermediate precision
condition of
measurement
Reproductibity
condition of
measurement
Systematic
measurement error
Measurement bias
Random
measurement error
Ref.: Sellami PhD thesis & Abran (2010)

Measurement
Precision
Measurement Uncertainty Calibration Metrological traceability
Measurement accuracy
Measurement trueness
Measurement
repeatability
Intermediate
measurement precision
Measurement
Reproducibility
Definitional uncertainty
Type A evaluation of measurement uncertainty
Type B evaluation of measurement uncertainty
Standard measurement uncertainty
Combined standard measurement uncertainty
Relative standard measurement uncertainty
Uncertainty budget
Target measurement uncertainty
Expanded measurement uncertainty
Coverage interval
Coverage probability
Coverage factor
Calibration
hierarchy
Verification
Validation
Correction
Metrological traceability chain
Metrological traceability to a
measurement unit
Metrological comparability of
measurement results
Metrological compatibilityof
measurement results
Table 6:
Detailed quality criteria
for a measurement
result

Information Needs
Information Product
Interpretation
Indicator
Derived MeasuresDerived Measures
Base MeasuresBase Measures
Measurable
Concept
Entity
Measurement Method
Attribute
Measurement Method
Attribute
(Analysis) Model
Measurement Function
Measurement
Information
Model
in
ISO 15939

Information Needs Information
Product
Interpretati
on
Indicator
Derived
Measures
Derived
Measures
Base
Measures
Base
Measures
Measur
able
Concep
t
Entity
Measurement
Method
Attribute
Measurement
Method
Attribute
(Analysis)
Model
Measurement
Function

VIM
Or
IEEE 24765?

Combination of units:
speed= Miles/hour
VIM
or
IEEE 24765

IEEE 24765

≠

List of topics
ISO 19761
5. The future…..?

Measurement
in
Software
Typical new ‘software metrics’:
– A modified algorithm
– Additions of conditions to the algorithms
– As many ‘metrics’ as can be extracted
automatically from files (codes, models,..)
• Typical empirical analyses:
– No hypothesis to be tested!
– Random search: highest correlation with
something else!
– Little verification of the relevance to the concept
to be measured!

Measurement
& Industry
For industry: measurement has little to do with
maths!
• Measurement as a technology requires
considerable consensual knowledge on:
– the concepts to be measured,
– credible references for measurement
– expected measurement errors
– …….. expected costs

Software
Industry &
Measurement
Out of the +1,000 of ‘software metrics’ proposed in the
literature, which ones have reached some maturity
with respect to industry needs?
– How do you recognize maturity in measurement?

Software
Industry &
Measurement
Out of the +1,000 of ‘software metrics’ proposed in the
literature, which ones have reached some maturity
with respect to industry needs?
– How do you recognize maturity in measurement?
Measurement Maturity = Standardization
• Which of the +1,000 ‘software metrics’ are now
recognized as ISO standards?

Functional Size: an
exception in
software
measurement
Only Functional Size Measurement (FSM) methods have
been adopted as ISO standards!
 What has been done differently?

Function Points:
Measurement
Infrastructure
User Groups built the essential metrology
support infrastructure:
– Procedural Measurement Manuals
– Central technical authority:
• Measurement Practice Committee
– Certification criteria & exams
– Guidelines for specific contexts
– Case studies for reference materials

37
Meta-standard
on Function
Points:
ISO 14143
14143-1
Definition of
concepts
14143-2
Conformity
Evaluation with
14143-1
14143-3
Verification
methods
14143-4
Reference Models
14143-5
Determination of
functional domains
14143-6
Selection
Guidelines

ISO
Function Points
Standards
1. ISO 20926 : IFPUG
2. ISO 24570 : NESMA
3. ISO 20968 : MRKII
4. ISO 29881 : FISMA
5. ISO 19761 : COSMIC
What is common/different in their designs?

1st Generation
Weaknesses
Weights-like Function Points:
• A structure with weights to integrate multi
variables:
 An end number with a symbol (FP) but without a well
defined measurement unit!
• A number of structural weaknesses pointed out in
the literature over the past 35 years:
 From nominal-ordinal-interval-ratio scale types
 Partially discredited in academia for almost 20 years.
 Automation in a dead end after +35 years!

List of topics
 Myths & Facts
 Advances in Functional Size Measurement
ISO 19761
5. The future…..?

2nd Generation
criteria
1- Adoption of consensual concepts from metrology :
 measurement method,
 measurement procedure,
 measurement unit, etc.
 many of the ISO definitions explicitly quoted ‘as is’ in COSMIC
glossary.
2- Designed to entirely meet the ISO 14143-1 meta-requirements:
 Measurement of FUR
 No reference to technical & quality characteristics
 No reference to effort
 Designed to be application domain independent
3- Open access

What is different in all software?
What is similar in all software?
COSMIC Key
Concepts

Footer Sample5/8/2016
Persistent
Storage
Software
being
measured
Boundary
Entries
Exits
Reads Writes
Functional Users
• Hardware devices,
• Other software or
• Humans
COSMIC
Generic Model of
Software

Functional
processes
Functional User
Requirements
Functional
Processes
Data
Manipulation
Data
Movement
Sub-processes
Theory:
Functional User
Requirements
Functional
Processes
Data Movements
(account for
associated data
manipulation)
1
1
n
2 - n
In practice:

Triggering
Event
causesa
Boundary
that is
moved into
a FP by the
FP’s
Triggering
Entry
Functional
Process
(FP)
Functional
User
togeneratea
Data
Group
COSMIC
Measurement
Process

Example of
Software Layers
465/8/2016
Keyboard
VDU
Screen
Printer
Hard Disk
Drive
Central
Processor
Keyboard
Driver
Screen
Driver
Print
Driver
Disk
Driver
Operating System Layer
Application Layer App 1 App 2 App ‘n’
Middleware Layer (Utilities, etc.)
Database Management
System Layer
DBMS 1 DBMS 2
Software
Layers

2nd Generation
COSMIC
Correction of all known structural weaknesses:
 No weight & no invalid mathematical operations
 Adoption of a clear & unique ‘measurement unit’:
– ‘a data movement of a single data group’ to which a size unit of
1 is assigned
• together with the 1CFP as its measurement symbol.
– 1 COSMIC CFP: represents a single measurable concept,
size of a FUR, making it a true ‘base quantity’.

 Guideline describing a
range of Approximate
Sizing methods
 Guideline on Assuring
the accuracy of
COSMIC measurements
 Guideline describing
Measurement Strategy
Patterns – to ensure
comparability
The COSMIC Functional Size Measurement Method
Version 4.0.1
GGuuiiddeelliinnee ffoorr EEaarrllyy oorr RRaappiidd
CCOOSSMMIICC FFuunnccttiioonnaall SSiizzee
MMeeaassuurreemmeenntt
bbyy uussiinngg aapppprrooxxiimmaattiioonn aapppprrooaacchheess
July 2015
The COSMIC Functional Size Measurement Method
Version 3.0.1
GGuuiiddeelliinnee ffoorr aassssuurriinngg tthhee aaccccuurraaccyy ooff
mmeeaassuurreemmeennttss
VERSION 0.93
February 2011
COSMIC

• Business applications
• Real-time software
• Data Warehouse software
• SOA software
• Mobile apps (in devt.)
• Agile Developments
www.cosmic-sizing.org
TThhee CCOOSSMMIICC FFuunnccttiioonnaall SSiizzee MMeeaassuurreemmeenntt MMeetthhoodd
VVeerrssiioonn 44..00..11
GGuuiiddeelliinnee ffoorr SSiizziinngg
BBuussiinneessss AApppplliiccaattiioonn SSooffttwwaarree
VERSION 1.3a
Febuary 2016
COSMIC:
Multi-domains

Physical Reality ………& ………… Models
COSMIC

GOAL/OBJECTIVE DRIVER INDICATOR COMMENTS/EFFECTS
FINANCIAL(F)
AssetManagement Existingasset
utilisation
 TotalAssets(FSAV)/#employees
($)
 FSAV–FSunits AssetValue
 PS–PortfolioSize
Revenue&Profitability Revenue
Growth
 Revenues/FSAV(%)
 Revenuesfromnewcustomers/
TotalRevenues(%)
 Newcustomers
acquiredusingFSMas
acontractualcondition
formeasuringthe
project–Derived
(Improveproject
governance)
Profitability  Profits/FSAV(%)
FinancialManagement Organisationa
lInvestments
 InvestmentsinIT
Project
Investments
 PCFS–ProjectCostperFSunit
 ECFS–EnterpriseCostperFSunit
 AMCFS – Application
MaintenanceCostperFSunit
Table1–FSM-basedmeasures:FinancialPerspective
CUSTOMER(C)
Customerpartnership
andinvolvement
Collaboration  %projectsusingintegratedteams
 SR–StabilityRatio
Customersatisfaction SLA  %SLAmet  iftheagreementuses
FSMasabasisforthe
contract
BusinessProcess
Support
Innovation
usage
 %ITsolutionssupportingprocess
improvementprojects
 projectmeasurement
usingFSM
Requirements
Management
 RequirementTurnoverIndex
[MELI01]
RTI=[(j CRFSj)/FinalFSunits]*100
 CRFS=ChangeRequest
FunctionSizeunits
 Showingthelevelof
turbulenceinrequisites
duringthedevelopment
phase
Problem
Management
 DR–DefectRatio
 AR–ApplicationReliability
BusinessGrowth MarketShare  %Marketshare  increasing%usingFSM
asaninitialcontract
condition
Table1–FSM-basedmeasures:CustomerPerspective
PROCESS(PR)
Application
Development&
Maintenance
Size  FSunit– FunctionalSizeunit, According to the FSM
method used, it can be
expressedforinstanceby:
 FP –FunctionPoints
 Cfsu - COSMIC
functionalsizeunits –
 PS–PortfolioSize
Effort  WE–WorkEffort
Productivity  PDR–ProjectDeliveryRate
 EP–EnterpriseProductivity
Support  ASR– ApplicationSupportRate
 DDR–DurationDeliveryRate
 AMPL – Application
MaintenanceLoadperPerson
Defectability&
Test
 RCR–Repair CostRatio
 SR–StabilityRatio
 DR–DefectRatio
 TPR–Testing ProficiencyRatio
 MTTR–MeanTimeTo Repair
ratio
 AR – Application Reliability
 DER–DefectDetectionRatio
 #defects/100FSunit accordingto
useracceptance
Reuse  FR–Functional Reuse%
 TR–Technical Reuse%
Table1–FSM-basedmeasures:ProcessPerspective
COSMIC:
Balanced
Scorecard

Sellami: Verification from
a Metrology Perspective
Khelifi: COSMIC & Design
of Standard Etalons
Almakadmeh: COSMIC
Scaling & Approcimation
Abu Talib – COSMIC
for Early Quality Assessment
Soubra: COSMIC – Automation
& Verification Protocol
Sarayreh: NFR & COSMIC
Measurement
Marin & Trudel: FSM & Defects
Detection
Monsalve: COSMIC & Business
Process Models
COSMIC:
Metrology-related
R&D

Portability in
ISO 9126 &
25000 series
Figure 4 – Quality model for external and internal quality
external and
internal
quality
functionality reliability
maturity
fault tolerance
recoverability
usability
understandability
learnability
operability
attractiveness
efficiency
time behaviour
resource
utilisation
maintainability
analysability
changeability
stability
testability
portability
adaptability
installability
co-existence
replaceability
suitability
accuracy
interoperability
security
functionality
compliance
reliability
compliance
usability
compliance
efficiency
compliance
maintainability
compliance
portability
compliance

Figure 1: Mapping system-FR and system-NFR to software-FR
Non Functional
Requirements

European Cooperation for Space Standardization - ECSS
ECSS 40 series:
• Portability
• Reliability
• Performance
• Maintainability
• Security
• Interfaces
• Configuration
• Databases
• …
Non Functional
Requirements

Portability in
ECSS standards
Fig.3SoftwareComponentPortabilityfunctions:systemmodelingview
Table 8
Function types and functions for portability requirements that may be allocated
to software
Function
Type
Function
Type
Name
Portability Functions
1
Software
Component
Portability
 Independence of the Operating
System Function (IOSF)
 Independence of the Middleware
Function (IMF)
 Independence of the Programming
Language Virtual Machine Function
(IPLVMF)
 Independence of the Browser
Function (IBF)
2
Data
Component
Portability
 Independence of the Database
Function (IDF)
 Distributed Data Base Management
System Function (DDBMSF)
3
Hardware
Component
Portability
 Independence of the Client Function
(ICF)
 Independence of the Server Function
(ISF)
 Independence of the Storage
Function (ISTF)
 Independence of the Network
Function (INF)
4
Isolation of
System
Calls
Portability
 Isolation of Software System Calls
Function (ISSCF)

Fig. 4 Software Component Portability functions: COSMIC modeling view
Portability
SOA Example
Fig. 11 COSMIC-SOA reference architectural model of system portability
requirement
1- Software Component Portability 2- Data Component Portability
4- Isolation of System Calls
Portability
3- Hardware Component Portability
Intermediary
Service
Portability Functional Type 1
Boundary
Functional
Process or
Service
Direct Data Movements
(E) ENTRY (X) EXIT
Indirect Data Movements
(R) READ (W) WRITE
Persistent
Storage
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
Isolation of the Software
System Calls Service
(ISSCS)
Persistent
Storage
Independence of the Operating
System Service (IOSS)
Independence of the Middleware
Service (IMS)
Independence of the Programming
Language Virtual Machine Service
(IPLVMS)
Independence of the Browser Service (IBS)
I
N
T
E
R
M
E
D
I
A
R
Y
Service
(IS)
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
R
X
W
E
R
X
W
E
R
X
W
E
R
X
W
E
Persistent
Storage
Independence of the Client Service
(ICS)
Independence of the Server Service
(ISS)
Independence of the Storage
Service (ISTS)
Independence of the Network
Service (INS)
I
N
T
E
R
M
E
D
I
A
R
Y
Service
(IS)
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
R
X
W
E
R
X
W
E
R
X
W
E
R
X
W
E
E
X
X
E
E
X
X
E
Persistent
Storage
Independence of the Database
Service (IDS)
Distributed Database
Management System Service
(DDBMSS)
R
W
R
W
E
X
X
E
E
X
X
E
Independence of the
Operating System
Function (IOSF)
Independence of
Middleware
Function (IMF)
Independence of the
Programming Language Virtual
Machine Function (IPLVMF)
E
X
X
E
E
X
X
E
E
X
X
E
Independence of the Browser
Function (IBF)
E
X
X
E
Independence of the
Client Function (ICF)
Independence of Server
Function (ISF)
Function (ISTF)
Function (INF)
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
Function (IDF)
E
X
X
E
E
X
X
E
Management System Function
(DDBMSF)
Isolation of the Software System
Calls Function (ISSCF)
E
X
X
E
IS1IS2
IS2IS3
IS3IS4
IS4IS5
IS5IS6
IS6IS7
IS7IS8
IS8IS9 IS9IS10
IS10IS11

Fig. 7 COSMIC reference framework of system portability requirements that
may be allocated to software1-Software Component Portability
2- Data Component Portability
3- Hardware Component Portability
4- Isolation of System Calls Portability
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
Isolation of Software System Calls
Function
(ISSCF)
Persistent
Storage
Independence of the Operating
System Function (IOSF)
Independence of the Middleware
Function (IMF)
Independence of the Programming
Language Virtual Machine Function
(IPLVMF)
Independence of the Browser Function
(IBF)
I
N
T
E
R
M
E
D
I
A
R
Y
Service
(IS)
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
R
X
W
E
R
X
W
E
R
X
W
E
R
X
W
E
Persistent
Storage
Independence of the Client Function
(ICF)
Independence of the Server Function
(ISF)
Function (ISTF)
Function (INF)
I
N
T
E
R
M
E
D
I
A
R
Y
Service
(IS)
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
E
X
X
E
R
X
W
E
R
X
W
E
R
X
W
E
R
X
W
E
E
X
X
E
E
X
X
E
Persistent
Storage
Function (IDF)
Management System
Function (DDBMSF)
R
W
R
W
E
X
X
E
E
X
X
E
Intermediary
Service
Boundary
Functional
Process
Direct Data Movements
(E) ENTRY (X) EXIT
Indirect Data Movements
(R) READ (W) WRITE
Persistent
Storage
E
X
X
E
E
X
X
E
Table 9
COSMIC-SOA measurement example for the interactions between a functional
process and its own service process
COSMIC-SOA Types
Data Movement
Description
Data
Movement
Type
Functional
Process
Service
Process
Independence
of the
Operating
System
Function
(IOSF)
Independenc
e of the
Operating
System
Service
(IOSS)
 IOSF sends a data group to
IOSS
X
 IOSS receives a data group
from IOSF
E
 IOSS sends a data group to
IOSF
X
 IOSF receives a data group
from IOSS
E
Functional Size (subtotal): 4 CFP
Table 10
COSMIC-SOA measurements of the interactions between the 11 functional
processes and their service processes
Fun
ID
COSMIC-SOA Types of exchange services for
System Portability NO. Data
MovementsFunctional Process Service Process
1 Independence of the
Operating System
Function (IOSF)
Independence of the
Operating System
Service (IOSS)
4
Middleware Function
(IMF)
Independence of the
Middleware Service
(IMS)
4
Programming Language
Virtual Machine
Function (IPLVMF)
Independence of the
Programming Language
Virtual Machine Service
(IPLVMS)
4
Browser Function (IBF)
Independence of the
Browser Service (IBS)
4
Database Function (IDF)
Independence of the
Database Service (IDS)
4
6 Distributed Database
Management System
Function (DDBMSF)
Distributed Data Base
Management System
Service (DDBMSS)
4
Client Function (ICF)
Independence of the
Client Service (ICS)
4
Server Function (ISF)
Independence of the
Server Service (ISS)
4
9
Independence of the
Storage Function (ISTF)
Independence of the
Storage Service (ISTS)
4
Network Function (INF)
Independence of the
Network Service (INS)
4
11 Isolation of the Software
System Calls Function
(ISSCF)
Isolation of the Software
System Calls Service
(ISSCS)
4
Functional Size: 44 CFP
Portability
SOA Example

List of topics
ISO 19761
5. The future…..?

An analogy...
2000 sf 4000 sf
Software
Functionality
Software
Functionality
120 CFP 240 CFP
CFP = COSMIC Function Point

- In the physical software itself
- In description of software functional requirements
- In business process modeling
- In engineering applications
- In information processing
- In information theory……………?
Universality
of
COSMIC
Concepts?

How do we measure in the physical world?
 with concepts & 7 base units
 + traceability to SI measurement ‘etalons’
What can we measure & how do we measure in the
software world?
 Are we getting close to an 8th base unit ?
 + traceability to an SI measurement ‘etalon’?

Measurement
as a
Technology
What does it take for software measurement to be
adopted as a new technology across an industry?
– Software measurement must already have been proven
to work well in a large variety of contexts:
• i.e. it must be mature as a technology, or maturing rapidly.
– Software measurement must become integrated into
the technological environment of the software industry.
– It must become integrated into the business context
(which includes its legal & regulatory aspects).

Conclusion
When will we get:
1. An International Base Unit
for software?
2. Our golden measurement
étalons for software design
& control?

Questions?
alain.abran@etsmtl.ca

The Metrology Journey towards an 8th Base Quantity for Software: How Far or How Close Are We?

More Related Content

What's hot (16)

Viewers also liked (8)

Similar to The Metrology Journey towards an 8th Base Quantity for Software: How Far or How Close Are We? (20)

More from COSMIC - Common Software Measurement International Consortium (18)

Recently uploaded (20)

The Metrology Journey towards an 8th Base Quantity for Software: How Far or How Close Are We?