Software Visualization - Promises & Perils

Software Visualization
Promises & Perils

Michele Lanza
REVEAL @ Faculty of Informatics
University of Lugano, Switzerland

The use of the crafts of typography,
graphic design, animation, and
cinematography with modern human-
computer interaction and computer
graphics technology to facilitate both the
human understanding and effective use of
computer software.

John Stasko et al.

Programming as a Multimedia Experience

MIT University Press, 1998

algorithm visualization

software visualization

program visualization

the visualization of
t h e h i g h e r- l e v e l
abstractions which
describe software



t h e h i g h e r- l e v e l
abstractions which
describe software

the actual program
program visualization code or data
structures in static or
dynamic form

static code/data
visualization


dynamic code/data
visualization

Software Visualization - Promises & Perils

what is
software?
Soft...

Software: The
intangible objects inside
the computer.
where?

Programming is a kind of writing.

Gerald M. Weinberg

Psychology of Computer
Programming

Dorset House, 1971

programming ~ writing

software ~ text

Deciduous shrub, glabrous or nearly so, with
weak, trailing sub-glaucous, often purple-
tinted stems, either decumbent and forming
low bushes 50 - 100 cm high, or climbing
over other shrubs, rarely more erect
and reaching 2 m. Prickles hooked,
all +- equal. L’ﬂets 2-3 pairs, 1-3.5
cm, ovate or ovate-elliptic, simply,
rarely double serrate, glabrous on
both sides or pubescent on the
veins (rarely all over) beneath,
rather thin; petiole usually
with some stalked glands;
stipules narrow, auricles
straight. Flowers 1-6, white
3-5 cm diam.; pedicels 2-4 cm
with stalked glands, rarely
smooth; buds short... etc. etc.

It would be a great mistake to suppose
that the dominant role of visual
observation has now been overthrown by
more ‘analytical’ methods.

John Ziman

Reliable Knowledge
An Exploration of the Grounds
for Belief in Science

Cambridge University Press, 1978

/***************************************************************************/ if(p<3&y==E)H=y^16; /* shift capt.sqr. H if e.p.*/
/* micro-Max, */ t=b[H];if(t&k|p<3&!(r&7)!=!t)break; /* capt. own, bad pawn mode */
/* A chess program smaller than 2KB (of non-blank source), by H.G. Muller */ i=99*w[t&7]; /* value of capt. piece t */
/***************************************************************************/ if(i<0||E-S&&b[E]&&y-E<2&E-y<2)m=I; /* K capt. or bad castling */
/* version 3.2 (2000 characters) features: */ if(m>=l)goto C; /* abort on fail high */
/* - recursive negamax search */ if(h=d-(y!=z)) /* remaining depth(-recapt.)*/
/* - quiescence search with recaptures */ {v=p<6?b[x+8]-b[y+8]:0; /* center positional pts. */
/* - recapture extensions */ b[G]=b[H]=b[x]=0;b[y]=u&31; /* do move, strip virgin-bit*/
/* - (internal) iterative deepening */ if(!(G&M)){b[F]=k+6;v+=30;} /* castling: put R & score */
/* - best-move-first 'sorting' */ if(p<3) /* pawns: */
/* - a hash table storing score and best move */ {v-=9*(((x-2)&M||b[x-2]!=u)+ /* structure, undefended */
/* - full FIDE rules (expt minor ptomotion) and move-legality checking */ ((x+2)&M||b[x+2]!=u)-1); /* squares plus bias */
if(y+r+1&S){b[y]|=7;i+=C;} /* promote p to Q, add score*/
#define F(I,S,N) for(I=S;I<N;I++) }
#define W(A) while(A) v=-D(24-k,-l-(l>e),m>q?-m:-q,-e-v-i, /* recursive eval. of reply */
#define K(A,B) *(int*)(T+A+(B&8)+S*(B&7)) J+J(0),Z+J(8)+G-S,F,y,h); /* J,Z: hash keys */
#define J(A) K(y+A,b[y])-K(x+A,u)-K(H+A,t) v-=v>e; /* delayed-gain penalty */
if(z==9) /* called as move-legality */
#define U 16777224 {if(v!=-I&x==K&y==L) /* checker: if move found */
struct _ {int K,V;char X,Y,D;} A[U]; /* hash table, 16M+8 entries*/ {Q=-e-i;O=F;return l;} /* & not in check, signal */
v=m; /* (prevent fail-lows on */
int V=112,M=136,S=128,I=8e4,C=799,Q,N,i; /* V=0x70=rank mask, M=0x88 */ } /* K-capt. replies) */
b[G]=k+38;b[F]=b[y]=0;b[x]=u;b[H]=t; /* undo move,G can be dummy */
char O,K,L, if(Y&8){m=v;Y&=~8;goto A;} /* best=1st done,redo normal*/
w[]={0,1,1,3,-1,3,5,9}, /* relative piece values */ if(v>m){m=v;X=x;Y=y|S&G;} /* update max, mark with S */
o[]={-16,-15,-17,0,1,16,0,1,16,15,17,0,14,18,31,33,0, /* step-vector lists */ } /* if non castling */
7,-1,11,6,8,3,6, /* 1st dir. in o[] per piece*/ t+=p<5; /* fake capt. for nonsliding*/
6,3,5,7,4,5,3,6}, /* initial piece setup */ if(p<3&6*k+(y&V)==S /* pawn on 3rd/6th, or */
b[129], /* board: half of 16x8+dummy*/ ||(u&~24)==36&j==7&& /* virgin K moving sideways,*/
T[1035], /* hash translation table */ G&M&&b[G=(x|7)-(r>>1&7)]&32 /* 1st, virgin R in corner G*/
&&!(b[G^1]|b[G^2]) /* 2 empty sqrs. next to R */
n[]=".?+nkbrq?*?NKBRQ"; /* piece symbols on printout*/ ){F=y;t--;} /* unfake capt., enable e.p.*/
}W(!t); /* if not capt. continue ray*/
D(k,q,l,e,J,Z,E,z,n) /* recursive minimax search, k=moving side, n=depth*/ }}}W((x=x+9&~M)-B); /* next sqr. of board, wrap */
int k,q,l,e,J,Z,E,z,n; /* (q,l)=window, e=current eval. score, E=e.p. sqr.*/ C:if(m>I/4|m<-I/4)d=99; /* mate is indep. of depth */
{ /* e=score, z=prev.dest; J,Z=hashkeys; return score*/ m=m+I?m:-D(24-k,-I,I,0,J,Z,S,S,1)/2; /* best loses K: (stale)mate*/
int j,r,m,v,d,h,i=9,F,G; if(!a->K|(a->X&M)!=M|a->D<=d) /* if new/better type/depth:*/
char t,p,u,x,y,X,Y,H,B; {a->K=Z;a->V=m;a->D=d;A->K=0; /* store in hash,dummy stays*/
struct _*a=A; a->X=X|8*(m>q)|S*(m<l);a->Y=Y; /* empty, type (limit/exact)*/
/* lookup pos. in hash table*/ } /* encoded in X S,8 bits */
j=(k*E^J)&U-9; /* try 8 consec. locations */ /*if(z==8)printf("%2d ply, %9d searched, %6d by (%2x,%2x)n",d-1,N,m,X,Y&0x77);*/
W((h=A[++j].K)&&h-Z&&--i); /* first empty or match */ }
a+=i?j:0; /* dummy A[0] if miss & full*/ if(z&8){K=X;L=Y&~M;}
if(a->K) /* hit: pos. is in hash tab */ return m;
{d=a->D;v=a->V;X=a->X; /* examine stored data */ }
if(d>=n) /* if depth sufficient: */
{if(v>=l|X&S&&v<=q|X&8)return v; /* use if window compatible */ main()
d=n-1; /* or use as iter. start */ {
}X&=~M;Y=a->Y; /* with best-move hint */ int j,k=8,*p,c[9];
Y=d?Y:0; /* don't try best at d=0 */
}else d=X=Y=0; /* start iter., no best yet */ F(i,0,8)
N++; /* node count (for timing) */ {b[i]=(b[i+V]=o[i+24]+40)+8;b[i+16]=18;b[i+96]=9; /* initial board setup*/
W(d++<n|z==8&N<1e7&d<98) /* iterative deepening loop */ F(j,0,8)b[16*j+i+8]=(i-4)*(i-4)+(j-3.5)*(j-3.5); /* center-pts table */
{x=B=X; /* start scan at prev. best */ } /*(in unused half b[])*/
Y|=8&Y>>4; /* request try noncastl. 1st*/ F(i,M,1035)T[i]=random()>>9;
m=d>1?-I:e; /* unconsidered:static eval */
do{u=b[x]; /* scan board looking for */ W(1) /* play loop */
if(u&k) /* own piece (inefficient!)*/ {F(i,0,121)printf(" %c",i&8&&(i+=7)?10:n[b[i]&15]); /* print board */
{r=p=u&7; /* p = piece type (set r>0) */ p=c;W((*p++=getchar())>10); /* read input line */
j=o[p+16]; /* first step vector f.piece*/ N=0;
W(r=p>2&r<0?-r:-o[++j]) /* loop over directions o[] */ if(*c-10){K=c[0]-16*c[1]+C;L=c[2]-16*c[3]+C;}else /* parse entered move */
{A: /* resume normal after best */ D(k,-I,I,Q,1,1,O,8,0); /* or think up one */
y=x;F=G=S; /* (x,y)=move, (F,G)=castl.R*/ F(i,0,U)A[i].K=0; /* clear hash table */
do{H=y+=r; /* y traverses ray */ if(D(k,-I,I,Q,1,1,O,9,2)==I)k^=24; /* check legality & do*/
if(Y&8)H=y=Y&~M; /* sneak in prev. best move */ }
if(y&M)break; /* board edge hit */ }

The use of the crafts of typography,
graphic design, animation, and
cinematography with modern human-
computer interaction and computer
graphics technology to facilitate both the
human understanding and effective use of
computer software.

Lanza, 2010

(software)
visualization myths

a picture is worth
a thousand words

n g
a picture is worth
ro words
w
a thousand

g
UML is software
n
r o
w
visualization

visual programming is

n g
visual programming is
r o
w

static
visualization


dynamic
visualization

ng
ro ng
w static
visualization
ro
w

ng
o
dynamic

wr visualization

about software of
software software
visualization

about
software
software of
software
information
visualization
visualization

William Playfair
1759 - 1823

Jacques Bertin
Semiology of Graphics,1967

It was the twenty-eight of November. An
immense confused mass of men, horses,
vehicles besieged the narrow entrances to
the bridges and began to flow over them.

It was the twenty-eight of November. An
immense confused mass of men, horses,
vehicles besieged the narrow entrances to
the bridges and began to flow over them.

Those in front, pushed by the weight of those behind were
crushed, trampled on, or forced into the ice-filled water of the
Berezina.
The confusion was so great that when Napoleon himself wished
to cross, it was necessary to use force to clear a passage.
Some there were who, determined to pass at all costs, cut a
horrible way for themselves with their swords. Others opened
an even crueler road for their carriages, driving them pitilessly
through the helpless crowd, crushing men and women, in their
odious greed sacrificing their companions in misery.

We acquire more information
through vision than through
all the other senses combined.

Colin Ware

Information Visualization
Perception for Design

Morgan Kaufmann, 2004

Sensation Perception
(Physical Process) (Cognitive Process)

Stimulus Sensory Organ Perceptual Organ

Brain

The Eye
A camera with a maximum
resolution of 625 dpi in the foveal
zone of the retina.

The Eye
zone of the retina.
The retina is coated with rods (for
dim light) and cones (for bright
light and colors).

The Eye
zone of the retina.
The retina is coated with rods (for
dim light) and cones (for bright
light and colors).
Three types of cones, specialized
on the blue, green, and red range
of the color spectrum.

Breaking News

We do not see
with the eyes

Category Form Color Spatial Motion Motion

Orientation Hue 2D position Flicker

Line length Intensity Direction

Line width

Size
Attribute
Shape

Curvature

Added marks

Enclosure

Categorical vs Quantitative Encoding
of Pre-attentive Attributes

Type Attribute Quantitative?

Orientation No

Line Length Yes

Line Width Partially

Size Partially
Form
Shape No

Curvature No

Added Marks No

Enclosure No

Hue No
Color
Intensity Partially

Position 2D Position Yes

Stephen Few Edward Tufte
Show Me the Numbers The Visual Display of Quantitative Information

Analytics Press, 2004 Morgan Kaufmann, 2004

about software of
software
visualization software

Computer graphics ✔
Gestalt principles ✔

Computer graphics ✔
Gestalt principles ✔
Semantics ✘

Interlude II
Software
Analysis

visualization ~ interaction

interaction ~ tool

visualization ~ interaction

interaction ~ tool

tool ~ cool

X-Ray Eclipse Plug-in xray.inf.usi.ch

Initialize Interface Internal Accessor Attribute

Regular
Initialize Interface Internal Accessor Attribute

Overriding

Extending

Abstract

Constant

Delegating

Setter

Getter

OOPSLA 2001, TSE 2005

Regular

Overriding

Extending

Abstract

Constant

Delegating

Setter

Getter

the unbearable
lightness of being
visual

Solution 1: Anecdotal Evidence

City Metaphor

VISSOFT 2007

City Metaphor

class building
package district

VISSOFT 2007

City Metaphor

class building
package district

nesting level color

VISSOFT 2007

City Metaphor
number of methods (NOM) height

number of attributes (NOA) base size

number of lines of code (LOC) color

class building
package district

nesting level color

VISSOFT 2007

Program Comprehension
ArgoUML
LOC 136,325

ICPC 2007

FacadeMDRImpl
NOA 3
skyscraper
NOM 349
LOC 3,413

CPPParser
NOA 85
ofﬁce building
NOM 204
LOC 9,111

JavaTokenTypes
NOA 173
parking lot
NOM 0
LOC 0

house PropPanelEvent
NOA 2
NOM 3
LOC 37

Program Comprehension

ICPC 2007

Design Quality Assessment
disharmony map

ArgoUML classes
brain class 8
god class 30
god + brain 6
data class 17
unaffected 1,715
SoftVis 2008

System Evolution Analysis

time traveling

WCRE 2008

System Evolution Analysis

time
ArgoUML
time traveling 8 major releases
6 years

WCRE 2008

http://codecity.inf.usi.ch

implemented in Smalltalk

ICSE 2008 tool demo

technical report 2010

State of the art?

Design desiderata
1 Avoid comparing using a technique against not using it.
2 Involve participants from the industry.
3 Provide a not-so-short tutorial of the experimental tool to the participants.
4 Avoid, whenever possible, giving the tutorial right before the experiment.
5 Use the tutorial to cover both the research behind the approach and the tool.
6 Find a set of relevant tasks.
7 Choose real object systems that are relevant for the tasks.
8 Include more than one object system in the design.
9 Provide the same data to all participants.
10 Limit the amount of time allowed for solving each task.
11 Provide all the details needed to make the experiment replicable.
12 Report results on individual tasks.
13 Include tasks on which the expected result is not always to the advantage of the
tool being evaluated.
14 Take into account the possible wide range of experience level of the participants.

Finding a baseline

1. program comprehension

2. design quality assessment

3. system evolution analysis

Finding a baseline

1. program comprehension

2. design quality assessment

Tasks
program comprehension 6
A1 Identity the convention used in the system to organize unit tests.
A2.1& What is the spread of term T in the name of the classes, their attributes and
A2.2 methods?
A3 Evaluate the change impact of class C, in terms of intensity and dispersion.
A4.1 Find the three classes with the highest number of methods.
Find the three classes with the highest average number of lines of code per
A4.2
method.

Tasks
A2.2 methods?
A4.2
method.
B1.1 Identify the package with the highest percentage of god classes.
B1.2 Identify the god class with the largest number of methods.
Identify the dominant (affecting the highest number of classes) class-level
B2.1
design problem.
B2.2 Write an overview of the class-level design problems in the system.

design quality assessment 4

Tasks
5
A2.2 methods?
A4.2
method.
B2.1
design problem.

design quality assessment 4

Tasks
quantitative 9
8
A2.2 methods?
A4.2
method.
B2.1
design problem.

qualitative 1

Main research questions

1
Does the use of CodeCity increase the correctness
of the solutions to program comprehension tasks,
compared to non-visual exploration tools, regardless
of the object system size?

Main research questions

1
Does the use of CodeCity increase the correctness
of the solutions to program comprehension tasks,
compared to non-visual exploration tools, regardless
of the object system size?

2
Does the use of CodeCity reduce the time needed to
solve program comprehension tasks, compared to
non-visual exploration tools, regardless of the object
system size?

Variables of the experiment
correctness
dependent
completion time
CodeCity
tool Eclipse + Excel
independent medium
object system size large

beginner
experience level advanced
controlled academia
background industry

Variables of the experiment
correctness
dependent FindBugs
completion time 1,320 classes
93,310 LOC
CodeCity
tool Eclipse + Excel
independent medium
object system size large

beginner
experience level advanced Azureus
controlled academia 4,656 classes
background industry 454,387 LOC

The experiment’s design

between-subjects
randomized-block


between-subjects
randomized-block

CodeCity
T1 large

Tool
T2 medium
Size

Ecl+Excl
T3 large

T4 medium

background academia industry
experience beginner advanced beginner advanced

B1 B2 B3 B4
between-subjects
randomized-block

CodeCity
T1 large

Tool
T2 medium
Size

Ecl+Excl
T3 large

T4 medium

Experimental runs
day 1
time

training session
(1 hour)

Experimental runs
day 1
time

training session
(1 hour)

e1
experiment session
(2 hours)
c1

Experimental runs
day 1 day 2
time

training session
(1 hour)

e1 e2
experiment session
(2 hours)
c1 c2

Experimental runs
day 1 day 2 day 3
time

training session
(1 hour)

e1 e2 e3
experiment session
(2 hours)
c1 c2

Experimental runs
day 1 day 2 day 3 day 4
time

training session
(1 hour)

e1 e2 e3
experiment session
(2 hours)
c1 c2 c4

Testing the waters
2009
November December
18 24 25 2 9

Lugano
1 3 1
1 1 1 1

Timeline of the experiment
2009 2010
November December January February ... April
18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14

Lugano
1 3 1 1 1 1
1 1 1 1 3

Bologna
2 1 6
1 1 1 1

Antwerp
5
6

Bern
4 1
6

Timeline of the experiment
2009 2010
November December January February ... April
18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14

Lugano
1 3 1 1 1 1
1 1 1 1 3
remote
sessions
Bologna
2 1 6
1 1 1 1

Antwerp
5
6
remote
session
Bern
4 1
6

Treatments and subjects
academia industry
beginner advanced advanced

large 2 2 6 10
CodeCity
medium 3 2 7 12
large 2 3 3 8
Ecl+Excl
medium 2 5 4 11
9 12 20 41

Collecting raw data

solution

Collecting raw data
completion time

Controlling time

common time

Controlling time

common time

info on subjects
Name (Task): Remaining time

Assessing correctness
1 T2: Findbugs, analyzed with CodeCity A3: Impact Analysis B1.2

A1 Multiple locations. The god class containing the largest number of methods in the system is
There are 40/41 [0.5pts] classes class MainFrame [0.8pts]
Dispersed. [1pt] defined in the following 3 packages [1/6pts for each]: defined in package edu.umd.cs.findbugs.gui2 [0.1pts]
which contains 119 [0.1pts] methods.
• edu.umd.cs.findbugs
A2.1
• edu.umd.cs.findbugs.bcel B2.1
Localized [0.5pts]
in package edu.umd.cs.findbugs.detect [0.5pts]. • edu.umd.cs.findbugs.detect The dominant class-level design problem is
DataClass [0.5pts]
A2.2 A4.1 which affects a number of 67 [0.5pts] classes.

Dispersed The 3 classes with the highest number of methods are [ 1 pts each correctly placed and 1 pts each misplaced]:
3 6
in the following (max. 5) packages [0.2pts for each]: 1. class AbstractFrameModelingVisitor
• edu.umd.cs.findbugs defined in package edu.umd.cs.findbugs.ba
contains 195 methods;
• edu.umd.cs.findbugs.anttask
2. class MainFrame
• edu.umd.cs.findbugs.ba defined in package edu.umd.cs.findbugs.gui2
• edu.umd.cs.findbugs.ba.deref
3. class BugInstance
• edu.umd.cs.findbugs.ba.jsr305 defined in package edu.umd.cs.findbugs
• edu.umd.cs.findbugs.ba.npe contains 118 methods
or
• edu.umd.cs.findbugs.ba.vna class TypeFrameModelingVisitor
defined in package edu.umd.cs.findbugs.ba.type
• edu.umd.cs.findbugs.bcel contains 118 methods.
• edu.umd.cs.findbugs.classfile
A4.2
• edu.umd.cs.findbugs.classfile.analysis
The 3 classes with the highest average number of lines of code per method are [ 1 pts each correctly placed and 1 pts each
• edu.umd.cs.findbugs.classfile.engine 3 6
misplaced]:
• edu.umd.cs.findbugs.classfile.impl
1. class DefaultNullnessAnnotations
• edu.umd.cs.findbugs.cloud defined in package edu.umd.cs.findbugs.ba
has an average of 124 lines of code per method;
• edu.umd.cs.findbugs.cloud.db
2. class DBCloud.PopulateBugs
• edu.umd.cs.findbugs.detect defined in package edu.umd.cs.findbugs.cloud.db
has an average of 114.5 lines of code per method;
• edu.umd.cs.findbugs.gui
3. class BytecodeScanner
• edu.umd.cs.findbugs.gui2
defined in package edu.umd.cs.findbugs.ba
• edu.umd.cs.findbugs.jaif has an average of 80.75 lines of code per method.

• edu.umd.cs.findbugs.model B1.1
• edu.umd.cs.findbugs.visitclass

oracles
The package with the highest percentage of god classes in the system is
• edu.umd.cs.findbugs.workflow edu.umd.cs.findbugs.ba.deref [0.8pts]
which contains 1 [0.1pts] god classes
out of a total of 3 [0.1pts] classes.

Assessing correctness
blinding 1 T2: Findbugs, analyzed with CodeCity A3: Impact Analysis B1.2

A1 Multiple locations. The god class containing the largest number of methods in the system is
There are 40/41 [0.5pts] classes class MainFrame [0.8pts]
Dispersed. [1pt] defined in the following 3 packages [1/6pts for each]: defined in package edu.umd.cs.findbugs.gui2 [0.1pts]
which contains 119 [0.1pts] methods.
• edu.umd.cs.findbugs
A2.1
• edu.umd.cs.findbugs.bcel B2.1
Localized [0.5pts]
in package edu.umd.cs.findbugs.detect [0.5pts]. • edu.umd.cs.findbugs.detect The dominant class-level design problem is
DataClass [0.5pts]
A2.2 A4.1 which affects a number of 67 [0.5pts] classes.

Dispersed The 3 classes with the highest number of methods are [ 1 pts each correctly placed and 1 pts each misplaced]:
3 6
in the following (max. 5) packages [0.2pts for each]: 1. class AbstractFrameModelingVisitor
• edu.umd.cs.findbugs defined in package edu.umd.cs.findbugs.ba
• edu.umd.cs.findbugs.anttask
2. class MainFrame
• edu.umd.cs.findbugs.ba defined in package edu.umd.cs.findbugs.gui2
• edu.umd.cs.findbugs.ba.deref
3. class BugInstance
• edu.umd.cs.findbugs.ba.jsr305 defined in package edu.umd.cs.findbugs
• edu.umd.cs.findbugs.ba.npe contains 118 methods
or
• edu.umd.cs.findbugs.ba.vna class TypeFrameModelingVisitor
defined in package edu.umd.cs.findbugs.ba.type
• edu.umd.cs.findbugs.bcel contains 118 methods.
• edu.umd.cs.findbugs.classfile
A4.2
• edu.umd.cs.findbugs.classfile.analysis
The 3 classes with the highest average number of lines of code per method are [ 1 pts each correctly placed and 1 pts each
• edu.umd.cs.findbugs.classfile.engine 3 6
misplaced]:
• edu.umd.cs.findbugs.classfile.impl
1. class DefaultNullnessAnnotations
• edu.umd.cs.findbugs.cloud defined in package edu.umd.cs.findbugs.ba
has an average of 124 lines of code per method;
• edu.umd.cs.findbugs.cloud.db
2. class DBCloud.PopulateBugs
• edu.umd.cs.findbugs.detect defined in package edu.umd.cs.findbugs.cloud.db
has an average of 114.5 lines of code per method;
• edu.umd.cs.findbugs.gui
3. class BytecodeScanner
• edu.umd.cs.findbugs.gui2
defined in package edu.umd.cs.findbugs.ba
• edu.umd.cs.findbugs.jaif has an average of 80.75 lines of code per method.

• edu.umd.cs.findbugs.model B1.1
• edu.umd.cs.findbugs.visitclass

oracles
The package with the highest percentage of god classes in the system is
• edu.umd.cs.findbugs.workflow edu.umd.cs.findbugs.ba.deref [0.8pts]
which contains 1 [0.1pts] god classes
out of a total of 3 [0.1pts] classes.

Statistical test

two-way analysis of variance (ANOVA)
95% conﬁdence interval

Correctness
Ecl+Excl
CodeCity

Does the use of CodeCity increase the 8
correctness of the solutions to program
comprehension tasks, compared to non- 7
visual exploration tools, regardless of the 6
object system size?
5

4

3

2

1

0
medium large

Correctness
Ecl+Excl
CodeCity

visual exploration tools, regardless of the 6
object system size?
5

24.26%
4

3

2
more correct with CodeCity
1
large effect size (d=0.89)
0
medium large

Correctness
Ecl+Excl
CodeCity

visual exploration tools, regardless of the
object system size?
6

24.26%
5

4

more correct with CodeCity 3
large effect size (d=0.89)
2
medium large

Completion time
Ecl+Excl
CodeCity

Does the use of CodeCity reduce the time 60
needed to solve program comprehension
tasks, compared to non-visual exploration 50
tools, regardless of the object system size?
40

30

20

10

0
medium large

Completion time
Ecl+Excl
CodeCity

tasks, compared to non-visual exploration 50
40

12.01%
30

20

less time with CodeCity 10
moderate effect size (d=0.63)
0
medium large

Completion time
Ecl+Excl
CodeCity

tasks, compared to non-visual exploration
50

12.01%
40

30
less time with CodeCity
moderate effect size (d=0.63)
20
medium large

after the ﬁrst round

CodeCity
vs

Ecl+Excl
+24% correctness
-12% completion time

193
a
riteri nce
ing C ie
Block Exper d

Replicability
nd nce
grou adva
Back nced
ent size
indust
ry adva
Tr eatm S ystem nced
indust
ry adva
Tool large nced
b er ity in dustry adva
Code Num CodeC y large ustry begin
ner
1 it ind
CodeC y large y ginner
IA01 1 it
large academ y be
nced
CodeC y adva
IA02 1
odeC
it
la rge academ nced
C ry adva
IA03 1 ity
large indust y nced
CodeC y adva
4 1 it adem
IA0 CodeC y large ac y nced
1 adem adva
AB01 CodeC y
it
large ac
vance
d
1 indu stry ad
AB02 CodeC y
it
large ry vance
d
IA05 1
CodeC y
it
large indust ad
nced
A01 1 eCit ium indust
ry adva
A Cod med ner
02 1 ity ium in dustry begin d
AA CodeC y med y nce
2 m academ adva
IA06 it

re
CodeC y mediu nced

su
indust
ry adva

res
IA07 2 it m
CodeC y mediu nced

fair amo unt
eP

too amo unt
indust
ry adva

very muc unt
fair amo
2 m

Tim
IA08 it
odeC mediu begin d
ner

fair litt h
C ry

very so m unt
fair
indust y

no amo le
eCity

litt uch
IA09 2 edium
Cod m nce

no amo uch
adva

fair so m le
ity academ y

very so m unt
03 2 ium

iate

t so ttle h
AB CodeC y med ner

uc
begin d

t

litt uch
adem

ed
IA10 2 it m

inte ficult le

t
mediu ac

rm
.2
CodeC y

no
nce

sim ple diate

no amo uch
m
dif ossib

li

fair so m le
y

B2
adva

t
inte

t
academ y

un
IA11 2 it m
mediu nced

sim rme
CodeC y

p

very
no
inte cult iate

fair so m uch
im
adva

no so m le
academ

no amo uch
t
2 m

triv ple

litt
it

no
IA12

dif rme diate
mediu

d
nced

t
very e
CodeC y

un
dif rme

inte cult diate

n
ial
ry adva
indust

too amo unt
2 m

dif ficult diate
inte rme
AB04 it

inte

fair muc unt
t
CodeC y mediu

no
.1
ner

fi
ial

fair amo
t
begin

B2

litt unt
fair e
ry

triv

h
triv ple
indust y

dif rme

n
A03 2 eCit ium

sim ial

fi
ner

very amo
A med

sim

le
ple
Cod

dif ficult
begin d

no amo unt
dif ficult
adem

triv ial

fair so m unt
5 2 ity

sim ficult

fair amo uch
triv ple diate
AB0

sim ial

fair amo
CodeC l large ac

triv

ple
nce

inte cult le

no amo unt
fair e
emy

sim ple
adva

dif rme diate

n
sim ial

dif ossib

t
im ple
inte ple

t so un
no
inte ficult diate
2

very amo uch
sim rme
AA04 Exc acad nced

triv
large

.2

sim ple diate

litt unt
ial
Ecl+

inte ficult diate
inte rme
y

B1

m
adva

p

t
triv
academ y

sim ple

no so m uch
fi
Excl

ial
IA13 3

no so m le
inte ple

dif ossib diate
sim ple

no so m uch
nced

sim rme
large

sim

ple

dif rme

no so m uch
Ecl+ adva

sim
dif ple

mu ch
triv ficult
academ y

dif cult le

fair
Excl

sim ple
3

im rme

ch
IA14

u
large nced

sim

triv ple

inte rme iate
sim ple diate
sim ial

t
Ecl+

no
sim ple
adva

inte rme diate
triv ial

t
academ

sim ple
Excl

im ficult diate
ult
inte ficult

inte rme diate
dif ial

d
3

p

re 4– –3 triviaial >1 –6 inteterme ediate d intermssible
ult

t
B06

t so
e nced

inte ple

im cult
inte rme

rm diate
.1

fi
sim ple diate
sim rme

t
A larg

fic
Ecl+

fic

sim rme diate
B1
dustry adva

e
im ple diate
dif ple

fi

dif ossib le
xcl

dif ficult
dif

po
3

ult le
7

inte ple

dif
AB0

sim rme
E in nced

inte ficult le
large

ult

im ossib
dif ossib
sim

dif ossib diate
e
Ecl+

inte ficult le
ustry adva

fic

te
inte rme diate
te

sim rme le
l

sk

dif ossib

triv ple
3

if
AA05 ind

inte rme iate

dia
Exc

ult le
p
dia

inte ossib
d

p
large

im rme
fic
A4 er Ta

rm diate
vance

p
triv ial

inte
im
Ecl+

le

sim ple diate
inte rme

inte rme diate
sim rme

inte ial
inte rme diate

d
ry

p
ssib
Excl indust ad

inte rme diate
te
3

p
im
AA06

sim ple
large nced

dif ple
P
.2

sim rme diate

p
triv ficult diate
inte ficult
im ficult diate

dia

fic
dif rme le

inte ple
Ecl+

po

sim rme
triv ple diate
e
inte rme
el

im ficult
ry adva

inte ossib

1– 0 sim rme diate
xcl indust

ev

im
AA07 3
edium

dif ple diate
dif rme
cl+E ner

im ossib le

te g. in ple
yL

te
begin

dif

inte
E m

inte ossib le

4 3 in rm
p

im ossib
ry

dia
im ficult iate
indust y

rm le
ult
3 Excl ium

inte rme iate
15

te
triv ial
ner

sim rme

te
im rme iate
dif ficult

inte ial
IA

dif ial
med

A4 iffic

ed
p
Ecl+ begin d

dif4 <1 1 trivia cult
inte ult

sim ossib diate

sim 4– le 0 diateE of tr mple edia
d
sim rme diate

p

inte ficult le

triv ple
Excl academ y

inte rme
ssib te

d
sim ple diate
IA16 4 m

p

ssib

fic
D
mediu

l
1– in6mple R diate
.1

fi
ple

ub k adva owlle er eable simp rmed7ia1 10 sim7–rmea ber siinterm l

3 mp edia .En
dia

dif ple le
nce

ia
Ecl+

inte rme

1–
adva

dif ficult le
po
sim

1– si3 term ev
academ y

im rme
dif
sim ial diate
4 Excl m

4–si etermse
IA01

inte rme diate

pin ar l
mediu nced

cli Ye ia
inte

im rme iate
sim ficult
dif
Ecl+

p

iv
adva

po

< 3
dif oss iate
inte ple

dif ficult
e
em

y
im ossib le
Excl

inte ple
rm

inte rme
4 m acad

im ficult

d
IA18

< 6 ult
dif ossib le
nced

im ossib
mediu

le
s
dif

d
sim rme diate

7– OP sim ficult ible
inte cult le
Ecl+

1 6 triv le
y

sim ple
adva

dif ficult diate
dif ple diate

4– 6 simp
academ y

fic
triv

sim ple
4 Excl m

p
IA19 mediu nced

–
l
inte ple

inte fiault um
ple

inte rme

p

1 1
Ecl+

sim rme diate
e
inte rme diate
A3

p
sim ficult
adva

p

10 d ple
rm

if J N
sim
academ y

sim ple diate

7– –3
triv rme diate

fi
l

4–
B08 4 +Exc ium

v
sim ple

inte ficult
nced

inte ial diate
inte ult
sim ple diate

4– 0
sk
c
A med

triv ple

inte rme
inte rme

1
Ecl

dif

1
adva

fic

< 6
dif
inte rme diate

6
academ y

1– d ta
sim ial
Excl

7–mp–3le
p
4 ium

ple
09

inte rme iate

an– 6
sim rme
iate
nced

4– 1
sim10 le
si 1 p
AB

inte rme iate
med

sim rme diate
Ecl+

dif –6 ple

3
sim ple

7– triv3ficult

1– 6
adva

inte rme

triv rme diate
sim ple diate

d
triv ple diate

inte
ed
adem

si ple

O

1
sim ple
l

4– 3
te
4

d
m

4 3
AA1 0 Exc

le sim l kn gin ng. si mple
mediu ac

10 ial
inte ple

inte ple > –10
rm
.2

inte rme

ial dia

1– 6
triv plen ow ner si mple

7– –6
4
Ecl+

triv ial diate
A2

sim rme
stry

1–
cks

inte

sim ialeg owle dg tr mple

1– 3
7

> 10
dif ple diate
AA11 4 Excl m indu

0

su
triv le 4 te
sim ple inn dg eab ivial
mediu d blo

< 3

10
inte

sim pele inn er eab mple

pr – 10
dif ple

> e3 s
triv rme
inte ficult
Ecl+

–
–6
triv ial
nts an

1 1
si le

< 1
sim ial

>
sim ficult diate
sim rme
sim ial
Excl

<
ad gin led d able trivia 7l 1–3
4 m

ple
AA12

7– –3
te

le
inte
iate

1– 1
1
mediu

triv

7– 10
atme

triv ple

ial b ev.E
inte ficult

sim ial

> 10
ple dia
Ecl+

1 0
sim ple

3
dif rme

4 0
im rme diate

1
sim ial
triv ple diate

l –
to tre

10
ed

tr ple

1
sim
l

inte ple

7– 10me

> –3
le
inte ssib te

ainte va ssib rte able iv ple R
4

sim rme

7– <1
AA13

ia
+Exc

7– –6
>ti 6
rm
.1

im norminn dg eabsi ivial
dia

e

1– 0
ed 4–
cts

1– 0
inte ple
Ecl

1 0
knintea nce ia te er triv cult iate
inte rme

1
A2

sim rme

1
dif rme le

1
inte
subje

4– 3
sim ple

< 3
7– –3
lem

triv no inn er
AA14 4

4 0
sim ple

d

1– 6
te k si g rmc eiate b trivia

1
of the

< 1
inte

k
sim

7– 10
sim ple

triv gin le r

7– –6
w e
triv ial

tr
po

1 3
n dg
sim ple

4– 1
te le d ficu lip le el

b

> 10
b g
sim ial

bleeiv
ment

4 0
triv ple

be
triv ple

dif Essibev
IA20

–3
sim kn ult wle e
te

1

4– 6
intebeple wle dg

1– 0
kp g
iffi knolt s
po lt

be cts nce ge
dif ial

> –6
triv ial

fi

1
ad po w edia e
triv ial

im ficuL
ial dia

1– 3
beia le
ssign

1– 6
be va ne gearc

4– 3
sim ple diate
sim ial

–
ial

d
e

7– 10
dif ple

1
c
inte ple

kn ia l
nc
sim ficult

1– 3
triv rme

4– 3
ea

ult

1– 6
be tevan ncd legea

e

> 10
e
ie
.2. Th

1 3
’p

1– 6
sim

ple

4– 3
fic

sim rme

o
A1

sim ficult

in d rme d

10
c

kn dif w plee dd te
sim ple

7– –3
le

1– 3
4– 6
d
dif

ad tevrm edia Etep

kn gin nce r
k g
A

7– –6
w
sim ple
ple

no m inn ee ia
x
te

4– 0
inte

be ow nce d able
Table

ad ow ne d
ad pe ne d able ab k egin tece dge le able

< 3
7– 6
sim

ple

jeno
sim ple

1
4
in d rm dia

4– 0
dif

ial dia
triv ial

sim ple

1 6
sim ple

1
ad intermncedia te
de

le now ne rmediaable

4– 1
1– 0
triv ficult iate
sim ial

ad van led r
ad imw ficled r

4 6
kn va ce ge
inte ple

te
triv

ple

b inan poss ge

1
inte ple

inte v rme

dsi ten dg ia te
te

7– –3
triv rme

< 6
Co

sim

1– 3
ow rm e d

v le lt
IA0 1

va rme de te

7– –6
ed
ple dia

ib
triv ial
d

10
b
IA0

u

ble
a

1– 1
IA0 2

3
sim ple diate
dif rme

sim ial

va le e d
iate adv mp d ia a

7– 10
triv

inte ple
IA0 3

eles

4– 6
sim rme

Ja

be gin led d

1 3
a

be gin ne gea
d

7– 10

4–
AB 4

4– 0
ce d

b
AB 01

inte

4 6
1

1 3
–
7– 10
kn pe in cmple edia te

Tha
sim rme

o
sim ficult

a

be ne.5.led r
dif ial

v

–
7
02

4– 6
ge
te

7– –6
kn pe led d simble l diatex dva si rt term edia

> –3
> 10
ex gin ne r
ex gin tencple d
le
AA 5

7– 6
ple

te p in nc rm g iate

be sim nce
AA 01

1– 0
inte
ta

IA0

1– 0
ble

4 0
sim ple
sim ple

be pe ne r
a pein nermd
ad an te ded iate
Da

le

1

1
7– 10
IA0 02

te a in

1
e
Osi ple

1– 3
sim ple

ial dia adv ansimed ed te
inte ple diate adv inrt term edab

be van letrivia edia
A.5 Data

1– 3
7– –6
dia ex va te led ed

ad va ne gea

be gin rt r
ad gin led r
sim

r

– 0
kn inOP ple
IA0 6

triv ple

ete a si ce ple iate

be ow ne
te

c rm ia
A.5

4– 3
e e

7 1
noA

1– 3
a
inte rme iate ad in wterm

4 0
7

> 10
ial dia
triv ial

kn gin

1
ad ow rt term
m

4– 6
IA0 8

3
7– –6
inte ple diate

ad ow ne d leble
ed
sim ial

kn gin ne r
ad va nce r
IA0

4 0
be gin ne
AB 9

inte ple

kn gin cesimgea l
triv rme

4– 6
le

1
IA1 03

kn va nce d

1– 0
ex ow ne r

7– –6
n
inte ple diate

o

sim ple ex per t in mple edia

ab

1

4– 6
p
IA1 0

ad ow led ge ble
kn ow nce d

1– 3
sim rme

kn pe led r
ge

4 0
d
ex pe nce mple
IA1 1

T

4– 6
dif ial diate

ble

1
ad va led ge able
d

4– 3
kn ow led gea
e

sim ficult ex pe rtsi term
AB 2

d

> –6
sim rme

mp ed
AA 04

7– 6
inte

ad va nce ge able

be ow nce gea

4 6
e
in d
03

kn ow led
be va led r

1– 0
triv ple kn per t si term

ex gin nce ge

10
ble
e

1
ad va nce d able

> –6
AA 05

ad ow rt
kn va led
rm ad ow led simp le
rm

4– 3
v
04

gin led d
ad gin ne gea
knedia n led ginte le

4– 0
195

AB

ex ow tece geaivable e
CodeCity Ex

inte ficult
d

4– 6

1
tr rm
IA1 3

inte rme

4– 6
IA1

triv

ad va rt r

4 6
AB 4

r
06

ia
x

7– 6
ple

b va ne r
periment

sim rme

> –6
AA 07

e

10
kn va nce d
ad va ce d
AA 05

kn gin nce r

4– 0
ial kn ow t
AB

ad van nce
triv rme

kn va led d able

1
AA 06

ad ow ne d
kn ow led d able
ad ow nce d
kn va nce d
le

4 6
IA1 07

ab

kn van led d
ad va nce ge able
kn ow nce d

kn va led r

> –6
ad ow nce ge
ge
5

be
dif ple

ge

be ow led d able
ad ow led ge able
kn van nce d able
ad ow led d
le
IA0 6
n

10
ad va led ge

ns
IA1

ab
sim
io

ud IA 1

ad va led ge able
va

kn ow nce ge
kn ow ce
kn ow led ge

e
sit

IA1 8
T1

ge

ne dge able

tio
ad va nce ge able
en 1
AB 9

ex ow rt
Po

ad pe led
08

le
r
De TO, lop er

ad va nce d able
t
pe

na
gin le ge
AA 09

ab
C ve lop
b

M ve De er
lo

ad va rt d able
AA 10

ow ce d
inte

ad va nce d
Jo
CodeCity Experim

ad van rt
AB

bi
ad va ce
as lo ve

11
De eve

.S n e t

ex van led d
St

ex va nce d
Ph .D ulta tud den

arAA 2

ad pe ce ge

om
174

C ast er S er

se AA1
So ead . St ude , Ph t

r
D

AA her3
So ftw of ude nt .D.

ad pe nce d
n
Ph ons er S tu

c 1
t p
e
ent

IA2 14

kn van nce d
Ag

ad va rt d

tc
0

st

ge ble
kn ow ce d
aly
t t

en
Participant:

30

le
ex ow led d
ad per led gea
ex va nce d
ad van nce d
De ftw are IT nt
Compl. Time

An

ad va nce

ab
Re

ad
ad va nce
34

r
de

m
ad pe nce d
Co oje ent me ngin ee

d
T1 A.4 Task Solu

r/
Co

42

Le eer
er,
M

Pr ud lop e E ngin

at
A.4.1 T1: Azu

ad va rt d
ad pe nce d
ge
174 tion Oracles

IA 01

ex va ce
ad
37
Total (excl. A4.2)

tre
t na

ex va rt d
reus, analyze

IA
IA 02

St ve ar E
21

H .D

ed
ec a

va ce d

va t
e Per Task

Co .D er S tu den chit m M
d with CodeCi

IA 03

ad van nce

by
nt
21

nc
B2.1
Completion Tim

ad va nce d
kn va rt d
194 B1.2
7 A.5 48.5Data

M ast er S ava , Sy er
A 04
5

ad pe nce
ty

29
CodeCity Experiment
A1

ad va nce d
B1.1

ge ble
A.4 Task Solution

Ph ast er S tu Ar ste
M ast r J ant nag
AB B01

d
26
A4.2 52.9
6

re
ex va nce d
Oracles

le
ex ow led d
ad per led gea
IA 02
19
A4.1 1.02

kn ow nce
M nio ult Ma
26
A.4.1 T1: Azureus

ab
A3 1.90 34.67

ex per nce d

te
T1

AA 05
9.85
a
A2.2

35
44.67

So ns . St tu den t
at

pe t d
ns ct

us
, analyzed with Code 4.42

AA 01
A2.1

ad
Either

27
A1 7.28 2.33

M ftw ulta ude den t
Participant: Cod

cl
02
0 2.92 37.97
D
174 eCity A.4 Task Solution Oracles 10.0

25

M ast are nt nt t
5.38 5.25

ad va nce d
10.00 47.33

ad van t
5

IA 06

,
va ce
A.

32
A1 4.32

on
Introduction There are no 1.83

ex va nce d
5.33

rt
8.80

IA
07
2.95 45.50

Se

Pr .D . St ud den t ct
28
unit tests in IA01 8.58

ad pe nce d
2.55

i
Ph .D . St tu den ite
1.75

IA 08
55.50

at
39
the system [1pt 6.08 9.37

ex va rt d
or

Ph .D er S tu rch
1.08

IA
A 09
5.62

m
2.42

38
], 9.43 1.33 32.83
CodeCity Experiment

IA B03

ct
Either IA02 10.00

S A

or
34

ad
2.42

So oje . St ude ent t
A.4.1 T1: Azureus, analyzed with CodeCity 42.83

ite
10.00

va t d
T1 6.92

IA 10
174 Centralized A.4 Task Solution Oracles

ad per nce
22

nf
7.92 2.58

ch
There are no unit in a single package 6.25 2.25 43.25

IA 11
6.08

22
The aim of this

Ar

li
IA03 6.92

as er

ed
4.00 10.00 53.25

D ftw ct M ude nt
A 12

22
[1pt]. Sinctests in the system hierarchy whose

na
6.00

er,

nc
experim 1.67

AA B04
6.25

So eve ar an nt
9.08

M ns ar er ch er
e there is only

ct
29
27.08

ag
2.58

Ph t
ent A1 or [1pt], root is in org. 4.75

rso
is to compare IA04

A 03

Co ftw lop e Ar ag
ite
practit 3.92 9.42
Participant: ioners analyzing me one test class

an
37.08

32
answer, the answ gudy.azureus2.u 4.83 10.00

A B05
tool efficiency (i.e., TestUser 2.17

Ph ofe . St ud ent t ct M
8.08

31

Pr .D . St ud den oje r

pe
A.4.1 T1: Azureus, analyzed with CodeCityely i.console.multiu 1.33

Ph .D . St tu , Pr ee
1.58 36.00

IA A04
Introdu Centralized in a er is complet Manager), if they AB0 1 4.25 3.75

23
ction dium to large-s in supporting

Ph .D er S ant ngin
single package hiera ser 1.67 10.00 46.00

ts’
wrong.

13
Either don’t give the 5.58

23
cale software software rchy whose root is 10.0 0 3.50 1.33 1.42 45.92

A 14

Ph ast ult e E
full correct AB02

ec
4.75

30
[1pt]. Since there 5.17
systems.

A B06
in org.gudy.azure 1.33 10.00 55.67

IA
is only one [1pt],

26

bj
You will use Co A1 There are no unit tests in the system test class (i.e., us2.ui.console.mult 3.67 1.33

AA B07
A2.1 6.33 3.17

30
3.25 44.67

su
answer, the answ TestUserManager), IA05 8.75

A 05
deCity to analyze iuser

M .D sso ude ent
2.08 8.33

40
The aim of this or er is completely wron if they don’t give 9.75 54.67

A A06
4.42

nt

e
1.50

39
experiment is to Azureus, a BitTorre g. the full correct 9.67 5.00

Th
10.00 1.33

IA A07
Either AA01 36.25

30
7.17 Correctness Per Task 7.33 Correctness
Participant:You are analyzin compare tool effic nt client wri Dispersed [0pt 2.50 10.00 46.25

pe d t
IA 15
Centralized in a single package hierarchy whose root is in org.gudy.azureus2

lo Stu en
practitioners

27

t

1.
s othe Code 4.17 Total 2.33

.S r

r en
tten in A2.1 in the system [1pt], rwise] 8.33

IA 16
iency in supporting Java. .ui.console.multiuser 10.00 A4.2

ve er tud
AA02 A2.1 1.42 24.67

39
There are no unit tests in the follo

A.
g med
given maximum ium to large-sc A1 A2.2 A34.42 A4.1 B1.1 0
10.0 B1.2 2.42 B2.1 (excl. A4.2)

01
34.67

21
Introduction software [1pt]. Since there is only one test (max.
wing class (i.e., TestUserManager), if they don’t give 5.92
10.00 1.17

e
IA 17
ale 4.58

23
100 minutes for software systems 5) packages the full correct
1.00 4.00 39.50

bl
or IA06 1.00 6.33 0.00

IA
A 19
10.0 1.00 3.33

24

De ast
solving 10 tas . answer, the answer is completely wrong. [0.2pts for each IA01 0.00 1.00
5.67 1.00 6.25 1.00 0.00 0 6.00 1.42 6.00
47.33

Ta
A B08

23
You will use CodeCit Centralized in aDispersed [0pts otherwise]zure root is in org.gudy.azureus2.ui.console.multiuser]: 7.25 3.75

A B09
ks (10 minute • com.aelitis.a
single package hierarchy whose 4.55 0.00 2.50 37.58

52
You are asked: y IA07 0.80 2.95 1.00 1.00 3.67

A A10
s per task). us.core IA02 1.00 0.50
2.25 1.00 0.00 7.83 1.00 6.30 1.25 6.308
46.0

28
The aim of this experiment to anacompare tool a BitTorre in supporting
in the following

A A11
is to lyze Azureus, efficiencynt clien
2.67 5.25

24
[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correct
(max. 5) packages IA08 0.00 3.00 1.00 0 0.00
10.0 1.50 37.83

AA A12
1.00 7.17

36
• not to software • com.aelitis.a IA03 0.00 1.00
3.67 0.00 0.00 8.50 0.00 3.00 8.08 3.00 8 45.5

A 13
t written in Java. A2.1 [0.2pts
answer, the answer•is completely wrong. zureus.core.cont for each] : 10.00 5.00
practitioners giveconsult any other particip 1.75 35.32

IA A14
IA09 1.00 5.00
Introduction• areanalyzing medium to large-scale software systems.
You to com.aelitis.azureus. ent IA04 0.00 6.33 0.00 2.00 1.00 9.33 1.00 0.00 3.83 0.00 7.75 1.00 3.08
1.00 5.00 1.75 5.00 3 42.2

20
n maximum
perform the tas 100 minutesant during the exp Dispersed [0pts otherwise]
• com.aelitis.a core AB0 3 6.83 2.83 30.33
for
ks in the specifie solving 10 task eriment; zureus.core. AB01 0.00 1.00 3.25 1.00 3.67 0.807.33 1.00 0.00 4.00 0.00 6.92 1.00 7.42 1.00 5.80 1.25 5.80 3 40.3
• to write dow d order; s (10 A2.1 in the following (max. 5) packages [0.2pts fordownload
• com.aelitis.azu 7.17 2.00 24.75
You willthis experimentnis to current timtool efficiency client written minutes per task).
IA10
You are asked: the compare
use CodeCity to analyze Azureus, a BitTorrent in supportingin Java. reus.core.content each]:
• com.aelitis.a AB02 0.00 1.00 3.83 1.002.40 0.705.921.00 0.00 5.17 10.00
0.00 0.00 5.75 1.00 0.50 4.70 3.83 4.70 0 34.5
The aim of after com e each time bef software zureus.core.imp IA11 4.171.00 5.58 2.00 36.83
30.50
pleting
• • to ann medium all the ore sta • com.aelitis.azu
[0pts otherwise]
Dispersed• com.aelitis.azureus.core l IA05 0.00 1.003.75 1.00 2.67 0.20 0.00 5.58 0.00 9.75
1.00 1.00 1.25 5.20 2.58 5.20
practitionersnot to consult any othe tasks; software systems. rting to read a task and
analyzing oun to large-scale reus.core.down
in the following (max. 5) • com.aelitis.azureus.c load
IA12
3.920.40 2.58
3.67
0.00 6.33
4.08
4.20 1.67 4.20 0 49.0
39.00
You areto perform ce the exprerimenter nt during the exp (10 minutes
participa
• given maximum 100 minutes for that you tasks eriment; once packages [0.2pts for each]:
ore.lws
AA01 0.00 4 0.802.671.00 1.0010.00.00 3.17
0 1.00 0.00 1.83 35.83
to reset your tasks in the spe solving 10 mo
the per task). • com.aelitis.azu
• com.aelitis.azureus.core.content AB0 3.75
0.00 5.83 10.00
9.58
6.00 2.83 6.00 7
2
reus.core.impl 0.00 3 1.00 2.501.003.170.00 43.6
10-minute cified order; are ving on Java. • com AA02 AA0
1.00 5.58 1.00 1.00 1.00 1.25
6.00 26.58
19
You will use to not e down the current s-per-task allocatclient written into another task, in
• CodeCity to analyze Azureus, a BitTorrent
• writ to retu
• com.aelitis.azureus.core .aelitis.azureus.c
• com.aelitis.azu ore.peermanager IA06 0.00 51.00 5.50 0.004.67 0.30
4.33
1.00 6.33 0.00 3.50 0.00 7.83 1.00 1.00 1.67 4.30 2.08 4.30 36.58
time eac
You are asked: plet rn to earlier tasks bech time before star ed timer; order • com.aelitis.azureus.core.download
reus. .peerdb AB0 5.17 4.83 43.92
after com ing
• for each task,all ause it affects ting to read a task • com.aelitis.acore.lws
• com.aelitis.azureus.core.content zureus.c IA07 1.00 41.00 7.08 1.00 3.670.17 1.00 4.75 0.00 3.25 0.00 10.00
1.00 1.00 1.83 6.17 4.17 6.17 49.75
• not to maximum otherthein for solving the experiment;
tasks; AA0 5.33 46.27
You are givenconsult any100experimthe required 10 tasks (10the timing; task). and once
2.33
• tocheck ce the to fill • com.aelitis.azu ore.stats 56.27
announ participant duringinfo
minutes minutes per • com.aelitis.azureus.core.impl reus.
• com.aelitis.acore.peerm IA08 1.00 IA13 0.50 3.00 1.00 4.67 0.83 1.00 10.000.00 4.50 1.00 5.83 1.00 10.001.00 7.33 2.00 7.33
3.85 55.92
the rmation. In the
• to perform tthe tasks st appropenteranswe are moving on case of mu • com.aelitis.azureus.core.download us.c anager.peerdb
to req your mo in the specifiedthat you
2.42 10.00 65.92
rese riat
10-minutes-p e order; and ltip
to another task le choice • com.aelitis.azu
zure ore.torrent IA09 1.00 IA141.00 6.67 1.00 9.00 1.00 2.05 1.00 6.95 10.00
0.00 1.00 1.00 10.001.00 8.00 2.67 8.00
3.67 50.33
ues r • com.aelitis.azureus.core.lws
• to • not downted. current time er-task allocatedprovide addread a informa , in order s 10.00 6.67 8.00 5
reus.core.stats 5.75 54.7
write to retuthe
You are asked: rn to earlier taskeach time before timer; to itional task and once
starting • com.aeli
• com.aelitis.azureus.core.impltis.azureus.p AB03 1.00AB06 1.00 5.67
1.00 1.00 4.67 10.0
1.00 0.00
0 6.83
1.00 1.00 10.001.00 8.00
3.17 30.83
tion, if • com.aelitis.azu lugins.net.b 9.33 4.42 1.58 6.17 837.5
• completing all participant s because experiment; • com.aelitis.azureus.core.peermanager.peerdb uddy 1.00 8.75 5.25
• not after for each erim , to filltasks; during the it affects the timing; 22.83
IA10 1.00AB07 1.00 0.17 3.75 1.00 8.42 0.00 0.00 1.00 1.00 1.58
6.17
to consultexp task the in the
The any other • com.aeli
reus.core.torrent
• com.aelitis.azureus.core.lws tis.azureus.p 6.08 3.25 6.75 5.33
1.835.00 28.00
ent is con required
• to announce the in the specified order; information. In the another task, in order
chethe the most app cluded with a sho
experimenter that you are moving on to case • com.aelitis.azu lugins.net.budd IA11 1.00 AA01.00 7.00
5 1.00 0.00 3.67 1.00 6.58 0.16 0.00 0.00 1.00
2.75 3.33
5.16 52.67
• to perform ck tasks ropriate answer and rt debriefing que of multiple choices
• com.aelitis.azureus.core.stats
reus.
• com.aelitis.a plugins.net.buddy
y.swt
IA12 1.00 AA06 1.00 6.83 1.00
2.00
1.00 2.50 1.00 2.17 0.00 2.92 5.17
1.00 10.00 1.00
1.00 1.75 7.83 62.6 7
50.25
to reset este 10-minutes-per-task allocated starting add 5.08 8.00 8.0056.75
• to write requyour d.current time each time before timer;ide to read astionnaire.once
prov • com.aelitis.azureus.core.peermanager.peerdbzureus.p 3.67
down the itionaltask rmation
info and , if • com.aelitis.azureus.core.torrent lugins.net.buddy.tracker
• com.aelitis.azu AB04 1.00 AA0 7
1.00 3.67
1.00 0.67 5.08 1.00 10.00
0.00
4.33
0.00 1.00 2.00 4.50
39.02
reus.
• com.aelitis.a plugins.net.buddy.swt 6.50 1.00 8.50 6.67 6.6743.32
after not to return tothe tasks;
• completing all earlier tasks because it affects the timing; • com.aelitis.azureus.core.stats zureus.plugi AA03 1.00 IA15 1.00 9.75 8.83
0.80 7.42 0.33 4.33 1.00 9.50
0.16
4.00
0.00 1.00 2.10 2.93 46.55
• com.aelitis.azureus.plugins.net.buddy ns.removerules
• com.aelitis.azu 4.30 1.00 9.92 6.29 6.13 50.53
• to • for each task, entfill in the required information. In the another multiple order
The experim to is 0 4.03 3.12
announce the experimenter that you are moving on to case of task, in choices
Thank you for concluded reus.
• com.aelitis.a plugins.net.buddy.tracker AB05 1.00 IA16 1.00 10.0 1.00 3.90 1.00 1.00
4.38 9.20
0.00 0.00 1.00
3.98 1.00 9.77 7.00 4.13 7.00 38.17 33.08
participatin with a short deb
check the most appropriate answer andtimer; riefing que information, • com.aelitis.azureus.core.torrentzureus.plugi 2.55 4.72 5.58 44.42
to reset your 10-minutes-per-task g in this experimentadditionalstionnaire. if
allocated provide • com.aelitis.azu
reus.
ns.s
• com.aelitis.azureus.plugins.net.buddy.swtharing.hoster AA04 1.00 IA01 1.00 1.00 5.13 0.50 1.00
4.58 9.82
0.00 0.00 1.00
5.08 1.00 6.58 6.50 1.92 6.50 50.33
! • com.aelitis.a plugins.remove 5.28 3.83 1.92
• not requested. earlier tasks because it affects the timing; • com.aelitis.azureus.plugins.net.buddy lugi rules IA13 1.00 IA181.00 1.00 3.83 0.83 1.00 3.50
1.00 0.00 0.00 9.42 6.83 3.58 33.83
to return to
! ! • com.aelitis.azu
zureus.p ns.starts 3.33
4.50
3.50 5.92 1.00 5.83 39.83
1.50
! ! Richard Wettel
• com.aelitis.azureus.plugins.net.buddy.tracker toprules.default
reus. 0.00 IA19 0.30 10.00 8.83 0.00 2.00 5.10 1.92 38.00
• for each task, to fill in the required information., In the case of multiple choices • com.aelitis.a plugins.sharing.hoster plugin IA14 0.80 0.33
6.08 1.08 1.00 0.67 3.42 1.00 6.00 1.00 4.43 41.33
2.33
The thenk you appropriate answer and provideMichele Lanza, Romain if
• com.aelitis.azureus.plugins.net.buddy.swtzureus.plugi 8 10.00 1.17 31.92
experiment is concluded with a short debriefing questionnaire. Ro
Tha most for part • com.aelitis.azureus.plugins.removeruns.tracker.dht
• com.aelitis.azu AB06 1.00 AB0 1.00 1.00 0.00
4.83 4.33
0.00 0.00 0.00 1.00
3.33 0.00 4.83 4.00 4.00 37.08
check icipating in this exp additional information, bbes reus.
• com.aelitis.a plugins.startsto les AB09 5.83 7.33 8.00
3.67 2.27 1.67 2.92 34.25
eriment! • com.aelitis.azureus.plugins.net.buddy.tracker prules.defaultplugin
zureus.plugi AB07 0.00 0.60 0.67
3.17 0.00
6.08 5.08 0.00
0.00 0.00
3.50 0.00 5.17 1.00 2.27
3.00 37.58
requested. • com.aelitis.azureus.plugins.sharing.hns.tracker.local
• com.aelitis.azu AA10 6.08 1.00 3.83 4.42 7.00 1.42 53.08
!
reus.
• com.aelitis.a plugins.tracker.dht oster AA05 1.00 1.00 1.00
6.17 0.00 4.08 1.00 0.00
5.25 1.00 3.33 1.00 6.00 58.75
5.42
! ! ! • com.aelitis.azureus.plugins.removeruleszureus.plugins.tr AA11 3.92 1.00 4.75 6.33 1.42
3.00 43.33
40.08
Richard Wettel, Mic • com.aelitis.azu acker.peerauth
• com.aelitis.azureus.plugins.startstoprules.defaultplugin AA06 0.00 1.00
AA12
0.00
6.75 0.00 4.75 1.00 0 0.00 5.92 0.00 5.67 1.00 4.002.42 4.50 32.67
hele Lanza, Rom
The experiment is concluded with a short debriefing questionnaire. ain Rob reus.
• com.aelitis.a plugins.tracker. 10.0
7.00 1.00 0 1.00 0 0.00
10.0 3.25 0.009.17 5.00 4.00 36.08
bes • com.aelitis.azureus.plugins.sharing.hoster local
zureus.ui.swt.co AA07 1.00 1.00 0.00
7.00 0.00 2.83 1.00 1.42 1.33
Thank you for participating in this experiment! • com.aelitis.azu
• com.aelitis.azureus.plugins.tracker.dht ntent.columns AA13
1.50 1.00 3.33 10.0
3.42 0.002.424.80
reus.plugins.tracker. IA15 1.00 0.80 0.00
6.33 0.00 1.008.00 0.00 2.33 1.00 3.80
• com.aelitis.a
• com.aelitis.azureus.plugins.startstoprules.defaultplugin
zureus.ui.swt.she peerauth AA14 4.75
• com.aelitis.azu
• com.aelitis.azureus.plugins.tracker.local lls.main IA16 0.00 0.00 0.67
6.83 0.00 5.58 1.00 0.67 0.00 1.00 1.00 4.34 e, in3.67 utes
min
IA20 tim
task 1.00 tion
reus.
! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes • com.aelitis.a ui.swt.content.columns IA01 1.00 1.00 1.00 0.50 1.00 1.00 subjects’ 1.00 comple 7.50
• com.aelitis.azureus.plugins.tracker.dht
• com.aelitis.azu
zureus.util
• com.aelitis.azureus.plugins.tracker.peerauth The 0.00
Table A.4. 1.00 0.00 1.00 1.00 7.50
6.50
Thank you for participating in this experiment! reus.ui.swt.shells.m IA18 1.00 1.00 1.00 0.50 1.00 6.50
• com.aelitis.azureus.plugins.tracker.local ain
• com.aelitis.azu
• com.aelitis.azureus.ui.swt.content.columns
IA19 0.00 1.00 0.60 0.00 1.00 1.00 0.00 1.00 0.00 4.60 3.60
reus.util
• com.aelitis.azureus.plugins.tracker.peerauth AB08 0.00 1.00 0.40 0.38 1.00 1.00 1.00 1.00 1.00 6.78 5.78
! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes • com.aelitis.azureus.ui.swt.shells.main AB09 1.00 1.00 0.60 0.50 1.00 1.00 0.00 1.00 1.00 7.10 6.10
• com.aelitis.azureus.ui.swt.content.columns AA10 1.00 1.00 0.80 0.38 1.00 1.00 0.00 1.00 1.00 7.18 6.18
• com.aelitis.azureus.util
• com.aelitis.azureus.ui.swt.shells.main
AA11 1.00 1.00 0.80 0.00 1.00 1.00 0.00 1.00 1.00 6.80 5.80
AA12 0.00 0.50 0.00 0.00 1.00 1.00 0.00 1.00 1.00 4.50 3.50
• com.aelitis.azureus.util AA13 0.00 1.00 0.00 0.25 1.00 1.00 0.00 1.00 1.00 5.25 4.25
AA14 1.00 1.00 0.60 0.00 1.00 1.00 0.90 1.00 1.00 7.50 6.50
IA20 0.00 1.00 1.00 0.38 1.00 1.00 0.00 1.00 1.00 6.38 5.38

Table A.3. The correctness of the subjects’ solutions to the tasks

Pre-experiment
Handouts Oracles Data
questionnaire

technical report 2010

Summary
list of
design
desiderata

Summary
list of
design
desiderata

6 months
4 locations
41 subjects

Summary
list of
design +24.26%
correctness
desiderata
-12.01%
completion
6 months time
4 locations
41 subjects

Take-away message
Visualizing software systems as
cities is a viable alternative to the
state of the practice approaches
for software exploration.

Software Visualization - Promises & Perils

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Software Visualization - Promises & Perils (20)

Recently uploaded (20)

Software Visualization - Promises & Perils