Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning

Human Computer
Interaction
Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE
USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING

Human-Computer Interaction Lab

BenShneiderman
BenBederson
JonFroehlich
JenGolbeck
LeahFindlater
MarshiniChetty
JennyPreece
AllisonDruin
MonaLeighGuha
TammyClegg
JuneAhn
EvanGolub
TimClausner
KentNorman
IraChinoy
KariKraus
AnneRose
CatherinePlaisant
computer science
hcil
JessicaVitak
NiklasElmqvist
NicholasDiakopoulos

HCIL Hackerspace
Founded in 2012

HCIL Hackerspace
Looking North

HCIL Hackerspace
Looking South

Three Soldering Stations
HCIL Hackerspace

Craft/Textile Station
HCIL Hackerspace

Two Mannequins
HCIL Hackerspace

Wall of Electronic Components HCIL Hackerspace

Two 3D-Printers
HCIL Hackerspace

One CNC Machine
HCIL Hackerspace

Physical Making
HCIL Student Leyla Norooz

Electronics Making
HCIL student Tansy McBurnie

E-Textile Design
HCIL Student Michael Gubbels showing SFF

Collaborative Working
HCIL students Joseph, Cy, Matt, and Jonah

Student Sewing HCIL student Matt sewing

Fun!
HCIL students Kotaro Hara and Allan Fong

More Fun!
HCIL students Sean, Michael, Alexa, and me

30.6
million U.S. adults with mobility impairment

15.2
million use an assistive aid

Incomplete Sidewalks
Physical Obstacles
Surface Problems
No Curb Ramps
Stairs/Businesses

The National Council on Disability noted that there is no comprehensive information on “the degree to which sidewalks are accessible” in cities.
National Council on Disability, 2007
The impact of the Americans with Disabilities Act: Assessing the progress toward achieving the goals of the ADA

The lack of street-level accessibility information can have a significant impact on the independence and mobility of citizens
cf. Nuernberger, 2008; Thapar et al., 2004

I usually don’t go where I don’t know [about accessible routes]
-P3, congenital polyneuropathy

“Man in Wheelchair Hit By Vehicle Has Died From Injuries”
-The Aurora, May 9, 2013

http://youtu.be/gWuryTNRFzw

http://accesscore.org
This is a mockup interface based on walkscore.com and walkshed.com

How might a tool like AccessScore:
Change the way people think about and understand their neighborhoods
Influence property values
Impact where people choose to live
Change how governments/citizens make decisions about infrastructural investments

AccessScore would not change how people navigate the city, for this we need a different tool…

NAVIGATION TOOLS ARE NOT ACCESSIBILITY AWARE

Routing for: Manual Wheelchair
1st of 3 Suggested Routes 16 minutes, 0.7 miles, 1 obstacle
!
!
!
!
A
B
Route 1
Route 2
Surface Problem
Avg Severity: 3.6 (Hard to Pass)
Recent Comments:
“Obstacle is passable in a manual chair but not in a motorized chair”
Routing for: Manual Wheelchair
A
1st of 3 Suggested Routes 16 minutes, 0.7 miles, 1 obstacle
!
ACCESSIBILITY AWARE NAVIGATION SYSTEMS

Where is this data going to come from?

Safe Routes to School Walkability Audit
Rock Hill, South Carolina
Walkability Audit Wake County, North Carolina
Walkability Audit Wake County, North Carolina
TRADITIONAL WALKABILITY AUDITS

Safe Routes to School Walkability Audit
Rock Hill, South Carolina
Walkability Audit
Wake County, North Carolina
Walkability Audit
Wake County, North Carolina
TRADITIONAL WALKABILITY AUDITS

http://www1.nyc.gov/311/index.page
MOBILE REPORTING SOLUTIONS

Similar to physical audits, these tools are built for in situ reporting and do not support remote, virtual inquiry—which limits scalability
Not designed for accessibility data collection

MARK & FIND ACCESSIBLE BUSINESSES
wheelmap.org
axsmap.com

MARK & FIND ACCESSIBLE BUSINESSES
wheelmap.org
axsmap.com
Focuses on businesses rather than streets & sidewalks
Model is still to report on places you’ve visited

Our Approach: Use Google Street View (GSV) as a massive data source for scalably finding and characterizing street-level accessibility

HIGH-LEVEL RESEARCH QUESTIONS
1.Can we use Google Street View (GSV) to find street- level accessibility problems?
2.Can we create interactive systems to allow minimally trained crowdworkers to quickly and accurately perform remote audit tasks?
3.Can we use computer vision and machine learning to scale our approach?

ASSETS’12 Poster
Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TOWARDS SCALABLE ACCESSIBILITY DATA COLLECTION

ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop
HCOMP’13 Poster
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14
TODAY’S TALK

ASSETS’12 Poster
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
CHI’13
ASSETS’13
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK

ASSETS’12 GOALS:
1.Investigate viability of reapproprating online map imagery to determine sidewalk accessibility via crowd workers
2.Examine the effect of three different interactive labeling interfaces on task accuracy and duration

WEB-BASED LABELING INTERFACE
FOUR STEP PROCESS
1. Find and mark accessibility problem
2. Select problem category
3. Rate problem severity
4. Submit completed image

WEB-BASED LABELING INTERFACE VIDEO
Video shown to crowd workers before they labeled their first image
http://youtu.be/aD1bx_SikGo

WEB-BASED LABELING INTERFACE VIDEO
http://youtu.be/aD1bx_SikGo

THREE LABELING INTERFACES
Point-and-click
Rectangular Outline
Polygonal Outline
Pixel Granularity

Los Angeles
DATASET: 100 IMAGES
New York
Baltimore
Washington DC

DATASET BREAKDOWN
34
29
27
11
19
0
10
20
30
40
No Curb Ramp
Surface Problem
Object in Path
Sidewalk Ending
No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC

DATASET BREAKDOWN
34
29
27
11
19
0
10
20
30
40
No Curb Ramp
Surface Problem
Object in Path
Sidewalk Ending
Used to evaluate false positive labeling activity

DATASET BREAKDOWN
34
29
27
11
19
0
10
20
30
40
No Curb Ramp
Surface Problem
Object in Path
Sidewalk Ending
Used to evaluate false positive labeling activity
This breakdown based on majority vote data from 3 independent researcher labels

What accessibility problems exist in this image?

R1
R2
R3
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other

Object in Path
Curb Ramp Missing
R1
R2
R3

Object in Path
Curb Ramp Missing
R1
R2
R3
x2
Researcher 1

x4
Object in Path
Curb Ramp Missing
R1
R2
R3
Researcher 2

Researcher 3
Object in Path
Curb Ramp Missing
R1
R2
R3
x8

Researcher 1
Researcher 2
Researcher 3

There are multiple ways to examine the labels.

Object in Path
Curb Ramp Missing
R1
R2
R3
Image Level Analysis
This table tells us what accessibility problems exist in the image

Pixel Level Analysis
Labeled pixels tell us where the accessibility problems exist in the image.

Why do we care about image level vs. pixel level?

Coarse
Precise
Point Location Level
Sub-block Level
Block Level
(Pixel Level)
(Image Level)

Coarse
Precise
Sub-block Level
Block Level
(Pixel Level)
(Image Level)
Pixel level labels could be used for training machine learning algorithms for detection and recognition tasks

Coarse
Precise
Localization Spectrum
Sub-block Level
Block Level
Class Spectrum
Multiclass
Object in Path
Curb Ramp Missing
Prematurely Ending Sidewalk
Surface Problem
Binary
Problem
No Problem
(Pixel Level)
(Image Level)
TWO ACCESSIBILITY PROBLEM SPECTRUMS
Different ways of thinking about accessibility problem labels in GSV
Coarse
Precise

Object in Path
Curb Ramp Missing
R1
R2
R3
Problem
Multiclass label
Binary Label
Sidewalk Ending
Surface Problem
Other

To produce a single ground truth dataset, we used majority vote.

R1
R2
R3
Maj. Vote
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other

ASSETS’12 MTURK STUDY METHOD
Independently posted 3 labeling interfaces to MTurk. Crowdworkers could work with only one interface.
For training, turkers required to watch first 1.5 mins of 3-min instructional video.
Hired ~7 workers per image to explore avg accuracy
Turkers paid ~3-5 cents per HIT. We varied number of images/HIT from 1-10.

ASSETS’12 MTURK DESCRIPTIVE RESULTS
Hired 132 unique workers
Worked on 2,325 assignments
Provided a total of 4,309 labels (AVG=1.9/image)

MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
0%
20%
40%
60%
80%
100%
Point-and-click
Outline
Rectangle
AVERAGE ACCURACY
Higher is better
0
10
20
30
40
50
Point-and-click
Outline
Rectangle
MEDIAN TASK TIME (SECS)
Lower is better

83.0%
82.6%
79.2%
0%
20%
40%
60%
80%
100%
Point-and-click
Outline
Rectangle
AVERAGE ACCURACY
All three interfaces performed similarly. This is without quality control.
0
10
20
30
40
50
Point-and-click
Outline
Rectangle
Higher is better
Lower is better

83.0%
82.6%
79.2%
0%
20%
40%
60%
80%
100%
Point-and-click
Outline
Rectangle
32.9
41.5
43.3
0
10
20
30
40
50
Point-and-click
Outline
Rectangle
AVERAGE ACCURACY
All three interfaces performed similarly. This is without quality control.
Point-and-click is the fastest; 26% faster than Outline & 32% faster than Rectangle
Higher is better
Lower is better

ASSETS’12 CONTRIBUTIONS:
1.Demonstrated that minimally trained crowd workers could locate and categorize sidewalk accessibility problems in GSV images with > 80% accuracy
2.Showed that point-and-click fastest labeling interface but that outline faster than rectangle

ASSETS’12 Poster
HCIC’13 Workshop
HCOMP’13 Poster
CHI’13
ASSETS’13
UIST’14
TODAY’S TALK

ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop
HCOMP’13 Poster
CHI’13
ASSETS’13
UIST’14
TODAY’S TALK

CHI’13 GOALS:
1.Expand ASSETS’12 study with larger sample.
•Examine accuracy as function of turkers/image
•Evaluate quality control mechanisms
•Gain qualitative understanding of failures/successes
2.Validate researcher ground truth with labels from three wheelchair users

Los Angeles
DATASET: EXPANDED TO 229 IMAGES
New York
Baltimore
Washington DC

GROUND TRUTH: MAJORITY VOTE 3 RESEARCHER LABELS
How “good” is our ground truth?

IN-LAB STUDY METHOD
Three wheelchair participants
Independently labeled 75 of 229 GSV images
Used think-aloud protocol. Sessions were video recorded
30-min post-study interview
We used Fleiss’ kappa to measure agreement between wheelchair users and researchers

Here is an example recording from the study session

IN-LAB STUDY RESULTS
Strong agreement (κmulticlass=0.74) between wheelchair participants and researcher labels (ground truth)
In interviews, one participant mentioned using GSV to explore areas prior to travel

CHI’13 MTURK STUDY METHOD
Similar to ASSETS’12 but more images (229 vs. 100) and more turkers (185 vs. 132)
Added crowd verification quality control
Recruited 28+ turkers per image to investigate accuracy as function of workers

University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
Kotaro Hara
Timer: 00:07:00 of 3 hours
10
3 hours
Labeling Interface

Kotaro Hara
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
3 hours
10
Verification Interface

CHI’13 MTURK LABELING STATS
Hired 185 unique workers Worked on 7,517 labeling tasks (AVG=40.6/turker) Provided a total of 13,379 labels (AVG=1.8/image)
Provided a total of 19,189 verifications
CHI’13 MTURK VERIFICATION STATS

CHI’13 MTURK LABELING STATS
Worked on 7,517 labeling tasks (AVG=40.6/turker)
Provided a total of 13,379 labels (AVG=1.8/image)
CHI’13 MTURK VERIFICATION STATS
Provided a total of 19,189 verifications
Median image labeling time vs. verification time: 35.2s vs. 10.5s

CHI’13 MTURK KEY FINDINGS
81% accuracy without quality control
93% accuracy with quality control

Some turker labeling successes...

TURKER LABELING EXAMPLES
Curb Ramp Missing

Object in Path

Prematurely Ending Sidewalk

Surface Problems

Surface Problems
Object in Path

And now some turker failures…

TURKER LABELING ISSUES
Overlabeling
Some Turkers Prone to High False Positives
No Curb Ramp

No Curb Ramp
Overlabeling
Incorrect Object in Path label. Stop sign is in grass.

Overlabeling
Surface Problems

Overlabeling
Surface Problems
Tree not actually an obstacle

Overlabeling
No problems in this image

Overlabeling

T1
T2
T3
Maj. Vote
3 Turker Majority Vote Label
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
T3 provides a label of low quality

To look into the effect of turker majority vote on accuracy, we had 28 turkers label each image

28 groups of 1:
We had 28 turkers
label each image:

28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:

28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:

28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
4 groups of 7:
3 groups of 9:

78.3%
83.8%
86.8%
86.6%
87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Average Image-level Accuracy (%)
Error bars: standard error
Image-Level Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels
Multiclass Accuracy

78.3%
83.8%
86.8%
86.6%
87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Multiclass Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels

78.3%
83.8%
86.8%
86.6%
87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Multiclass Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels
Accuracy saturates after 5 turkers

78.3%
83.8%
86.8%
86.6%
87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Multiclass Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels
Stderr: 0.2%
Stderr=0.2%

78.3%
83.8%
86.8%
86.6%
87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Multiclass Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels
Binary Accuracy

78.3%
83.8%
86.8%
86.6%
87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Multiclass Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels
Binary Accuracy
1 Label
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Problem

78.3%
83.8%
86.8%
86.6%
87.9%
80.6%
86.9%
89.7%
90.6%
90.2%
50%
60%
70%
80%
90%
100%
1 turker (N=28)
3 turkers (N=9)
5 turkers (N=5)
7 turkers (N=4)
9 turkers (N=3)
Multiclass Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 Labels
Binary Accuracy
1 Label
Problem

81.2%
85.8%
88.1%
89.3%
91.8%
92.7%
90.7%
50%
60%
70%
80%
90%
100%
EVALUATING QUALITY CONTROL MECHANISMS
Image-Level, Binary Classification
1 labeler
1 labeler,
3 verifiers
(majority vote)
1 labeler,
3 verifiers
(zero tolerance)
3 labelers
(majority vote)
3 labelers
(majority vote)
3 verifiers
(majority vote)
3 labelers
(majority vote)
3 verifiers
(zero tolerance)
5 labelers (majority vote)

81.2%
85.8%
88.1%
89.3%
91.8%
92.7%
90.7%
50%
60%
70%
80%
90%
100%
EVALUATING QUALITY CONTROL MECHANISMS
Image-Level, Binary Classification
1 labeler
1 labeler,
3 verifiers
(majority vote)
1 labeler,
3 verifiers
(zero tolerance)
3 labelers
(majority vote)
3 labelers
(majority vote)
3 verifiers
(majority vote)
3 labelers
(majority vote)
3 verifiers
(zero tolerance)
5 labelers (majority vote)
3 labelers + 3 verifiers = 93%

CHI’13 CONTRIBUTIONS:
1.Extended and reaffirmed findings from ASSETS’12 about viability of GSV and crowd work for locating and categorizing accessibility problems
2.Validated our ground truth labeling approach
3.Assessed simple quality control approaches

All of the approaches so far relied purely on manual labor, which limits scalability

Automatic Workflow Adaptation for Crowdsourcing
Lin et al. 2012; Dai et al. 2011 ; Kamar et al. 2012

Computer Vision & Streetview
Goodfellow et al., 2014; Chu et al., 2014; Naik et al., 2014

Tohme
遠目
Remote Eye
・

svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb Ramp Detection
svControl
Automatic Task Allocation
svVerify
Manual Label Verification
svLabel
Manual Labeling
Tohme
遠目
Remote Eye
・
Design Principles
1.Computer vision is cheap (zero cost)
2.Manual verification is far cheaper than manual labeling
3.Automatic curb ramp detection is hard and error prone
4.Fixing a false positive is easy, fixing a false negative is hard (requires manual labeling).

The “lack of curb cuts is a primary obstacle to the smooth integration of those with disabilities into the commerce of daily life.”
Kinney et al. vs. Yerusalim & Hoskins, 1993
3rd Circuit Court of Appeals

“Without curb cuts, people with ambulatory disabilities simply cannot navigate the city”
Kinney et al. vs. Yerusalim & Hoskins, 1993
3rd Circuit Court of Appeals

Dataset
svDetect
svCrawl
Web Scraper
Tohme
遠目
Remote Eye
・

svCrawl
Web Scraper
svDetect
Tohme
遠目
Remote Eye
・
Curb Ramp Detection on Street View image
False positives
False negatives = missed curb ramps

svCrawl
Web Scraper
Dataset
svDetect
svControl
Tohme
遠目
Remote Eye
・

svCrawl
Web Scraper
Dataset
svDetect
svControl
svVerify
Tohme
遠目
Remote Eye
・

svCrawl
Web Scraper
Dataset
svDetect
svControl Automatic Task Allocation
svVerify
Tohme
遠目
Remote Eye
・
svVerify can only fix false positives, not false negatives! That is, there is no way for a worker to add new labels at this stage!

svCrawl
Web Scraper
Dataset
svDetect
svControl
svVerify
svLabel
Manual Labeling
Tohme
遠目
Remote Eye
・

svCrawl
Web Scraper
Dataset
svDetect
svControl
svVerify
svLabel
Manual Labeling
Tohme
遠目
Remote Eye
・
.

svCrawl
Web Scraper
Dataset
svDetect
svControl Automatic Task Allocation
svVerify Manual Label Verification
svLabel
Manual Labeling
Tohme
遠目
Remote Eye
・

svCrawl
Web Scraper
Dataset
svDetect
svVerify
svLabel
Manual Labeling
Tohme
遠目
Remote Eye
・
Complexity:
Cardinality:
Depth:
CV:
0.14
0.33
0.21
0.22

svVerify
Tohme
遠目
Remote Eye
・
Complexity:
Cardinality:
Depth:
CV:
0.14
0.33
0.21
0.22
Predict presence of false negatives with linear SVM and Lasso regression
svCrawl
Web Scraper
Dataset
svLabel
Manual Labeling

svCrawl
Web Scraper
Dataset
svDetect
svVerify
Tohme
遠目
Remote Eye
・
Complexity:
Cardinality:
Depth:
CV:
0.82
0.25
0.96
0.54

Google Street View Intersection Panoramas and GIS Metadata
3D Point-cloud Data
Top-down Google Maps Imagery
Scraper & Dataset

Saskatoon
Los Angeles
Baltimore
Washington D.C.
Washington D.C.
Baltimore
Los Angeles
Saskatoon

Washington D.C.
Dense urban area
Semi-urban residential areas
Scraper & Dataset

Washington D.C.
Baltimore
Los Angeles
Saskatoon
* At the time of downloading data in summer 2013
Scraper & Dataset
Total Area: 11.3 km2
Intersections: 1,086
Curb Ramps: 2,877
Missing Curb Ramps: 647
Avg. GSV Data Age: 2.2 yrs

How well does GSV data reflect the current state of the physical world?

Washington D.C.
Baltimore
Physical Audit Areas
GSV and Physical World > 97.7% agreement
273 Intersections
Dataset | Validating Dataset
Small disagreement due to construction.

Washington D.C.
Baltimore
Physical Audit Areas
273 Intersections
> 97.7% agreement
Dataset
Key Takeaway
Google Street View is a viable source of curb ramp data

AUTOMATIC CURB RAMP DETECTION
1.Curb Ramp Detection
2.Post-Processing Output
3.SVM-Based Classification

Deformable Part Models
Felzenszwalb et al. 2008
http://www.cs.berkeley.edu/~rbg/latent/

Deformable Part Models
Felzenszwalb et al. 2008
http://www.cs.berkeley.edu/~rbg/latent/
Root filter
Parts filter
Displacement cost

Multiple redundant detection boxes
Detected Labels Stage 1: Deformable Part Model
Correct
1
False Positive
12
Miss
0

Curb ramps shouldn’t be in the sky or on roofs
Correct
1
False Positive
12
Miss
0

Detected Labels Stage 2: Post-processing

Detected Labels Stage 3: SVM-based Refinement
Filter out labels based on their size, color, and position.
Correct
1
False Positive
5
Miss
0

Correct
1
False Positive
3
Miss
0

Correct
6
False Positive
11
Miss
1

Correct
6
False Positive
6
Miss
1
Detected Labels Stage 2: Post-processing

Correct
6
False Positive
4
Miss
1

Some curb ramps never get detected
False positive detections
Correct
6
False Positive
4
Miss
1

Some curb ramps never get detected
False positive detections
Correct
6
False Positive
4
Miss
1
These false negatives are expensive to correct!

Used two-fold cross validation to evaluate CV sub-system

0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Precision (%)
Recall (%)
COMPUTER VISION SUB-SYSTEM RESULTS
Precision Higher, less false positives
Recall Higher, less false negatives

0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Precision (%)
Recall (%)
Goal: maximize area under curve

0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Precision (%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM

0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Precision (%)
Recall (%)
Stage 1: DPM
Stage 3: SVM
More than 20% of curb ramps were missed

0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Precision (%)
Recall (%)
Stage 1: DPM
Stage 3: SVM
Confidence threshold of -0.99, which results in 26% precision and 67% recall

Occlusion
Illumination
Scale
Viewpoint Variation
Structures Similar to Curb Ramps
Curb Ramp Design Variation
CURB RAMP DETECTION IS A HARD PROBLEM

Can we predict difficult intersections & CV performance?

Automatic Task Allocation | Features to Assess Scene Difficulty for CV
Number of connected streets from metadata
Depth information for intersection complexity analysis
Top-down images to assess complexity of an intersection
Number of detections and confidence values

3x
Manual Labeling | Labeling Interface

Automatic Detection and Manual Verification
Can Tohme achieve equivalent or better accuracy at a lower time cost compared to a completely manual approach?

STUDY METHOD: CONDITIONS
Manual labeling without smart task allocation
&
vs.
CV + Verification without smart task allocation
Tohme
遠目
Remote Eye
・
vs.
Evaluation

Accuracy
Task Completion Time
Evaluation
STUDY METHOD: MEASURES

Recruited workers from Mturk
Used 1,046 GSV images (40 used for golden insertion)
Evaluation
STUDY METHOD: APPROACH

RESULTS
Labeling Tasks
Verification Tasks
# of distinct turkers:
242
161
1,270
582
# of HITs completed:
# of tasks completed:
6,350
4,820
# of tasks allocated:
769
277
Evaluation
We used Monte Carlo simulations for evaluation

84%
88%
86%
0%
20%
40%
60%
80%
100%
Accuracy Measures (%)
Precision
Recall
F-measure
Manual Labeling
CV and Manual Verification
&
94
0
20
40
60
80
100
Task Completion Time / Scene (s)
Manual Labeling
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY
COST (TIME)

84%
68%
88%
58%
86%
63%
0%
20%
40%
60%
80%
100%
Precision
Recall
F-measure
Manual Labeling
&
94
42
0
20
40
60
80
100
Manual Labeling
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
ACCURACY
COST (TIME)

84%
68%
83%
88%
58%
86%
86%
63%
84%
0%
20%
40%
60%
80%
100%
Precision
Recall
F-measure
Manual Labeling
&
94
42
81
0
20
40
60
80
100
Manual Labeling
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
ACCURACY
COST (TIME)

84%
68%
83%
88%
58%
86%
86%
63%
84%
0%
20%
40%
60%
80%
100%
Precision
Recall
F-measure
Manual Labeling
&
94
42
81
0
20
40
60
80
100
Manual Labeling
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
13% reduction in cost
ACCURACY
COST (TIME)

svControl
svVerify
svLabel
Manual Labeling
Evaluation | Smart Task Allocator
~80% of svVerify tasks were correctly routed
~50% of svLabel tasks were correctly routed

svControl
svVerify
svLabel
Manual Labeling
Evaluation | Smart Task Allocator
If svControl worked perfectly, Tohme’s cost would drop to 28% of a manually labelling approach alone.

Example Labels from Manual Labeling

Evaluation | Example Labels from Manual Labeling

This is a driveway. Not a curb ramp.
Evaluation | Example Labels from Manual Labeling

Examples Labels from CV + Verification

Raw Street View Image
Evaluation | Example Labels from CV + Verification

False detection
Automatic Detection

Automatic Detection + Human Verification

Automatic Detection

False verification
Automatic Detection + Human Verification

UIST’14 CONTRIBUTIONS:
1.First CV system for automatically detecting curb ramps in images
2.Showed that automated methods could be used to improve labeling efficiency for curb ramps
3.Validated GSV as a viable curb ramp dataset

TOWARDS SCALABLE ACCESSIBILITY DATA COLLECTION
ASSETS’12 Poster
HCIC’13 Workshop
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13
ASSETS’13
UIST’14
The Future

8,209
Intersections in DC
BACK OF THE ENVELOPE CALCULATIONS
Manually labeling GSV with our custom interfaces would take 214 hours With Tohme, this drops to 184 hours We think we can do better  Unclear how long a physical audit would take

FUTURE WORK: COMPUTER VISION
Context integration & scene understanding 3D-data integration Improve training & sample size Mensuration

FUTURE WORK: FASTER LABELING & VERIFICATION INTERFACES

FUTURE WORK: TRACK PHYSICAL ACCESSIBILITY CHANGES OVER TIME

FUTURE WORK: ADDITIONAL SURVEYING TECHNIQUES
Transmits real-time imagery of physical space along with measurements

THE CROWD-POWERED STREETVIEW ACCESSIBILITY TEAM!
Kotaro Hara
Jin Sun
Victoria Le
Robert Moore
Sean Pannella
Jonah Chazan
David Jacobs
Jon Froehlich
Zachary Lawrence
Graduate Student
Undergraduate
High School
Professor

Flickr User: Pedro Rocha
https://www.flickr.com/photos/pedrorocha/3627562740/
Flickr User: Brooke Hoyer
https://www.flickr.com/photos/brookehoyer/14816521847/
Flickr User: Jen Rossey
https://www.flickr.com/photos/jenrossey/3185264564/
Flickr User: Steven Vance https://www.flickr.com/photos/jamesbondsv/8642938765
Flickr User: Jorge Gonzalez
https://www.flickr.com/photos/macabrephotographer/6225178809/
Flickr User: Mike Fraser https://www.flickr.com/photos/67588280@N00/10800029263//
PHOTO CREDITS
Flickr User: Susan Sermoneta
https://www.flickr.com/photos/en321/344387583/

This work is supported by:
Faculty Research Award
Human Computer
Interaction
Laboratory
makeability lab

Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning

More Related Content

Similar to Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning (20)

More from Jon Froehlich (7)

Recently uploaded (20)

Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning