Crowdsource Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study

What is crowdsourcing?
Crowdsourced damage mapping
2015 Nepal earthquake
Research on collaborative/crowdsourced damage mapping
Crowdsourced Damage Mapping for Disaster
Emergency Response - the 2015 Nepal Earthquake
Case Study
United Nations/India Workshop on the Use of Earth Observation Data
in Disaster Management and Risk Reduction: Sharing the Asian
Experience, 8-10 March 2016
Michal Bodn´ar
Beihang University
March 16, 2016
Michal Bodn´ar Crowdsourced Damage Mapping for Disaster Emergency Response - the 201

Table of Contents
1 What is crowdsourcing?
2 Crowdsourced damage mapping
Remote sensing based damage assessment
History of crowdsourced damage mapping
3 2015 Nepal earthquake
Tomnod campaign
4 Research on collaborative/crowdsourced damage mapping

Deﬁnition
”A distributed problem solving and production model. Problems are
broadcast to an unknown group of solvers in the form of an open call for
solutions. Users – also known as the crowd – typically form into online
communities, and crowd submits solutions. The crowd also sorts from
the solutions, ﬁnding the best ones”.
Chilton, 2010

Famous crowdsourcing projects
Wikipediaa
Zooniverseb
OpenStreetMapc
many others ...
ahttps://www.wikipedia.org/
bhttps://www.zooniverse.org
chttps://www.openstreetmap.org

User-Generated (Spatial) Content
Source: Michal Bodnar

Crowdsourcing
Poblet et al., 2014 divided crowdsourcing activities into 4 types:
Types of crowdsourcing
1 Crowd as a sensor - passive, raw data (data from mobile devices)
2 Crowd as a social computer - passive, unstructured data (data from
social media, such as Twitter, Facebook)
3 Crowd as a reporter - active, semi-structured data (social media with
the purpose of informing, such as Ushahidi)
4 Crowd as microtasker - active, structured data, using special tools
(such as HOT, Tomnod, SBTF, ...)

Crowdsourcing pyramid
Figure: Crowdsourcing pyramid (Poblet, 2014)

Table of Contents
Tomnod campaign

rapid damage assessment is one of the key parts of the emergency
response stage for majority of the disasters (earthquake, typhoon,
hurricane, ...) (Boccardo and Tonolo, 2013)
conducting rapid damage assessment by analyzing (very)
high-resolution satellite imagery has been very much used in recent
years, with 2010 Haiti earthquake regarded as a real expansion in
using such sources of information
in such cases, method of visual image inspection has been proved as
the most accurate one over the (semi-)automated methods (Dong
and Shan, 2013; Barrington et al., 2011; Voight et al., 2011; Chini,
2009; Lemoine et al. 2013)
the timeliness of such derived product is the most important factor
(Lemoine et al., 2013; Voight et al., 2011; Chini, 2009; Barrington
et al., 2011)

in cases, when the extent of the damage is high, remote sensing
experts are not capable of analyzing the satellite imagery solely by
themselves in a timely manner ⇒ approach of crowdsourcing has
been used
employing ”crowd as a microtasker” strategy
consisting of 3 parts (Barrington et al. 2011):
1 dividing the task into manageable components (microtasking)
2 motivating a large user base to contribute (crowdsourcing)
3 combining all responses of various quality into a complete solution
(consensus)

Source: https://irevolution.ﬁles.wordpress.com/2014/05/screen-shot-2014-05-29-at-6-30-27-am.png

Image source: http://ﬁles.umwblogs.org/blogs.dir3114ﬁles201304MadsNissen Rampen144.jpg

Image source: http://christianals.com/wp/wp-content/uploads/2012/05/Haiti001.jpg

Image source: http://inapcache.boston.com/universal/site graphics/blogs/bigpicture/new zealand quake/bp36.jpg

Tomnod campaign
Table of Contents
Tomnod campaign

Tomnod campaign
Nepal earthquake 2015
Situation overview
April 25, 2016, 11:56 am, 7.6 magnitude earthquake as recorded by
Nepal’s Seismological Centre (NSC)
76 km northwest of the capital city of Kathmandu
followed by more than 300 aftershocks greater than magnitude 4.0
(as of June 7, 2015), out of which four aftershocks were greater
than 6.0 (Government of Nepal, 2015)
31 out of 75 districts were aﬀected, out of which 14 were declared
’crisis-hit’

Tomnod campaign
Source: Michal BodnarMichal Bodn´ar Crowdsourced Damage Mapping for Disaster Emergency Response - the 201

Tomnod campaign
Nepal earthquake 2015 - Damage assessment

Tomnod campaign
Tomnod
a crowdsourcing project which aims to engage public to help with
searching through the satellite imagery
mission: to utilize power of crowdsourcing to identify objects and
places in satellite images
belongs to Digital Globe company, which also provides the very
high-resolution imagery
their campaigns have wide range of focus (humanitarian crises,
disasters, mapping basic infrastructure)

Tomnod campaign
HOT (Humanitarian OpenStreetMap Team)
evolved from OSM, ”the Wikipedia of Maps”
mission: to apply the principle of open data sharing to humanitarian
response and economic development
it was formed during (informally) and then after 2010 Haiti
earthquake
free for everyone to use, but requires a sign up process and a bit of
experience as well

Tomnod campaign
Tomnod vs HOT

Tomnod campaign
Tomnod campaign
the campaign started on April 27
four diﬀerent features were mapped:
1 Damaged building
2 Damaged road
3 Major destruction
4 Tent/shelter
the users were asked to tag these features by comparing pre- and
post- event imagery

Tomnod campaign

Tomnod campaign
Tomnod image coverage

Tomnod campaign
Tomnod campaign
I have analyzed Tomnod data in these ways:
1 using intrinsic methods
by looking at its geographical distribution
by investigating the dataset’s attributes (agreement and score)
by looking at the taggers and their statistics
2 by comparing against the oﬃcial data from
UNOSAT/NGA/Copernicus
the data for the analysis was emailed by the Tomnod team and
contained the tagged features until May 3
for my analysis, I have worked with vector data of both Tomnod and
UNOSAT/NGA/Copernicus

Tomnod campaign
Tomnod - Number of tags per class (as of May 3)

Tomnod campaign
Tomnod - Geographical distribution
Nepal has 5 levels of administrative units1
Nepal ADMIN units
1 ADMIN 1 level - 5 Development regions
Far-Western
Mid-Western
Western
Central
Eastern
2 ADMIN 2 level - 14 Zones
3 ADMIN 3 level - 75 Districts
4 ADMIN 4 level - 3157 Village development committees
5 ADMIN 5 level units
1https://en.wikipedia.org/wiki/Administrative_divisions_of_Nepal

Tomnod campaign
Distribution per ADMIN 1 region [%]
Region
Damaged
building
Damaged road
Major
destruction
Tent/shelter
Eastern 0.1 0 0.07 0
Central 90.37 50 70.31 93.87
Western 7.09 29.27 6.03 6.13
Mid-
Western
0 0 0 0
Far-
Western
0 0 0 0
out of
Nepal
2.44 20.73 23.59 0
Source: Michal Bodnar.

Tomnod campaign
Distribution per ADMIN 2 region [%]
Region Zone
Damaged
building
Damaged
road
Major
destruction
Ten-
t/shelter
Central Bagmati 76.9 30.49 30.19 93.87
Janakpur 8.72 13.41 26.66 0
Narayani 4.74 8.54 6.72 0
Western Dhaulagiri 0 0 0 0
Gandaki 6.76 25.61 5.93 5.94
Lumbini 0.34 3.66 0.01 0.19

Tomnod campaign
Distribution per Bagmati zone [number of features]
District
Damaged
building
Damaged
road
Major
destruction
Tent/shelter Total
Bhaktapur 70 0 23 0 93
Dhading 79 1 241 11 332
Kabhrepalanchok 86 4 20 0 110
Kathmandu 177 2 61 308 548
Lalitpur 149 7 19 144 319
Nuwakot 967 7 353 27 1354
Rasuwa 38 0 6 0 44
Sindhupalchok 39 2 184 0 225

Tomnod campaign
Distribution per Janakpur zone [number of features]
District
Damaged
building
Damaged
road
Major
destruction
Tent/shelter Total
Dhanusha 2 1 24 0 27
Mahottari 8 3 391 0 402
Sarlahi 147 3 383 0 553
Sindhuti 24 4 3 0 31
Ramechhap 1 0 0 0 1
Dolakha 0 0 0 0 0

Tomnod campaign
Top 5 ADMIN 3 districts per tag type
Posi-
tion
Damaged
building
Damaged
road
Major
destruction
Tent/shelter
1 Nuwakot Manang Chitawan Kathmandu
2 Kathmandu Nuwakot Mahottari Lalitpur
3 Lalitpur Gorkha Sarlahi Gorkha
4 Sarlahi Lalitpur Nuwakot Nuwakot
5 Kabhrapalnchok Chitawan Dhading Dhading

Tomnod campaign
First observations
ADMIN 1 - in total, 80 % of all the features were within Central
Development Region
ADMIN 2- 53 % of all the features were tagged in Bagmati zone,
followed by 17,5 % features within Janakpur region
ADMIN 3 - Nuwakot district (Bagmati zone) contained 23,8 % of all
the features tagged, followed by Sarlahi (Janakpur zone) and
Kathmandu (Bagmati zone) districts
Gandaki zone (epicentre of the earthquake) had a very few features
classiﬁed compared to Bagmati zone
there were 21 % and 23,5 % of Damaged road and Major
destruction features, which fell into area of out Nepal, respectively

Tomnod campaign
Tomnod campaign dataset
Dataset attributes
FID - a unique identiﬁer
type - the type of the tagged feature
tagger id - an ID of the user who placed a tag
score - the conﬁdence score calculated by Tomnod’s CrowdRank
algorithm for each feature
agreement - the number of people who placed a tag over a same
feature

Tomnod campaign
Tomnod - agreement attribute
minimum 1
maximum 43
mean 7.39
standard deviation 4.97
Source: ArcGIS 10.2, Michal Bodnar

Tomnod campaign
Tomnod - Frequency of agreement values per class

Tomnod campaign
Tomnod - Statistics of agreement values per class
Damaged
building
Damaged road
Major
destruction
Tent/shelter
minimum 1 1 1 2
maximum 42 19 43 12
mean 7.05 4.95 8.27 4.09
standard
deviation
4.46 4.41 5.39 1.62
Source:ArcGIS 10.2, Michal Bodnar.

Tomnod campaign
Tomnod - score attribute
minimum 0.85011
maximum 1
mean 0.973867

Tomnod campaign
Tomnod - Frequency of score values per class

Tomnod campaign
Tomnod - Statistics of score values per class
Damaged
building
Damaged road
Major
destruction
Tent/shelter
minimum 0.850111 0.855924 0.850126 0.851148
maximum 1 1 1 1
mean 0.971581 0.976194 0.97762 0.961046
standard
deviation
0.041003 0.041529 0.037503 0.041183
Source: ArcGIS 10.2, Michal Bodnar.

Tomnod campaign
Tomnod - Statistics about taggers
42,5 % of taggers tagged only 1 feature
77,85 % of taggers tagged 5 or less features
11,86 % of taggers tagged 10 or more features
Taggers and tags
number of taggers 1141
minimum number of tags 1
maximum number of tags 117
mean number of tags 4.99

Tomnod campaign
Tomnod - Statistics about taggers

Tomnod campaign
Tomnod - top 10 taggers
Top 10 taggers tagged together for almost 13 % of all the tagged
features.
Top 10 taggers of Tomnod campaign
Rank
ID
No. of
tags
average
score
average
agreement
tagged
1 3234567 117 0.961266 3.6 MD2
, T/S
2 92197 97 0.97919 8.39 MD, DB, T/S
3 14927859 90 0.959876 5.42 MD, DB, T/S
4 584393 72 0.990796 9.78 MD
5 882027 66 0.969293 3.27 T/S
6 713428 63 0.977455 7.57 MD, DB
7 4809316 57 0.966001 6.96 DB
8 14877422 52 0.977467 7.75 MD, DB
9 3922680 52 0.984814 6.90 MD, DB
10 14900490 49 0.978841 6.81 MD, DB
2Major destruction

Tomnod campaign
Comparison against oﬃcial datasets
What was compared?
1 Tomnod Damaged building vs. UNOSAT-Copernicus building
damage
2 Tomnod Major destruction vs. NGA damaged zones
3 Tomnod Tent/shelter vs. NGA IDP Camps
Methodology
1 the total number of items on district level was compared
2 the completeness (error of ommission and commission were
computed)

Tomnod campaign
Comparison against official datasets
Limitations
the data from official institutions do not necessarily have to be the
most precise one
the data from official institutions can also lack completeness (most
probably do)
the date of the release of the datasets differed from each other
the underlying imagery used for various assessments was also not the
same

Tomnod campaign
Comparison no.1 - Tomnod Damaged building vs
UNOSAT-Copernicus
Source: ArcGIS 10.2, Michal BodnarMichal Bodn´ar Crowdsourced Damage Mapping for Disaster Emergency Response - the 201

Tomnod campaign
UNOSAT-Copernicus
Tomnod UNOSAT-Copernicus
Date of dataset May 3 May 7
Number of features 2087 6634
Classiﬁcation Damaged Destroyed
Severe damage
Moderate damage
Data type Point Point

Tomnod campaign
UNOSAT-Copernicus

Tomnod campaign
Comparison no.2 - Tomnod Major destruction vs NGA

Tomnod campaign
Tomnod NGA
Date of dataset May 3 May 7
Classiﬁcation Damaged Destroyed
Severe damage
Moderate damage
Possible damage
Data type Point Polygon

Tomnod campaign

Tomnod campaign
Comparison no.3 - Tomnod Tent/shelter vs NGA

Tomnod campaign
Tomnod NGA
Date of dataset May 3 April 29
Data type Point Polygon

Tomnod campaign

Tomnod campaign
Conclusion from analyzing Tomnod data
Major destruction feature had the highest number of features
Damaged road feature did not cause enough attention from
volunteers and counted together only 82 features tagged
in general it seems like volunteers can be quite good at spotting the
features of the same shape and material, such was the case of
Tent/shelter feature
the most challenging seems to be the building damage assessment
surprisingly, a high proportion of Major destruction features was
tagged outside of Nepal, which would not have a use for ground
responders
Gandaki zone, where the earthquake struck, had really low
proportion of tagged features compared to Bagmati zone, where
capital city of Kathmandu is located

Tomnod campaign
Conclusion from analyzing Tomnod data
the average agreement value was 7.3 and the overall distribution
proved have a long tail tendency
in total there were 1141 taggers for 5695 features with the
maximum number of 117 tags for a volunteer
43 % of volunteers tagged only 1 feature, while only 12 % of them
tagged 10 or more
the next analysis could look at:
the quality level of individual taggers
correlation between agreement or score values and the accuracy
the comparison against the ground truth data

Tomnod campaign
My opinion on crowdsourced damage assessment
generally, it is a great way how to engage people in both working for
good cause and recognizing a power of satellite imagery
due to importance of timeliness, it can become a good source of
information in the emergency response stage
it can serve as a good indication of the signalizing the hotspots of
where damage was occured
the quality of collaborative damage mapping still remains
questionable, but we need to learn which features the volunteers are
best at classifying

Quality of crowdsourced damage assessment
assessing the quality by comparison against ground truth is usually
conducted a long time after the disaster, once the ﬁeld survey was
conducted
such approach is however not helpful to be used during the
emergency response stage
it is important to look at the quality of the crowdsourced damage
assessment using intrinsic data measures and assess its
ﬁtness-for-purpose
for crowdsourced damage assessment, this task is really hard one as
such process is an output of three variables:
1 Volunteers
2 Image
3 Damage

Factors inﬂuencing the quality of crowdsourced damage
mapping

First steps - GEOCAN accuracy model
Foulser-Piggott et al.,2013 proposed a qualitative multi-attribute
decision model as a ﬁrst approach to assess quality of the
crowdsourced damage based on the multiple factors
to each of the attributes, they assigned a certain weight
this weight assessment was based solely on the experience and
assumptions of the authors, but not by performing any kind of
experimental analysis

Enhanced decision model - experimental study
in the previous model, various factors are omitted, especially
demographic characteristics of the volunteers
there will be an experimental study performed at Beihang university
on a certain number of volunteers, who will be studied over a certain
period of time over a series of events
volunteers will be chosen in a way they would provide a sample of
the population which is usually employed in such events (different
level of experience, age, nationality, gender)
each event, they will be asked to perform a damage assessment of
the area for a certain time by analyzing different type of imagery
(airborne, satellite)
the results will be then used to compute the relative influence of
each of the attributes discussed on the overall quality and to build a
multi-criteria qualitative decision model

Thank You for Your attention!

References
[1] Goodchild, M.F. Citizens as sensors: the world of volunteered geography [J].
Geojournal , V69 (4), 2007: 211-221
[2] Dong L., Shan J. A comprehensive review of earthquake-induced building damage
detection with remote sensing techniques [J]. ISPRS Journal of Photogrammetry and
Remote Sensing, V84, 2013: 85-99
[3] Poblet M., Garc´ıa-Cuesta E., Casanovas P. IT Enabled Crowds: Leveraging the
Geomobile Revolution for Disaster Management [C]. Proceedings of the Sintelnet
WG5 Workshop on Crowd Intelligence: Foundations, Methods and Practices: 2014
[4] Goodchild M.F., Linna Li. Assuring the quality of volunteered geographic
information [J]. Spatial Statistics, V1, 2012: 110 - 120
[5] Foulser-Piggott R., Spence R. and Brown D. The use of remote sensing for
building damage assessment following 22nd February 2011 Christchurch earthquake:
the GEOCAN study and its validation [R]. Cambridge Architectural Research Limited,
2013: 1-45
[6] Barron Ch., Neis P., Zipf A. A Comprehensive Framework for Intrinsic
OpenStreetMap Quality Analysis [J]. Transactions in GIS, V18 (6), 2014: 877-895
[7] Pourabdollah A., Morley J., Feldman S., Jackson M. Towards and Authoritative
OpenStreetMap: Conﬂating OSM and OS OpenData National Maps’ Road Network
[J]. ISPRS International Journal of Geo-Information, V2, 2013: 704-728

References
[8] Chilton S. Crowdsourcing is radically changing the geodata landscape: case study
of OpenStreetMap [G].
[9] Barrington L. et al. Crowdsourcing earthquake damage assessment using remote
sensing imagery [J]. Annals of Geophysics, 54, 6, 2011: 680-687.
[10] Lemoine G., Corbane C., Louvrier C., Kauﬀmann M. Intercomparison and
validation of building damage assessments based on post-Haiti 2010 earthquake
imagery using multi-source reference data [J]. Natural Hazards and Earth System
Sciences Discussion: 1, 2013: 1445 - 1486.
[11] Government of Nepal, National Planning Commission. Nepal Earthquake 2015:
Post Disaster Needs Assessment [R], 2015: 1-20.
[12] Voigt S., Schneiderhan T., Twele A., Gahler M., Stein E. and Mehl H. Rapid
Damage Assessment and Situation Mapping: Learning from the 2010 Haiti Earthquake
[J]. Photogrammetric Engineering & Remote Sensing, V77 (9), 2011: 923-931
[13] Chini M. Earthquake damage mapping techniques using SAR and Optical remote
sensing satellite data [J]. Advances in Geoscience and Remote Sensing, 2009: 269-279
[14] Boccardo O., Giulio Tonolo F. Haiti earthquake damage assessment: review of the
remote sensing role [J]. International Archives of the Photogrammetry, Remote
Sensing and Spatial Information Sciences, 2012, V39-B4: 529-532

References
[15] Haklay M. How good is volunteered geographical information? A comparative
study of OpenStreetMap and Ordnance Survey datasets [J]. Environment and
Planning B: Planning and Design, V37, 2010: 682-703
[16] Helbich M., Amelunxen Ch., Neis P., Zipf A. Comparative Spatial Analysis of
Positional Accuracy of OpenStreetMap and Proprietary Geodata [C]. GI forum 2012:
Geovisualization, Society and Learning, 2012: 24-33.
[17] Jackson P.S., Mullen W., Agouris P., Crooks A., Croitoru A., Stefanidis A.
Assessing Completeness and Spatial Error of Features in Volunteered Geographic
Information [J]. ISPRS International Journal of Geo-Information, V2, 2013: 507-530
[18] Haklay M., Basiouka S., Antoniou V., Ather A. How Many Volunteer Does It
Take To Map An Area Well? The validity of Linus’ law to Volunteered Geographic
Information[J]. The Cartographic Journal, V47 (4), 2010: 315-332
[19] Barron Ch., Neis P., Zipf A. A comprehensive Framework for Intrinsic
OpenStreetMap Quality Analysis [J]. Transactions in GIS, V18 (6), 2014: 877-895.

Crowdsource Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study

More Related Content

Viewers also liked (20)

Similar to Crowdsource Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study (20)

Recently uploaded (20)

Crowdsource Damage Mapping for Disaster Emergency Response - the 2015 Nepal Earthquake Case Study