SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity
Eneko Agirre Daniel Cer
University of the Basque Country Stanford University
Donostia, 20018, Basque Country Stanford, CA 94305, USA
e.agirre@ehu.es danielcer@stanford.edu

Mona Diab Aitor Gonzalez-Agirre
Center for Computational Learning Systems University of the Basque Country
Columbia University Donostia, 20018, Basque Country
mdiab@ccls.columbia.edu agonzalez278@ikasle.ehu.es

Abstract equivalence is directional, e.g. a car is a vehicle, but
a vehicle is not necessarily a car. Additionally, STS
Semantic Textual Similarity (STS) measures differs from both TE and PARA in that, rather than
the degree of semantic equivalence between
being a binary yes/no decision (e.g. a vehicle is not a
two texts. This paper presents the results of
the STS pilot task in Semeval. The training
car), STS incorporates the notion of graded semantic
data contained 2000 sentence pairs from pre- similarity (e.g. a vehicle and a car are more similar
viously existing paraphrase datasets and ma- than a wave and a car).
chine translation evaluation resources. The STS provides a unified framework that allows for
test data also comprised 2000 sentences pairs an extrinsic evaluation of multiple semantic compo-
for those datasets, plus two surprise datasets nents that otherwise have tended to be evaluated in-
with 400 pairs from a different machine trans-
dependently and without broad characterization of
lation evaluation corpus and 750 pairs from a
lexical resource mapping exercise. The sim-
their impact on NLP applications. Such components
ilarity of pairs of sentences was rated on a include word sense disambiguation and induction,
0-5 scale (low to high similarity) by human lexical substitution, semantic role labeling, multi-
judges using Amazon Mechanical Turk, with word expression detection and handling, anaphora
high Pearson correlation scores, around 90%. and coreference resolution, time and date resolution,
35 teams participated in the task, submitting named-entity handling, underspecification, hedging,
88 runs. The best results scored a Pearson semantic scoping and discourse analysis. Though
correlation >80%, well above a simple lexical
not in the scope of the current pilot task, we plan to
baseline that only scored a 31% correlation.
This pilot task opens an exciting way ahead, explore building an open source toolkit for integrat-
although there are still open issues, specially ing and applying diverse linguistic analysis modules
the evaluation metric. to the STS task.
While the characterization of STS is still prelim-
inary, we observed that there was no comparable
1 Introduction
existing dataset extensively annotated for pairwise
Semantic Textual Similarity (STS) measures the semantic sentence similarity. We approached the
degree of semantic equivalence between two sen- construction of the first STS dataset with the fol-
tences. STS is related to both Textual Entailment lowing goals: (1) To set a definition of STS as a
(TE) and Paraphrase (PARA). STS is more directly graded notion which can be easily communicated to
applicable in a number of NLP tasks than TE and non-expert annotators beyond the likert-scale; (2) To
PARA such as Machine Translation and evaluation, gather a substantial amount of sentence pairs from
Summarization, Machine Reading, Deep Question diverse datasets, and to annotate them with high
Answering, etc. STS differs from TE in as much as quality; (3) To explore evaluation measures for STS;
it assumes symmetric graded equivalence between (4) To explore the relation of STS to PARA and Ma-
the pair of textual snippets. In the case of TE the chine Translation Evaluation exercises.

385

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 385–393,
Montr´ al, Canada, June 7-8, 2012. c 2012 Association for Computational Linguistics
e

In the next section we present the various sources length of the hypothesis was typically much shorter
of the STS data and the annotation procedure used. than the text, and we did not want to bias the STS
Section 4 investigates the evaluation of STS sys- task in this respect. We may, however, explore using
tems. Section 5 summarizes the resources and tools TE pairs for STS in the future.
used by participant systems. Finally, Section 6 Microsoft Research (MSR) has pioneered the ac-
draws the conclusions. quisition of paraphrases with two manually anno-
tated datasets. The first, called MSR Paraphrase
2 Source Datasets (MSRpar for short) has been widely used to evaluate
Datasets for STS are scarce. Existing datasets in- text similarity algorithms. It contains 5801 pairs of
clude (Li et al., 2006) and (Lee et al., 2005). The sentences gleaned over a period of 18 months from
first dataset includes 65 sentence pairs which cor- thousands of news sources on the web (Dolan et
respond to the dictionary definitions for the 65 al., 2004). 67% of the pairs were tagged as para-
word pairs in Similarity(Rubenstein and Goode- phrases. The inter annotator agreement is between
nough, 1965). The authors asked human informants 82% and 84%. Complete meaning equivalence is
to assess the meaning of the sentence pairs on a not required, and the annotation guidelines allowed
scale from 0.0 (minimum similarity) to 4.0 (maxi- for some relaxation. The pairs which were anno-
mum similarity). While the dataset is very relevant tated as not being paraphrases ranged from com-
to STS, it is too small to train, develop and test typ- pletely unrelated semantically, to partially overlap-
ical machine learning based systems. The second ping, to those that were almost-but-not-quite seman-
dataset comprises 50 documents on news, ranging tically equivalent. In this sense our graded annota-
from 51 to 126 words. Subjects were asked to judge tions enrich the dataset with more nuanced tags, as
the similarity of document pairs on a five-point scale we will see in the following section. We followed
(with 1.0 indicating “highly unrelated” and 5.0 indi- the original split of 70% for training and 30% for
cating “highly related”). This second dataset com- testing. A sample pair from the dataset follows:
prises a larger number of document pairs, but it goes
The Senate Select Committee on Intelligence
beyond sentence similarity into textual similarity. is preparing a blistering report on prewar
When constructing our datasets, gathering natu- intelligence on Iraq.
rally occurring pairs of sentences with different de-
grees of semantic equivalence was a challenge in it- American intelligence leading up to the
self. If we took pairs of sentences at random, the war on Iraq will be criticized by a powerful
vast majority of them would be totally unrelated, and US Congressional committee due to report
soon, officials said today.
only a very small fragment would show some sort of
semantic equivalence. Accordingly, we investigated
In order to construct a dataset which would reflect
reusing a collection of existing datasets from tasks
a uniform distribution of similarity ranges, we sam-
that are related to STS.
pled the MSRpar dataset at certain ranks of string
We first studied the pairs of text from the Recog-
similarity. We used the implementation readily ac-
nizing TE challenge. The first editions of the chal-
cessible at CPAN1 of a well-known metric (Ukko-
lenge included pairs of sentences as the following:
nen, 1985). We sampled equal numbers of pairs
T: The Christian Science Monitor named a US from five bands of similarity in the [0.4 .. 0.8] range
journalist kidnapped in Iraq as freelancer Jill separately from the paraphrase and non-paraphrase
Carroll. pairs. We sampled 1500 pairs overall, which we split
H: Jill Carroll was abducted in Iraq. 50% for training and 50% for testing.
The first sentence is the text, and the second is The second dataset from MSR is the MSR Video
the hypothesis. The organizers of the challenge an- Paraphrase Corpus (MSRvid for short). The authors
notated several pairs with a binary tag, indicating showed brief video segments to Annotators from
whether the hypothesis could be entailed from the Amazon Mechanical Turk (AMT) and were asked
text. Although these pairs of text are interesting we 1
http://search.cpan.org/˜mlehmann/
decided to discard them from this pilot because the String-Similarity-1.04/Similarity.pm

386

Given the strong connection between STS sys-
tems and Machine Translation evaluation metrics,
we also sampled pairs of segments that had been
part of human evaluation exercises. Those pairs in-
cluded a reference translation and a automatic Ma-
chine Translation system submission, as follows:
The only instance in which no tax is levied is
when the supplier is in a non-EU country and
the recipient is in a Member State of the EU.
Figure 1: Video and corresponding descriptions from
The only case for which no tax is still
MSRvid
perceived ”is an example of supply in the
European Community from a third country.

We selected pairs from the translation shared task
of the 2007 and 2008 ACL Workshops on Statistical
Machine Translation (WMT) (Callison-Burch et al.,
2007; Callison-Burch et al., 2008). For consistency,
we only used French to English system submissions.
The training data includes all of the Europarl human
ranked fr-en system submissions from WMT 2007,
with each machine translation being paired with the
Figure 2: Definition and instructions for annotation
correct reference translation. This resulted in 729
unique training pairs. The test data is comprised of
to provide a one-sentence description of the main ac- all Europarl human evaluated fr-en pairs from WMT
tion or event in the video (Chen and Dolan, 2011). 2008 that contain 16 white space delimited tokens or
Nearly 120 thousand sentences were collected for less.
2000 videos. The sentences can be taken to be In addition, we selected two other datasets that
roughly parallel descriptions, and they included sen- were used as out-of-domain testing. One of them
tences for many languages. Figure 1 shows a video comprised of all the human ranked fr-en system
and corresponding descriptions. submissions from the WMT 2007 news conversa-
tion test set, resulting in 351 unique system refer-
The sampling procedure from this dataset is sim-
ence pairs.2 The second set is radically different as
ilar to that for MSRpar. We construct two bags of
it comprised 750 pairs of glosses from OntoNotes
data to draw samples. The first includes all possible
4.0 (Hovy et al., 2006) and WordNet 3.1 (Fellbaum,
pairs for the same video, and the second includes
1998) senses. The mapping of the senses of both re-
pairs taken from different videos. Note that not all
sources comprised 110K sense pairs. The similarity
sentences from the same video were equivalent, as
between the sense pairs was generated using simple
some descriptions were contradictory or unrelated.
word overlap. 50% of the pairs were sampled from
Conversely, not all sentences coming from different
senses which were deemed as equivalent senses, the
videos were necessarily unrelated, as many videos
rest from senses which did not map to one another.
were on similar topics. We took an equal number of
samples from each of these two sets, in an attempt to 3 Annotation
provide a balanced dataset between equivalent and
non-equivalent pairs. The sampling was also done In this first dataset we defined a straightforward lik-
according to string similarity, but in four bands in the ert scale ranging from 5 to 0, but we decided to pro-
[0.5 .. 0.8] range, as sentences from the same video vide definitions for each value in the scale (cf. Fig-
had a usually higher string similarity than those in ure 2). We first did pilot annotations of 200 pairs se-
the MSRpar dataset. We sampled 1500 pairs overall, 2
At the time of the shared task, this data set contained dupli-
which we split 50% for training and 50% for testing. cates resulting in 399 sentence pairs.

387

lected at random from the three main datasets in the dard file (and single system output). The first ver-
training set. We did the annotation, and the pairwise sion of the results were published using this method,
Pearson ranged from 84% to 87% among ourselves. but the overall score did not correspond well to the
The agreement of each annotator with the average individual scores in the datasets, and participants
scores of the other was between 87% and 89%. proposed two additional evaluation metrics, both of
In the future, we would like to explore whether them based on Pearson correlation. The organizers
the definitions improve the consistency of the tag- of the task decided that it was more informative, and
ging with respect to a likert scale without defini- on the benefit of the community, to also adopt those
tions. Note also that in the assessment of the qual- evaluation metrics, and the idea of having a single
ity and evaluation of the systems performances, we main evaluation metric was dropped. This decision
just took the resulting SS scores and their averages. was not without controversy, but the organizers gave
Using the qualitative descriptions for each score in more priority to openness and inclusiveness and to
analysis and evaluation is left for future work. the involvement of participants. The final result ta-
Given the good results of the pilot we decided to ble thus included three evaluation metrics. For the
deploy the task in Amazon Mechanical Turk (AMT) future we plan to analyze the evaluation metrics, in-
in order to crowd source the annotation task. The cluding non-parametric metrics like Spearman.
turkers were required to have achieved a 95% of ap-
4.1 Evaluation metrics
proval rating in their previous HITs, and had to pass
a qualification task which included 6 example pairs. The first evaluation metric is the Pearson correla-
Each HIT included 5 pairs of sentences, and was tion for the concatenation of all five datasets, as de-
paid at 0.20$ each. We collected 5 annotations per scribed above. We will use overall Pearson or sim-
HIT. In the latest data collection, each HIT required ply ALL to refer to this measure.
114.9 second for completion. The second evaluation metric normalizes the out-
In order to ensure the quality, we also performed put for each dataset separately, using the linear least
post-hoc validation. Each HIT contained one pair squares method. We concatenated the system results
from our pilot. After the tagging was completed for five datasets and then computed a single Pear-
we checked the correlation of each individual turker son correlation. Given Y = {yi } and X = {xi }
with our scores, and removed annotations of turkers (the gold standard scores and the system scores,
which had low correlations (below 50%). Given the respectively), we transform the system scores into
high quality of the annotations among the turkers, X = {xi } in order to minimize the squared error
2
we could alternatively use the correlation between i (yi − xi ) . The linear transformation is given by
the turkers itself to detect poor quality annotators. xi = xi ∗ β1 + β2 , where β1 and β2 are found an-
alytically. We refer to this measure as Normalized
4 Systems Evaluation Pearson or simply ALLnorm. This metric was sug-
gested by one of the participants, Sergio Jimenez.
Given two sentences, s1 and s2, an STS system The third evaluation metric is the weighted mean
would need to return a similarity score. Participants of the Pearson correlations on individual datasets.
can also provide a confidence score indicating their The Pearson returned for each dataset is weighted
confidence level for the result returned for each pair, according to the number of sentence pairs in that
but this confidence is not used for the main results. dataset. Given ri the five Pearson scores for
The output of the systems performance is evaluated each dataset, and ni the number of pairs in each
using the Pearson product-moment correlation co- dataset, the weighted mean is given as i=1..5 (ri ∗
efficient between the system scores and the human ni )/ i=1..5 ni We refer to this measure as weighted
scores, as customary in text similarity (Rubenstein mean of Pearson or Mean for short.
and Goodenough, 1965). We calculated Pearson for
each evaluation dataset separately. 4.2 Using confidence scores
In order to have a single Pearson measure for each Participants were allowed to include a confidence
system we concatenated the gold standard (and sys- score between 1 and 100 for each of their scores.
tem outputs) for all 5 datasets into a single gold stan- We used weighted Pearson to use those confidence

388

scores3 . Table 2 includes the list of systems which the 110K OntoNotes-WordNet dataset from which
provided a non-uniform confidence. The results the other organizers sampled the surprise data set.
show that some systems were able to improve their After the submission deadline expired, the orga-
correlation, showing promise for the usefulness of nizers published the gold standard in the task web-
confidence in applications. site, in order to ensure a transparent evaluation pro-
cess.
4.3 The Baseline System
We produced scores using a simple word overlap 4.5 Results
baseline system. We tokenized the input sentences Table 1 shows the results for each run in alphabetic
splitting at white spaces, and then represented each order. Each result is followed by the rank of the sys-
sentence as a vector in the multidimensional to- tem according to the given evaluation measure. To
ken space. Each dimension had 1 if the token was the right, the Pearson score for each dataset is given.
present in the sentence, 0 otherwise. Similarity of In boldface, the three best results in each column.
vectors was computed using cosine similarity. First of all we want to stress that the large majority
We also run a random baseline several times, of the systems are well above the simple baseline,
yielding close to 0 correlations in all datasets, as ex- although the baseline would rank 70 on the Mean
pected. We will refer to the random baseline again measure, improving over 19 runs.
in Section 4.5. The correlation for the non-MT datasets were re-
ally high: the highest correlation was obtained was
4.4 Participation
for MSRvid (0.88 r), followed by MSRpar (0.73 r)
Participants could send a maximum of three system and On-WN (0.73 r). The results for the MT evalu-
runs. After downloading the test datasets, they had ation data are lower, (0.57 r) for SMT-eur and (0.61
a maximum of 120 hours to upload the results. 35 r) for SMT-News. The simple token overlap base-
teams participated, submitting 88 system runs (cf. line, on the contrary, obtained the highest results
first column of Table 1). Due to lack of space we for On-WN (0.59 r), with (0.43 r) on MSRpar and
can’t detail the full names of authors and institutions (0.40 r) on MSRvid. The results for MT evaluation
that participated. The interested reader can use the data are also reversed, with (0.40 r) for SMT-eur and
name of the runs to find the relevant paper in these (0.45 r) for SMT-News.
proceedings. The ALLnorm measure yields the highest corre-
There were several issues in the submissions. The lations. This comes at no surprise, as it involves a
submission software did not ensure that the nam- normalization which transforms the system outputs
ing conventions were appropriately used, and this using the gold standard. In fact, a random base-
caused some submissions to be missed, and in two line which gets Pearson correlations close to 0 in all
cases the results were wrongly assigned. Some par- datasets would attain Pearson of 0.58914 .
ticipants returned Not-a-Number as a score, and the Although not included in the results table for lack
organizers had to request whether those where to be of space, we also performed an analysis of confi-
taken as a 0 or as a 5. dence intervals. For instance, the best run according
Finally, one team submitted past the 120 hour to ALL (r = .8239) has a 95% confidence interval of
deadline and some teams sent missing files after the [.8123,.8349] and the second a confidence interval
deadline. All those are explicitly marked in Table 1. of [.8016,.8254], meaning that the differences are
The teams that included one of the organizers are not statistically different.
also explicitly marked. We want to stress that in
these teams the organizers did not allow the devel- 5 Tools and resources used
opers of the system to access any data or informa-
tion which was not available for the rest of partic- The organizers asked participants to submit a de-
ipants. One exception is weiwei, as they generated scription file, special emphasis on the tools and re-
sources that they used. Table 3 shows in a simpli-
3
http://en.wikipedia.org/wiki/Pearson_
4
product-moment_correlation_coefficient# We run the random baseline 10 times. The mean is reported
Calculating_a_weighted_correlation here. The standard deviation is 0.0005

389

Run ALL Rank ALLnrm Rank Mean Rank MSRpar MSRvid SMT-eur On-WN SMT-news
00-baseline/task6-baseline .3110 87 .6732 85 .4356 70 .4334 .2996 .4542 .5864 .3908
aca08ls/task6-University Of Sheffield-Hybrid .6485 34 .8238 15 .6100 18 .5166 .8187 .4859 .6676 .4280
aca08ls/task6-University Of Sheffield-Machine Learning .7241 17 .8169 18 .5750 38 .5166 .8187 .4859 .6390 .2089
aca08ls/task6-University Of Sheffield-Vector Space .6054 48 .7946 44 .5943 27 .5460 .7241 .4858 .6676 .4280
acaputo/task6-UNIBA-DEPRI .6141 46 .8027 38 .5891 31 .4542 .7673 .5126 .6593 .4636
acaputo/task6-UNIBA-LSARI .6221 44 .8079 30 .5728 40 .3886 .7908 .4679 .6826 .4238
acaputo/task6-UNIBA-RI .6285 41 .7951 43 .5651 45 .4128 .7612 .4531 .6306 .4887
baer/task6-UKP-run1 .8117 4 .8559 4 .6708 4 .6821 .8708 .5118 .6649 .4672
baer/task6-UKP-run2 plus postprocessing smt twsi .8239 1 .8579 2 .6773 1 .6830 .8739 .5280 .6641 .4937
baer/task6-UKP-run3 plus random .7790 8 .8166 19 .4320 71 .6830 .8739 .5280 -.0620 -.0520
croce/task6-UNITOR-1 REGRESSION BEST FEATURES .7474 13 .8292 12 .6316 10 .5695 .8217 .5168 .6591 .4713
croce/task6-UNITOR-2 REGRESSION ALL FEATURES .7475 12 .8297 11 .6323 9 .5763 .8217 .5102 .6591 .4713
croce/task6-UNITOR-3 REGRESSION ALL FEATURES ALL DOMAINS .6289 40 .8150 21 .5939 28 .4686 .8027 .4574 .6591 .4713
csjxu/task6-PolyUCOMP-RUN1 .6528 31 .7642 59 .5492 51 .4728 .6593 .4835 .6196 .4290
danielcer/stanford fsa† .6354 38 .7212 70 .4848 66 .3795 .5350 .4377 .6052 .4164
danielcer/stanford pdaAll† .4229 77 .7160 72 .5044 62 .4409 .4698 .4558 .6468 .4769
danielcer/stanford rte† .5589 55 .7807 55 .4674 67 .4374 .8037 .3533 .3077 .3235
davide buscaldi/task6-IRIT-pg1 .4280 76 .7379 65 .5009 63 .4295 .6125 .4952 .5387 .3614
davide buscaldi/task6-IRIT-pg3 .4813 68 .7569 61 .5202 58 .4171 .6728 .5179 .5526 .3693
davide buscaldi/task6-IRIT-wu .4064 81 .7287 69 .4898 65 .4326 .5833 .4856 .5317 .3480
demetrios glinos/task6-ATA-BASE .3454 83 .6990 81 .2772 87 .1684 .6256 .2244 .1648 .0988
demetrios glinos/task6-ATA-CHNK .4976 64 .7160 73 .3215 86 .2312 .6595 .1504 .2735 .1426
demetrios glinos/task6-ATA-STAT .4165 79 .7129 75 .3312 85 .1887 .6482 .2769 .2950 .1336
desouza/task6-FBK-run1 .5633 54 .7127 76 .3628 82 .2494 .6117 .1495 .4212 .2439
dvilarinoayala/task6-BUAP-RUN-1 .4997 63 .7568 62 .5260 56 .4037 .6532 .4521 .6050 .4537
dvilarinoayala/task6-BUAP-RUN-2 -.0260 89 .5933 89 .1016 89 .1109 .0057 .0348 .1788 .1964
dvilarinoayala/task6-BUAP-RUN-3 .6630 25 .7474 64 .5105 59 .4018 .6378 .4758 .5691 .4057
enrique/task6-UNED-H34measures .4381 75 .7518 63 .5577 48 .5328 .5788 .4785 .6692 .4465
enrique/task6-UNED-HallMeasures .2791 88 .6694 87 .4286 72 .3861 .2570 .4086 .6006 .5305
enrique/task6-UNED-SP INIST .4680 69 .7625 60 .5615 47 .5166 .6303 .4625 .6442 .4753
georgiana dinu/task6-SAARLAND-ALIGN VSSIM .4952 65 .7871 50 .5065 60 .4043 .7718 .2686 .5721 .3505
georgiana dinu/task6-SAARLAND-MIXT VSSIM .4548 71 .8258 13 .5662 43 .6310 .8312 .1391 .5966 .3806
jan snajder/task6-takelab-simple .8133 3 .8635 1 .6753 2 .7343 .8803 .4771 .6797 .3989
jan snajder/task6-takelab-syntax .8138 2 .8569 3 .6601 5 .6985 .8620 .3612 .7049 .4683
janardhan/task6-janardhan-UNL matching .3431 84 .6878 84 .3481 83 .1936 .5504 .3755 .2888 .3387
jhasneha/task6-Penn-ELReg .6622 27 .8048 34 .5654 44 .5480 .7844 .3513 .6040 .3607
jhasneha/task6-Penn-ERReg .6573 28 .8083 28 .5755 37 .5610 .7857 .3568 .6214 .3732
jhasneha/task6-Penn-LReg .6497 33 .8043 36 .5699 41 .5460 .7818 .3547 .5969 .4137
jotacastillo/task6-SAGAN-RUN1 .5522 57 .7904 47 .5906 29 .5659 .7113 .4739 .6542 .4253
Konstantin Z/task6-ABBYY-General .5636 53 .8052 33 .5759 36 .4797 .7821 .4576 .6488 .3682
M Rios/task6-UOW-LEX PARA .6397 36 .7187 71 .3825 80 .3628 .6426 .3074 .2806 .2082
M Rios/task6-UOW-LEX PARA SEM .5981 49 .6955 82 .3473 84 .3529 .5724 .3066 .2643 .1164
M Rios/task6-UOW-SEM .5361 59 .6287 88 .2567 88 .2995 .2910 .1611 .2571 .2212
mheilman/task6-ETS-PERP .7808 7 .8064 32 .6305 11 .6211 .7210 .4722 .7080 .5149
mheilman/task6-ETS-PERPphrases .7834 6 .8089 27 .6399 7 .6397 .7200 .4850 .7124 .5312
mheilman/task6-ETS-TERp .4477 73 .7291 68 .5253 57 .5049 .5217 .4748 .6169 .4566
nitish aggarwal/task6-aggarwal-run1 .5777 52 .8158 20 .5466 52 .3675 .8427 .3534 .6030 .4430
nmalandrakis/task6-DeepPurple-DeepPurple hierarchical .6228 43 .8100 26 .5979 23 .5984 .7717 .4292 .6480 .3702
nmalandrakis/task6-DeepPurple-DeepPurple sigmoid .5540 56 .7997 41 .5558 50 .5960 .7616 .2628 .6016 .3446
nmalandrakis/task6-DeepPurple-DeepPurple single .4918 66 .7646 58 .5061 61 .4989 .7092 .4437 .4879 .2441
parthapakray/task6-JU CSE NLP-Semantic Syntactic Approach∗ .3880 82 .6706 86 .4111 76 .3427 .3549 .4271 .5298 .4034
rada/task6-UNT-CombinedRegression .7418 14 .8406 7 .6159 14 .5032 .8695 .4797 .6715 .4033
rada/task6-UNT-IndividualDecTree .7677 9 .8389 9 .5947 25 .5693 .8688 .4203 .6491 .2256
rada/task6-UNT-IndividualRegression .7846 5 .8440 6 .6162 13 .5353 .8750 .4203 .6715 .4033
sbdlrhmn/task6-sbdlrhmn-Run1 .6663 23 .7842 53 .5376 54 .5440 .7335 .3830 .5860 .2445
sbdlrhmn/task6-sbdlrhmn-Run2 .4169 78 .7104 77 .4986 64 .4617 .4489 .4719 .6353 .4353
sgjimenezv/task6-SOFT-CARDINALITY .7331 15 .8526 5 .6708 3 .6405 .8562 .5152 .7109 .4833
sgjimenezv/task6-SOFT-CARDINALITY-ONE-FUNCTION .7107 19 .8397 8 .6486 6 .6316 .8237 .4320 .7109 .4833
siva/task6-DSS-alignheuristic .5253 60 .7962 42 .6030 21 .5735 .7123 .4781 .6984 .4177
siva/task6-DSS-average .5490 58 .8047 35 .5943 26 .5020 .7645 .4875 .6677 .4324
siva/task6-DSS-wordsim .5130 61 .7895 49 .5287 55 .3765 .7761 .4161 .5728 .3964
skamler /task6-EHU-RUN1v2 † .3129 86 .6935 83 .3889 79 .3605 .5187 .2259 .4098 .3465
sokolov/task6-LIMSI-cosprod .6392 37 .7344 67 .3940 78 .3948 .6597 .0143 .4157 .2889
sokolov/task6-LIMSI-gradtree .6789 22 .7377 66 .4118 75 .4848 .6636 .0934 .3706 .2455
sokolov/task6-LIMSI-sumdiff .6196 45 .7101 78 .4131 74 .4295 .5724 .2842 .3989 .2575
spirin2/task6-UIUC-MLNLP-Blend .4592 70 .7800 56 .5782 35 .6523 .6691 .3566 .6117 .4603
spirin2/task6-UIUC-MLNLP-CCM .7269 16 .8217 16 .6104 17 .5769 .8203 .4667 .5835 .4945
spirin2/task6-UIUC-MLNLP-Puzzle .3216 85 .7857 51 .4376 69 .5635 .8056 .0630 .2774 .2409
sranjans/task6-sranjans-1 .6529 30 .8018 39 .6249 12 .6124 .7240 .5581 .6703 .4533
tiantianzhu7/task6-tiantianzhu7-1 .4533 72 .7134 74 .4192 73 .4184 .5630 .2083 .4822 .2745
weiwei/task6-weiwei-run1 † .6946 20 .8303 10 .6081 19 .4106 .8351 .5128 .7273 .4383
yeh/task6-SRIUBC-SYSTEM1† .7513 11 .8017 40 .5997 22 .6084 .7458 .4688 .6315 .3994
ygutierrez/task6-UMCC DLSI-MultiLex .6630 26 .7922 46 .5560 49 .6022 .7709 .4435 .4327 .4264
ygutierrez/task6-UMCC DLSI-MultiSem .6529 29 .8115 23 .6116 16 .5269 .7756 .4688 .6539 .5470
ygutierrez/task6-UMCC DLSI-MultiSemLex .7213 18 .8239 14 .6158 15 .6205 .8104 .4325 .6256 .4340
yrkakde/task6-yrkakde-DiceWordnet .5977 50 .7902 48 .5742 39 .5294 .7470 .5531 .5698 .3659
yrkakde/task6-yrkakde-JaccNERPenalty .6067 47 .8078 31 .5955 24 .5757 .7765 .4989 .6257 .3468

Table 1: The first row corresponds to the baseline. ALL for overall Pearson, ALLnorm for Pearson after normaliza-
tion, and Mean for mean of Pearsons. We also show the ranks for each measure. Rightmost columns show Pearson for
each individual dataset. Note: ∗ system submitted past the 120 hour window, post-deadline fixes, † team involving
one of the organizers.

390

Run ALL ALLw MSRpar MSRparw MSRvid MSRvidw SMT-eur SMT-eurw On-WN On-WNw SMT-news SMT-newsw
davide buscaldi/task6-IRIT-pg1 .4280 .4946 .4295 .4082 .6125 .6593 .4952 .5273 .5387 .5574 .3614 .4674
davide buscaldi/task6-IRIT-pg3 .4813 .5503 .4171 .4033 .6728 .7048 .5179 .5529 .5526 .5950 .3693 .4648
davide buscaldi/task6-IRIT-wu .4064 .4682 .4326 .4035 .5833 .6253 .4856 .5138 .5317 .5189 .3480 .4482
enrique/task6-UNED-H34measures .4381 .2615 .5328 .4494 .5788 .4913 .4785 .4660 .6692 .6440 .4465 .3632
enrique/task6-UNED-HallMeasures .2791 .2002 .3861 .3802 .2570 .2343 .4086 .4212 .6006 .5947 .5305 .4858
enrique/task6-UNED-SP INIST .4680 .3754 .5166 .5082 .6303 .5588 .4625 .4801 .6442 .5761 .4753 .4143
parthapakray/task6-JU CSE NLP-Semantic Syntactic Approach .3880 .3636 .3427 .3498 .3549 .3353 .4271 .3989 .5298 .4619 .4034 .3228
tiantianzhu7/task6-tiantianzhu7-1 .4533 .5442 .4184 .4241 .5630 .5630 .2083 .4220 .4822 .5031 .2745 .3536

Table 2: Results according to weighted correlation for the systems that provided non-uniform confidence alongside
their scores.

fied way the tools and resources used by those par- tween non-expert annotators and annotations from
ticipants that did submit a valid description file. In the authors is very high, showing the high quality of
the last row, the totals show that WordNet was the the dataset. The dataset was split 50% as train and
most used resource, followed by monolingual cor- test, with the exception of the surprise test datasets:
pora and Wikipedia. Acronyms, dictionaries, mul- a subset of WMT from a different domain and the
tilingual corpora, stopword lists and tables of para- OntoNotes-WordNet mapping. All datasets are pub-
phrases were also used. licly available.5
Generic NLP tools like lemmatization and PoS The exercise was very successful in participation
tagging were widely used, and to a lesser extent, and results. 35 teams participated, submitting 88
parsing, word sense disambiguation, semantic role runs. The best results scored a Pearson correlation
labeling and time and date resolution (in this or- over 80%, well beyond a simple lexical baseline
der). Knowledge-based and distributional methods with 31% of correlation. The metric for evaluation
got used nearly equally, and to a lesser extent, align- was not completely satisfactory, and three evalua-
ment and/or statistical machine translation software, tion metrics were finally published. We discuss the
lexical substitution, string similarity, textual entail- shortcomings of those measures.
ment and machine translation evaluation software. There are several tasks ahead in order to make
Machine learning was widely used to combine and STS a mature field. The first is to find a satisfac-
tune components. Several less used tools were also tory evaluation metric. The second is to analyze the
listed but were used by three or less systems. definition of the task itself, with a thorough analysis
The top scoring systems tended to use most of of the definitions in the likert scale.
the resources and tools listed (UKP, Takelab), with We would also like to analyze the relation be-
some notable exceptions like Sgjimenez which was tween the STS scores and the paraphrase judgements
based on string similarity. For a more detailed anal- in MSR, as well as the human evaluations in WMT.
ysis, the reader is directed to the papers of the par- Finally, we would also like to set up an open frame-
ticipants in this volume. work where NLP components and similarity algo-
rithms can be combined by the community. All in
6 Conclusions and Future Work all, we would like this dataset to be the focus of the
community working on algorithmic approaches for
This paper presents the SemEval 2012 pilot eval- semantic processing and inference at large.
uation exercise on Semantic Textual Similarity. A
simple definition of STS beyond the likert-scale was Acknowledgements
set up, and a wealth of annotated data was pro-
We would like to thank all participants, specially (in al-
duced. The similarity of pairs of sentences was
phabetic order) Yoan Gutierrez, Michael Heilman, Ser-
rated on a 0-5 scale (low to high similarity) by hu-
gio Jimenez, Nitin Madnami, Diana McCarthy and Shru-
man judges using Amazon Mechanical Turk. The
tiranjan Satpathy for their contributions on evaluation
dataset includes 1500 sentence pairs from MSRpar
metrics. Eneko Agirre was partially funded by the
and MSRvid (each), ca. 1500 pairs from WMT,
and 750 sentence pairs from a mapping between 5
http://www.cs.york.ac.uk/semeval-2012/
OntoNotes and WordNet senses. The correlation be- task6/

391

Word Sense Disambiguation
Named Entity recognition

Time and date resolution
Semantic Role Labeling
Distributional similarity
Distributional thesaurus

Tables of paraphrases
Monolingual corpora
Multilingual corpora

Lexical Substitution

Textual entailment
Machine Learning

String similarity
MT evaluation
KB Similarity
Dictionaries

Lemmatizer

POS tagger
Stop words

Alignment
Acronyms

Wikipedia
WordNet

Syntax
MWE

Other
SMT
aca08ls/task6-University Of Sheffield-Hybrid x x x x x x x
aca08ls/task6-University Of Sheffield-Machine Learning x x x x x x x
aca08ls/task6-University Of Sheffield-Vector Space x x x x x
baer/task6-UKP-run1 x x x x x x x x x x x x x x
baer/task6-UKP-run2 plus postprocessing smt twsi x x x x x x x x x x x x x x
baer/task6-UKP-run3 plus random x x x x x x x x x x x x x x
croce/task6-UNITOR-1 REGRESSION BEST FEATURES x x x x x x
croce/task6-UNITOR-2 REGRESSION ALL FEATURES x x x x x x
croce/task6-UNITOR-3 REGRESSION ALL FEATURES ALL DOMAINS x x x x x x
csjxu/task6-PolyUCOMP-RUN x x x x
danielcer/stanford fsa x x x x x x x
danielcer/stanford pdaAll x x x x x x x
danielcer/stanford rte x x x x x x x x
davide buscaldi/task6-IRIT-pg1 x x x x x
davide buscaldi/task6-IRIT-pg3 x x x x x
davide buscaldi/task6-IRIT-wu x x x x x
demetrios glinos/task6-ATA-BASE x x x x x x x
demetrios glinos/task6-ATA-CHNK x x x x x x x
demetrios glinos/task6-ATA-STAT x x x x x x x
desouza/task6-FBK-run1 x x x x x x x x x x x x x
desouza/task6-FBK-run2 x x x x x x x x
desouza/task6-FBK-run3 x x x x x x
dvilarinoayala/task6-BUAP-RUN-1 x x
dvilarinoayala/task6-BUAP-RUN-2 x
dvilarinoayala/task6-BUAP-RUN-3 x x
jan snajder/task6-takelab-simple x x x x x x x x x x x x x
jan snajder/task6-takelab-syntax x x x x x x x x x
janardhan/task6-janardhan-UNL matching x x x x x x
jotacastillo/task6-SAGAN-RUN1 x x x x x x x x
Konstantin Z/task6-ABBYY-General
M Rios/task6-UOW-LEX PARA x x x x x x x x
M Rios/task6-UOW-LEX PARA SEM x x x x x x x x
M Rios/task6-UOW-SEM x x x x x x x
mheilman/task6-ETS-PERP x x x x x x x
mheilman/task6-ETS-PERPphrases x x x x x x x x x
mheilman/task6-ETS-TERp x x x x x x x
parthapakray/task6-JU CSE NLP-Semantic Syntactic Approach x x x x x x x x x x
rada/task6-UNT-CombinedRegression x x x x x x x x x
rada/task6-UNT-IndividualDecTree x x x x x x x x x
rada/task6-UNT-IndividualRegression x x x x x x x x x
sgjimenezv/task6-SOFT-CARDINALITY x x x
sgjimenezv/task6-SOFT-CARDINALITY-ONE-FUNCTION x x x
skamler /task6-EHU-RUN1v2 x x x x x
sokolov/task6-LIMSI-cosprod x x x x
sokolov/task6-LIMSI-gradtree x x x x
sokolov/task6-LIMSI-sumdiff x x x x
spirin2/task6-UIUC-MLNLP-Blend x x x x x x x x x x x
spirin2/task6-UIUC-MLNLP-CCM x x x x x x x x x x x
spirin2/task6-UIUC-MLNLP-Puzzle x x x x x x x x x x x
sranjans/task6-sranjans-1 x x x x x x x x
sranjans/task6-sranjans-2 x x x x x x x x x x x
sranjans/task6-sranjans-3 x x x x x x x x x x x
tiantianzhu7/task6-tiantianzhu7-1 x x x x
tiantianzhu7/task6-tiantianzhu7-2 x x x
tiantianzhu7/task6-tiantianzhu7-3 x x x x
weiwei/task6-weiwei-run1 x x x x x x
yeh/task6-SRIUBC-SYSTEM1 x x x x x x x
ygutierrez/task6-UMCC DLSI-MultiLex x x x x x x x
ygutierrez/task6-UMCC DLSI-MultiSem x x x x x x x
ygutierrez/task6-UMCC DLSI-MultiSemLex x x x x x x x x
yrkakde/task6-yrkakde-DiceWordnet x x x
Total 8 6 10 33 5 5 9 20 47 7 31 37 49 13 13 4 7 12 43 9 4 13 17 10 5 15 25

Table 3: Resources and tools used by the systems that submitted a description file. Leftmost columns correspond to
the resources, and rightmost to tools, in alphabetic order.

392

European Communitys Seventh Framework Programme mantic nets and corpus statistics. IEEE Transactions
(FP7/2007-2013) under grant agreement no. 270082 on Knowledge and Data Engineering, 18(8):1138–
(PATHS project) and the Ministry of Economy under 1150, August.
grant TIN2009-14715-C04-01 (KNOW2 project). Daniel Herbert Rubenstein and John B. Goodenough. 1965.
Cer gratefully acknowledges the support of the Defense Contextual correlates of synonymy. Commun. ACM,
8(10):627–633, October.
Advanced Research Projects Agency (DARPA) Machine
E. Ukkonen. 1985. Algorithms for approximate string
Reading Program under Air Force Research Labora-
matching. Information and Contro, 64:110–118.
tory (AFRL) prime contract no. FA8750-09-C-0181 and
the support of the DARPA Broad Operational Language
Translation (BOLT) program through IBM. The STS an-
notations were funded by an extension to DARPA GALE
subcontract to IBM # W0853748 4911021461.0 to Mona
Diab. Any opinions, ﬁndings, and conclusion or recom-
mendations expressed in this material are those of the
author(s) and do not necessarily reﬂect the view of the
DARPA, AFRL, or the US government.

References
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn,
Christof Monz, and Josh Schroeder. 2007. (meta-)
evaluation of machine translation. In Proceedings of
the Second Workshop on Statistical Machine Transla-
tion, StatMT ’07, pages 136–158.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn,
Christof Monz, and Josh Schroeder. 2008. Further
meta-evaluation of machine translation. In Proceed-
ings of the Third Workshop on Statistical Machine
Translation, StatMT ’08, pages 70–106.
David L. Chen and William B. Dolan. 2011. Collect-
ing highly parallel data for paraphrase evaluation. In
Proceedings of the 49th Annual Meetings of the Asso-
ciation for Computational Linguistics (ACL).
B. Dolan, C. Quirk, and C. Brockett. 2004. Unsuper-
vised construction of large paraphrase corpora: Ex-
ploiting massively parallel news sources. In COLING
04: Proceedings of the 20th international conference
on Computational Linguistics, page 350.
Christiane Fellbaum. 1998. WordNet: An Electronic
Lexical Database. MIT Press.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance
Ramshaw, and Ralph Weischedel. 2006. Ontonotes:
The 90% solution. In Proceedings of the Human Lan-
guage Technology Conference of the North American
Chapter of the ACL.
Michael D. Lee, Brandon Pincombe, and Matthew Welsh.
2005. An empirical evaluation of models of text doc-
ument similarity. In Proceedings of the 27th Annual
Conference of the Cognitive Science Society, pages
1254–1259, Mahwah, NJ.
Y. Li, D. McLean, Z. A. Bandar, J. D. O’Shea, and
K. Crockett. 2006. Sentence similarity based on se-

393

SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity (20)

More from pathsproject (20)

Recently uploaded (20)

SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity