bồn tắm jacuzzi.docx
- 1. End-to-End Argumentation Mining in Student Essays Isaac Persing and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {persingq,vince}@hlt.utdallas.edu Abstract Understanding the argumentative structure of a persuasive essay involves addressing two challenging tasks: identifying the components of the essay’s argument and identifying the relations that occur between them. We ex- amine the under-investigated task of end-to- end argument mining in persuasive studentes- says, where we (1) present the first results on end-to-end argument mining in student es- says using a pipeline approach; (2) address error propagation inherent in the pipeline ap- proach by performing joint inference over the outputs of the tasks in an Integer Linear Pro- gramming (ILP) framework; and (3) propose a novel objective function that enables F-score to be maximized directly by an ILP solver. We evaluate our joint-inference approach with our novel objective function on a publicly- available corpus of 90 essays, where it yields an 18.5% relative error reduction in F-score over the pipeline system. 1 Introduction There has beena surge of interest in argumenta- tion mining in recent years. Argumentation mining typically involves addressing two subtasks: (1) ar- gumentcomponentidentification (ACI),which con- sists of identifying the locations and types of the components that make up the arguments (i.e., Ma- jor Claims, Claims, and Premises), and (2) relation identification (RI), which involves identifying the type of relation that holds between two argument components (i.e., Support, Attack, None). As a first step towards mining arguments in persuasive essays, 1384 Stab and Gurevych (S&G) annotated a corpus of 90 student essays with argument components and their relations (Stab and Gurevych, 2014a). To illustrate, consider the following excerpt from one essay: From this point of view, I firmly believe that (1) we should attach more importance to co- operation during primary education. First of all, (2) through cooperation, children can learn about interpersonalskills which are significant in the future life of all students. (3) What we acquired from team work is not only how to achieve the same goal with others but more importantly, how to get along with others. In this example, premise (3) supports claim (2), which in turn supports major claim (1). Using their annotated corpus, S&Gpresented ini- tial results on simplified versions of the ACI and RI tasks (Stab and Gurevych, 2014b). Specifically, they applied their learned ACI classifier to classify only gold argument components (i.e., text spans corre- sponding to a Major Claim, Claim, or Premise in the gold standard) or sentences that contain no gold argument components (as non-argumentative). Sim- ilarly, they applied their learned RI classifier to clas- sify only the relation between two gold argument components. In other words, they simplified both tasks by avoiding the challenging task of identify- ing the locations of argument components. Conse- quently, their approachcannot be applied in a realis- tic setting where the input is an unannotated essay. Motivated by this weakness, we examine in this paper argument mining in persuasive student essays in a considerably more challenging setting than that of S&G: the end-to-end setting. In other words, we Proceedings of NAACL-HLT 2016, pages 1384–1394, San Diego, California, June 12-17, 2016. Ⓧ c 2016 Association for Computational Linguistics
- 2. 1385 perform argument mining on raw, unannotated es- says. Our work makes three contributions. First, we present the first results on end-to-end argument mining in student essays using a pipeline approach, where the ACItask is performed prior to the RItask. Second, to avoid the error propagation problem in- herent in the pipeline approach, we perform jointin- ference over the outputs of the ACI and RI classi- fiers in an IntegerLinear Programming (ILP)frame- work (Roth and Yih, 2004), where we design con- straints to enforce global consistency. Finally, we argue that the typical objective function used exten- sively in ILP programs for NLP tasksis not ideal for tasks whose primary evaluation metric is F-score, and subsequently propose a novel objective function that enables F-score to be maximized directly in an ILP framework. We believe that the impact of our work goes beyond argument mining, as our F-score optimizing objective function is general enough to be applied to any ILP-based joint inference tasks. 2 RelatedWork Recall that identifying argumentative discourse structures consists of (1) identifying the locations and types of the argument components, and (2) iden- tifying how they are relatedto eachother. Below we divide related works into five broad categories based on which of these subtasks they addressed. Argument location identification. Works in this category aimed to classify whether a sentence con- tains an argument or not (Florou et al., 2013; Moens etal., 2007; Song etal., 2014; Swansonet al., 2015). The usefulness of existing works is somewhat lim- ited by the task’s coarseness:it won’t tell us which portion of a potentially long sentence contains the argument, for instance, but it can serve as a poten- tially useful first step in argument mining. Argument component typing. Works in this cat- egory focused on determining the type of an argu- ment. The vastmajority of previous works per-form argument component typing at the sentence level. For instance, Rooney et al. (2012) classi- fied sentences into premises, conclusions, premise- conclusions, and non-argumentative components; Teufel (1999) classified each sentence into one of seven rhetorical classes (e.g., claim, result, pur- pose); Burstein et al. (2003), Ong et al. (2014), and Falakmasir et al. (2014) assigned argumentative la- bels (e.g., claim, thesis, conclusion) to an essay’s sentences;Levy etal. (2014) detectedsentences that support or attack an article’s topic; Lippi and Torroni (2015; 2016) detected sentences containing claims; and Rinott et al. (2015) detected sentences containing evidence for a given claim. Sentence- level argument component typing has limitations, however. For example, it can identify sentences con- taining claims, but it cannot tell how many claims a sentence has or where in the sentence they are. Argument location identification and typing. Some works focused on the more difficult task of clause-level argument component typing (Park and Cardie, 2014; Goudas et al., 2015; Sardianos et al., 2015), training a Conditional Random Field to jointly identify and type argument components. Argumentcomponenttyping and relationidenti- fication. Given the difficulty of clause-level argu- ment component location identification, recent argu- ment mining works that attempted argument compo- nent typing and relation identification are not end-to- end. Specifically, they simplified the task by assum- ing as input gold argument components (Stab and Gurevych, 2014b; Peldszus and Stede, 2015). End-to-end argument mining. To our knowl- edge, only Palau and Moens (2009) addressed all the argument mining subtasks. They employed a hand-crafted context-free grammar (CFG) to gener- ate (i.e., extract and type) argument components at the clause level and identify the relations between them. A CFGapproach is less appealing in the essay domain because (1) constructing a CFG is a time- and labor-intensive task, (2) which would be more difficult in the less-rigidly structuredessaydo-main, which contains fewer rhetoricalmarkers indi- cating component types (e.g. words like “reject”, or “dismiss” which indicate a legal document’s conclu- sion); and (3) about 20% of essay arguments’ struc- tures are non-projective (i.e., when mapped to the ordered text, their argument trees have edges that cross), and thus cannot be captured by CFGs. 3 Corpus Our corpus consists of 90 persuasive essays col- lected and annotated by S&G. Some relevant statis- tics are shown in Table 1. Each essay is an average
- 3. 1386 Table 1: Corpus statistics. of 4.6 paragraphs (18.6 sentences) in length and is written in response to a topic such as “should high school make music lessons compulsory?” or “com- petition or co-operation-which is better?”. The corpus annotations describe the essays’argu- ment structure, including the locations and types of the components that make up the arguments, and the types of relations that hold betweenthem. The three annotated argument component types include: Ma- jor Claims,which express the author’s stance with respect to the essay’s topic, Claims, which are con- troversial statements that should not be accepted by readers without additional support, and Premises, # Rule (a) Potentialleft boundary locations (b) Potentialright boundary locations which are reasons authors give to persuade read- ers about the truth of another argument component statement. The two relation types include: Support, which indicates that one argument component sup- ports another, and Attack,which indicates that one argument component attacks another. 4 Pipeline-Based Argument Mining Next, we describe our end-to-end pipeline argument mining system, which will serve as our baseline. In this system, ACI is performed prior to RI. 4.1 Argument Component Identification We employ a two-step approach to the ACI task, where we first heuristically extract argument com- ponent candidates (ACCs) from an essay, and then classify each ACC as either a premise, claim, major claim, or non-argumentative, as described below. 4.1.1 Extracting ACCs We extract ACCs by constructing a set of low pre- cision, high recall heuristics for identifying the lo- cations in each sentence where an argument com- ponent’s boundaries might occur. The majority of these rules depend primarily on a syntactic parse tree we automatically generated for all sentences in the corpus using the Stanford CoreNLP (Manning et al., 2014) system. Since argument components are a clause-level annotation and therefore a large major- ity of annotated argument components are substrings Table 2: Rules for extracting ACC boundary locations. of a simple declarative clause (an S node in the parse tree),we begin by identifying each S node in a sen- tence’s tree. Given an S clause, we collect a list of left and right boundaries where an argument component may be- gin or end. The rules we used to find these bound- aries are summarized in Table 2. We then construct ACCs by combining each left boundary with each right boundary that occurs after it. As a result, we are able to find exact (boundaries exactly match) and approximate (over half of tokens shared) matches for 92.1% and 98.4% respectively of all ACs. 4.1.2 Training the ACI Classifier We train a classifier for ACI using MALLET’s (Mc- Callum, 2002) implementation of maximum en- tropy classification. We create a training instance from each ACC extracted above. If the ACC’s left and right endpoints exactly match an annotated argument component’s, the corresponding training instance’s class label is the same as that of the component. Otherwise, its class label is “non- 1 An additional point that requires explanation is that thelast two right boundary rules mention “possibly nested” nodes. In boundary rule 7, for example, this means that the S node might end in a PP node, which itself has a PP node as its last child, and so on. We generate a separate right boundary immediately before each of these PP nodes. Essays: 90 Paragraphs: 417 Sentences: 1,673 Major Claims: 90 Claims: 429 Premises: 1,033 Support Relations: 1,312 Attack Relations: 161 1 Exactly where the S node begins. 2 After an initial explicit connective,or if the con- nective is immediately followed by a comma, af- ter the comma. 3 After nth comma that is an immediate child of the S node. 4 After nth comma. # Rule 5 Exactly where the S node ends, or if S ends in a punctuation, immediately before the punctua- tion. 6 If the S node ends in a (possibly nested) SBAR node, immediately before the nth shallowest SBAR.1 7 If the S node ends in a (possibly nested)PP node, immediately before the nth shallowest PP.
- 4. 1387 − argumentative”. Each training instance is repre- sented using S&G’s structural, lexical, syntactic, in- dicator, and contextual features forsolving the same problem. Briefly, the structural features describe an ACC and its covering sentence’s length, punctua- tions, and location in the essay. Lexical features de- scribe the 1 3 grams of the ACC and its covering sentence. Syntactic features are extracted from the ACC’s covering sentence’s parse tree and include things such as production rules. Indicator features describe any explicit connectives that immediately precede the ACC. Contextual features describe the contents of the sentences preceding and following the ACC primarily in ways similar to how the struc- tural features describe the covering sentence. 4.2 Relation Identification We consider RI between pairs of argument compo- nents to be a five class classification problem. Given a pair of ACCs A1 and A2 where A1 occurs before A2 in the essay,either they are unrelated, A1 sup- ports A2, A2 supports A1, A1 attacks A2,or A2 at- tacksA1. Below we describe how we train and apply our classifier for RI. We learn our RI classifier using MALLET’s im- plementation of maximum entropy classification. Each training instance, which we call a training re- lation candidate (RC), consists of a pair of ACCs and one of the above five labels. By default, the in- stance’s label is “no relation” unless each ACC has the exact boundaries of a gold standard argument component and one of the remaining four relations holds between the two gold argument components. We create training RCs as follows. We construct RCs corresponding to true relations out of all pairs of argument components in a training essay having a gold relation. As the number of potential RCs far exceedsthe number of gold relations in an essay,we undersample the “no relation” class in the following way. Given a pair of argument components A and B between which there is a gold relation, we define Ap to be the closest previous ACC in the essay as gen- erated in Section 4.1.1 such that Ap’s text doesn’t of which have the “no relation” label, as long as the pairs’ text sequences do not overlap. We believe the resulting “no relation” training instances are infor- mative since each one is “close” to a gold relation. We represent each instance using S&G’s structural, lexical, syntactic, and indicator features for solving the same problem. Briefly, RC structural features describe many of the same things about each ACC as did the ACC structuralfeatures, though they also describe the difference between the ACCs (e.g. the difference in punctuation counts). Lexical features consist primarily of the unigrams appearing in each ACC and word pairs, where each word from one ACC is paired with each word from the other. Syn- tactic and indicator features encode the same infor- mation about each ACC as the ACC syntactic and indicator features did. We now apply the classifier to test essay RCs, which are created as follows. Given that this is a pipelined argument mining system, in order to en- sure that the RI system’s output is consistent with that of the ACI system, we generate test RCs from all possible pairs of ACCs that the ACI system pre- dicted are real components (i.e. it labeled them something other than “non-argumentative”). 5 Joint Inference for Argument Mining 5.1 Motivation There are two major problems with the pipeline ap- proach described in the previous section. First, many essay-levelwithin-task constraints are not enforced. For instance, the ACI task has the constraint that each essay has exactly one major claim, and the RI task has the constraint that each claim has no more than one parent. This problem arises because our ACI and RI classifiers, like those of S&G, classify each ACI and RI test instance independently of other test instances. We propose to enforce such essay- level within-task constraints in an ILP framework, employing ILP to perform joint inference over the outputs of our ACI and RI classifiers so that the re- sulting classifications satisfy these constraints.2 overlap with A. We also define As as the closest succeeding ACC after A such that As and Ado not overlap. We define Bp and Bs similarly with respect to B. From these ACCs, we generate the the four in- stances (Ap, B), (As, B), (A, Bp), and (A, Bs), all 2 Notethat while we partition documents into folds in our cross-validation experiments, S&G partition instances into folds. Hence, S&G’s evaluation setting prevents themfrom en- forcing essay-level constraints in addition to being unrealistic in practice.
- 5. 1388 a i i i a The second problem with the pipeline approachis that errorsmade early on in the pipeline propa- gate. For instance, assume that a Support relation exists between two argument components in a test essay. If the pipeline system fails to (heuristically) extract one or both of these argument components, or if it successfully extracts them but misclassifies one or both of them asnon-argumentative, then thepipeline system will not be able to identify the rela-tionship between them because no test RCs will be created from them. The above problem arises be- cause the RI classifier assumes the most probable output of the ACIclassifier for eachACCasinput. Hence,one possible solution to this problem is to make use of the n-best outputs of the ACI classifier for each argument component type, as this increases the robustness of the pipeline to errors made by theACI classifier. We obtain the n-best ACI outputs and employ them as follows. Recall that the ACI system usesa maximum entropy classifier, and therefore its out- put for each ACC is a list of probabilities indicat- ing how likely it is that the ACC belongs to each of the four classes (premise, claim, major claim, or non-argumentative). This means that it is possible to rank all the ACCs in a text by these probabilities. We use this idea to identify (1) the 3 most likely premise ACCs from each sentence, (2) the 5 most likely claim ACCs from each paragraph, and (3) the5 most likely major claim ACCs from each essay.3 Given these most likely ACC lists, we combine pairs of ACCs into test RCs for the RI classifier in the following way. As long as the ACCs do not over- lap, we pair (1) eachlikely premise ACCwith every other likely ACC of any type occurring in the same paragraph, and (2) each likely claim ACCwith each likely major claim ACC. We then present these test RCs to the RI classifier normally, making no other changes to how the pipeline system works. Employing n-best ACI outputs, however, intro- duces another problem: the output of the RI classi- tionship between two ACCs that the ACI classifier labeled non-argumentative. To enforce this cross- task consistency constraint, we also propose to em- ploy ILP. The rest of this section details our ILP- basedjoint inference approachfor argument mining, which addresses both of the aforementioned prob- lems with the pipeline approach. 5.2 Basic ILP Approach We perform joint inference over the outputs of the ACI and RI classifiers by designing and enforc- ing within-task and cross-taskconstraints in the ILP framework. Specifically, we create one ILP program for each test essay, as described below. Let Xni,Xpi, Xci, and Xmi be binary indi- cator variables representing whether the ILP solver believes ACC i has type none, premise, claim, and major claim, respectively. Let Cni, Cpi, Cci, and Cmi be the probabilities that ACC i has type none, premise, claim, and major claim, respectively, as dictated by the ACI maximum entropy classifier’s output.4 Let a be the count of ACCs. Let Y ni,j, Y si,j, Y ai,j, Y rsi,j, and Y rai,j be binary indicator variables representing whether the ILP solver believes ACCs i and j have no relation, i is supported by j, i is attacked by j, j is supported by i, and j is attacked by i, respectively, where (i, j) appears in the set of RCs B that we presented to the RI system as modified in Section 5.1. We assume all other ACC pairs have no relation. Let Dni,j, Dsi,j, Dai,j, Drsi,j, and Drai,j be the probabilities that component candidates i and j have no relation, i is supported by j, i is attacked by j, j is supported by i, and j is attacked by i,respectively, as dictated by the modified RI classifier described in Section 5.1.4 Given these definitions and probabilities, our ILP program’s default goal is to find an assignment of these variables X and Y in order to maximize P (X) + P (Y ), where: P (X) = 1 Σ log(Cn Xn + Cp Xp classifier because the RI classifier may posit a rela- 3 We select more premise than claim ACCs (3 per sentence vs 5 per paragraph) because the corpus contains over twice as many premises as claims. We select major claim ACCs only from the first and last paragraph because major claims never occur in middle paragraphs. +CciXci + CmiXmi) 4 We additionally reserve .001 probability mass to distribute evenly among Cni, Cpi, Cci, and Cmi (or Dni,j , Dsi,j , Dai,j , Drsi,j , and Drai,j ) to prevent math errors involving taking thelog of 0 which might otherwiseoccur in theformulasbelow. fier may no longer be consistent with that of the ACI i=1 i (1)
- 6. 1389 Σ | | Σ Σ . . . . Σ Σ 1 P (Y ) = Σ log(Dn i,jY ni,j + Dsi,j Y si,j |B| (i,j)∈B +Dai,jY ai,j + Drsi,jY rsi,j + Drai,jY rai,j) (2) {i|(i Σ ,j )∈B} (Y si,j + Y ai,j) (9) subject to the integrity constraints that: (3) an ACC is either not an argument component or it has ex- + {k|(j,k) ∈B} (Y rsj,k + Y raj,k) − Xpj ≥ 0 actly one of the real argument component types, (4) a pair of component candidates (i, j) must have ex- for i < j, i ∈ / P ar(j) : Xpj − Y ni,j ≤ 0 for k > j, k ∈ / P ar(j) : Xpj − Y nj,k ≤ 0 (10) actly one of the five relation types, and (5) if there is a relation between ACCs i and j, i and j must each be real components.5 Xni + Xpi + Xci + Xmi = 1 (3) Y ni,j + Y si,j + Y ai,j + Y rsi,j + Y rai,j = 1 (4) where i < j and j < k mean ACC i appears before j, and j appears before k, and P ar(j) is the set of ACCs in j’s covering paragraph. Our constraints on claims state that: (11) a claim has no more than one parent6 , and (12) if a claim has a parent, that parent must be a major claim. (Xpi + Xci + Xmi) + (Xpj + Xcj + Xmj ) (5) .{i|(i, j) ∈ B}. + .{k|(j, k) ∈ B}. Xcj −2(Y si,j + Y ai,j + Y rsi,j + Y rai,j) ≥ 0 In the objective function, a and B serve to bal- ance the contribution of the two tasks, preventing one from dominating the other. + (Y si,j + Y ai,j) {i|(i,j)∈B} + (Y rsj,k + Y raj,k) {k|(j,k) ∈B} (11) 5.3 Enforcing Consistency Constraints So far we have described integrity constraints, but recall that our goal is to enforce consistency by im- posing within-task and cross-taskconstraints, which force the ILP solutions to more closely resemble real essay argument structures. Our consistency con- straints fall into four categories. Our constraintson majorclaimsare that: (6) there is exactly one major claim in each essay,(7) major claims always occur in the first or last paragraph, and (8) major claims have no parents. a ≤ {i|(i, j) ∈ B} + {k|(j, k) ∈ B} + 1 for j < k, Xmk − Xcj − Y rsj,k − Y raj,k ≥ −1 for j > i, Xmi − Xcj − Y si,j − Y ai,j ≥ −1 (12) The last category, which comprises constraints that do not fit well into any other category, are: (13) the boundaries of actual components never overlap, (14) each paragraph must have at least one claim or major claim, and (15) each sentence may have at most two argument components.7 f or i overlaps j , Xni + Xnj ≥ 1 (13) Xmi = 1 (6) i=1 Xmi = 0 | i ∈ / first or last paragraph (7) ∀ paragraphs P : Xci i∈P + Xmi ≥ 1 (14) Y si,j + Xmj ≤ 1 , Y ai,j + Xmj ≤ 1 (8) ∀ sentences S : Σ Xpi + Xci + Xmi ≤ 2 (15) Y rsi,j + Xmi ≤ 1 , Y rai,j + Xmi ≤ 1 i∈S We solve each ILP program using Gurobi.8 Our constraints on premises are that: (9) a
- 7. 1390 − premise has atleast one parent, and (10) a premise is related only to components in the same paragraph. 5 Notethat because of previous integrity constraints, the first term in constraint 5 is 1 only if we predict that i is a real com- ponent, the second term is 1 only if we predict that j is a real component, and the third term is 2 only if we predict that there is a relationship between them. Otherwise, each term is 0. Thus term 3 prevents us from predicting a relationship between i and j unless the first and second terms cancel it out through predicting that i and j are real components. 6 The coefficient of Xcj is equal to the number of terms in the two summations on the left hand side of the equation. The intention is that component j’s claim status should have equal weight with the relations it might potentially participate in, so it is possible for j to participate as a child in a relation with all other available components unless it is a claim, in which case it can participate in at most one relation as a child. 7 Unlike the other constraints, the last two constraints are only mostly true: 5% of paragraphs have no claims or major claims, and 1.2% of sentences have 3 or more components. 8 http://www.gurobi.com
- 8. 1391 i,g Σ 2 r are the values of G in equation 17 ascalcu- TP = .{j . ∃i |gl(j) = pl(i) ∧ i = j}. (21) Σ . . . . 5.4 F-score Maximizing Objective Function The objective function we employ in the previous subsection attempts to maximize the average prob- ability of correct assignment of variables over the ACI and RI problems. This kind of objective func- tion, which aims to maximize classification accu- racy, was originally introduced by Roth and Yih (2004) in their seminal ILP paper, and hassince then been extensively applied to NLP tasks. However,it is arguably not an ideal objective function for our task, where F-score rather than classification accu- racy is used as the evaluation metric. In this section, we introduce a novel method for Equation 17? Our key idea is inspired by the E-step of the Expectation-Maximization algorithm (Demp- ster et al., 1977): while we cannot compute the ac- tual TP , FP , and FN due to the lack of gold anno- tations, we can compute their expected values using the probabilities returned by the ACI and RI classi- fiers. Using the notation introduced in Section 5.2, the expected TP , FP , and FN values for the ACI task can be computed as follows: TPe = Σ CgiXgi (18) F Pe = Σ (1 − Cgi)Xgi (19) constructing an ILP objective function that directly maximizes the average F-score over the two prob- FNe = Σ i,g Xgi i,g Chi h/=g + Xni(1 − Cni) (20) i lems. Recall that F-score can be simplified to: 2TP F = 2TP + FP + FN (16) where TP, FP, and FN are the counts of true pos- itives, false positives, and false negatives respec- tively. Unfortunately, we cannot use this equation for F-score in an ILP objective function for two rea- sons. First, this equation involves division, which cannot be handled using ILP since ILP can only han- dle linear combinations of variables. Second, TP, FP, and FN needto be computed using gold annotations, which we don’t have in a test document. We propose to instead maximize F by maximizing the following: G = α2TPe − (1 − α)(F Pe + FNe) (17) where TPe , FPe , and FNe , are estimated values for TP , FP , and FN respectively, and α attempts to balance the importance of maximizing the numera- tor vs minimizing the denominator.9 We ignore the 2TP term in the denominator because minimizing it would directly reduce the numerator. To maximize average F-score, we can therefore where g and h can be any argumentative class from the ACI problem (i.e. premise (p), claim (c), or ma- jor claim (m)). The formulas we use to calculate TPe , FPe , and FNe for the RI problem are identi- cal except C is replaced with D, X is replaced with Y , and g and h can be any class from the RI problem other than no-relation. 6 Evaluation 6.1 Experimental Setup Corpus. As mentioned before, we use as our cor- pus the 90 essays annotatedwith argumentative dis- course structures by S&G. All of our experiments are conducted via five-fold cross-validation on this corpus. In each fold experiment, we reserve 60% of the essays for training, 20% for development (select- ing features and tuning α), and 20% for testing. Evaluation metrics. To calculate F-score on each task using Equation 16, we need to explain what constitutes a true positive, false positive, or false negative on each task. Given that j is a true argu- ment component and i is an ACC,the formulas for the ACI task are: attempt to maximize the function Gc+Gr , where Gc . . . lated using the estimated values from the ACI and F P = {i. pl(i) = / n∧$j | gl(j) = pl(i)∧i = j}. (22) . RIproblem respectively. The question that still remains is, how can we es- timate values for TP , FP , and FN mentioned in 9 We tune α on the development set, allowing it to take any value from 0.7, 0.8, or 0.9, as this range tended to perform well and G . .
- 9. 1392 in early experiments. F N = {j $i |gl(j) = pl(i) ∧ i = j} (23) where gl(j) is the gold standard label of j, pl(i) is the predicted label of i,n is the non-argumentative class, and i = . j means i is a match for j. i and j are considered an exact match if they have exactly
- 10. 1393 P +R 11 Boldfaced results in Table 3 are highly significant (p < System ACI RI Avg F MC-F C-F P-F P R F S-F A-F P R F Approx BASE 11.1 26.9 51.9 64.0 33.6 44.0 6.1 0.8 5.7 6.2 5.8 24.9 OUR 22.2 42.6 66.0 56.6 57.9 57.2 21.3 1.1 16.8 28.0 20.4 38.8 Exact BASE 7.4 24.2 43.2 50.4 29.6 37.3 4.4 0.8 4.1 4.7 4.3 20.8 OUR 16.9 37.4 53.4 47.5 46.7 47.1 13.6 0.0 12.7 15.4 12.9 30.0 Table 3: Five-fold cross-validation average percentages for argument component identification (ACI) and relation identification (RI) for OUR systemand the pipeline-based BASEline system. Column abbreviations are Major Claim F-score (MC-F), Claim F- score (C-F), Premise F-score (P-F), Precision (P), Recall (R), F-score (F), Support F-score (S-F), and Attack F-score (A-F). the same boundaries, whereas they are considered an approximate match if they share over half their tokens. We perform most of our analysis on approximate match results rather than exact match results as it can be difficult even for human annotators to iden- tify exactly the same boundaries for an argument component.10 We use the same formulas for calcu- lating these numbers for the RI problem except that j and i represent a true relation and an RC respec- tively, two relations approximately (exactly) match if both their source and target ACCs approximately (exactly) match, and n is the no-relation class. 6.2 Results and Discussion Approximate and exactmatch results of the pipeline approach (BASE) and the joint approach (OUR)are shown in Table 3. Aswe cansee, using approximate matching, OUR system achieves highly significant1 1 improvements over the pipelined baseline system by a variety of measures.12 The most important of these improvements is shown in the last column, where our system outperforms the baseline by 13.9% ab- 10 Approximate match has been used in evaluating opinion mining systems (e.g., Choi et al. (2006), Yang and Cardie (2013)), where researchers have also reported difficulties in having human annotators identify exactly the same boundaries for an opinion expression and its sources and targets. They have adopted an even more relaxed notion of approximate match: they consider two text spans an approximate match if they share at least one overlapping token. solute F-score (a relative error reduction of 18.5%). This is the most important result because it most di- rectly measures our performance in pursuit of our ultimate goal, to maximize the average F-score over both the ACI and RI problems. The highly signif- icant improvements in other measures, particularly the improvements of 13.2% and 14.6% in ACI and RI F-score respectively, follow as a consequence of this maximization. Using exactmatching, the differ- ences in scores between OUR system and BASE’s are smaller and highly significant with respect to a smaller number of measures. In particular, under Exact matching OURsystem’s performances on the RI-P and RI-R metrics are significant (p < 0.02), while under Approx matching, they are highly sig- nificant (p < 0.002). 6.3 Ablation Results To analyze the performance gains yielded by each improvement to our system, we show ablation re- sults in Table 4. Each row of the table shows the re- sults of one ablation experiment on the test set. That is, we obtain them by removing exactly one feature set or improvement type from our system. The Baseline feature sets we remove include those for the ACI task (Cb) from Section 4.1.2 and those for the RI task (Rb) from Section 4.2.13 The ILP improvement sets we remove are the Default ILP (Id) system14 from Section 5.2, the Major claim (Im), Premise (Ip), Claim (Ic), and Other (Io) con- straints from Section 5.3 Equations 6−8, 9−10, 0.002, paired t-test) compared to the baseline. 12 All the results in Tables 3 and 4 are averaged across five folds, so it is not true that Favg = 2Pavg Ravg . Our F-score avg avg averaging method is preferable to calculating F-scores using the above formula because theformula can be exploited to give arti- ficially inflated F-scores by alternating between high precision, low recall, and low precision, high recall labelings on different folds. 13 When we remove a baseline feature set, we represent each instance to the corresponding classifier using no features. Asa result, the classifier’s predictions are based solely on the fre- quency of the classes seen during training. 14 Notethat removing the default ILP system(Id) necessitates simultaneously removing all other ILP-related improvements. Thus, a systemwithout it is equivalent to BASE, but with RCs generated as described in Section 5.1.
- 11. 1394 — − Mod ACI RI Avg F P R F P R F ALL 54.9 58.8 56.7 16.7 26.4 20.4 38.6 Cb 44.0 38.5 40.9 14.2 14.8 14.4 27.7 Rb 57.9 58.7 58.2 16.6 24.0 18.2 38.2 Id 64.0 33.6 44.0 6.6 21.8 10.1 27.0 Im 44.6 46.7 45.6 14.7 27.8 19.2 32.4 Ip 51.5 58.4 54.7 13.2 26.1 17.3 36.0 Ic 49.0 51.5 50.2 14.2 27.9 18.8 34.5 Io 40.5 65.9 50.1 7.0 29.2 11.2 30.7 If 61.1 40.4 48.5 22.3 15.7 18.2 33.4 Table 4: Ablation results. How OUR system performs on one development set as measured by percent Precision, Recall, and F-score if each improvement or feature set is removed. 11 12, and 13 15 respectively, and the F-score maximizing objective function from Section 5.4. Broadly, we see from the last column that all of our improvement setsare beneficial (usually signifi- cantly15 ) to the system, as performance drops with their removal. Notice also that wheneverremov- ing an ILP improvement set harms average F-score,it also simultaneously harms ACI and RI F-scores, usually significantly. This holds true even when the improvement set deals primarily only with one task (e.g. Io for the ACItask), suggesting that our system is benefiting from joint inference over both tasks. 6.4 Error Analysis and Future Work Table 3 shows that OUR system has more trouble with the RI task than the ACI task. A closer inspec- tion of OUR system’s RI predictions reveals that its low precision is mostly due to predicted relation- ships wherein one of the participating ACCs is not a true argument component. Since false positives in the ACI task have an outsized impact on RI preci- sion, it may be worthwhile to investigate ILP objec- tive functions that more harshly penalize false posi- tive ACCs. The RI task’s poor recall has two primary causes. The first is false negatives in the ACI task. It is im- possible for an RI system to correctly identify a re- lationship between two ACs if the ACI system fails to identify either one of them asanAC.We be-lieve ACI recall, and by extension, RI recall, can be 15 Boldfaced results are significantly lower than ALL, the systemwith all improvements left intact, with p < 0.05. improved by exploiting the following observations. First, we noticed that many argument components OUR system fails to identify, regardless of their type, contain words that are semantically similar to words in the essay’s topic (e.g., if the topic men- tions “school”, argument components might men- tion “students”). Hence, one way to improve ACI recall, and by extension, RI recall, would be to cre- ate ACI features using a semantic similarity mea- sure such as the Wikipedia Link-based similarity measure (Milne and Witten, 2008). Second, major claims are involved in 32% of all relationships, but OUR system did an especially poor job at identify- ing them due to their scarcity. Since we noticed that major claims tend to include strong stancetaking lan- guage (e.g., words like “should”, “must”, and “be- lieve”), it may be possible to improve major claim identification by constructing an arguing language lexicon as in Somasundaran and Wiebe (2010), then encoding the presence of any of these arguing words as ACC features. The second major cause of OUR system’s poor RI recall is its failure to identify relationships be- tween two correctly extracted ACs. We noticed many of the missed relationships involve ACs that mention some of the same entities. Thus, a coref- erence resolver could help us build features that de- scribe whether two ACCs are talking about the same entities. 7 Conclusion We presented the first results on end-to-end argu- ment mining in persuasive student essays using a pipeline approach, improved this baseline approach by designing and employing global consistency con- straints to perform joint inference over the outputs of the tasks in an ILP framework and proposed a novel objective function that enables F-score to be max- imized directly by an ILP solver. In an evaluation on Stab and Gurevych’s corpus of 90 essays, our ap- proach yields an 18.5% relative error reduction in F- score over the pipeline system. Acknowledgments We thank the three anonymous reviewers for their detailed comments. This work wassupported in part by bồn tắmjacuzzi NSFGrants IIS-1147644 and IIS- 1219142.
- 12. 1395 References Jill Burstein, Daniel Marcu, and Kevin Knight. 2003. Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems, 18(1):32–39. Yejin Choi, Eric Breck, and Claire Cardie. 2006. Joint extraction of entities and relations for opinion recog- nition. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 431–439. Arthur P. Dempster, Nan M. Laird, and Donald B. Ru- bin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39:1–38. Mohammad Hassan Falakmasir, Kevin D. Ashley,Chris- tian D. Schunn, and Diane J. Litman. 2014. Identify- ing thesis and conclusion statements in student essays to scaffold peer review. In Intelligent Tutoring Sys- tems, pages 254–259. Springer International Publish- ing. Eirini Florou, Stasinos Konstantopoulos,Antonis Kouk- ourikos, and Pythagoras Karampiperis. 2013. Argu- ment extraction for supporting public policy formu - lation. In Proceedings of the 7th Workshop on Lan- guage Technology for Cultural Heritage, Social Sci- ences, and Humanities, pages 49–54. Theodosios Goudas,Christos Louizos, Georgios Petasis, and Vangelis Karkaletsis. 2015. Argument extraction from news, blogs, and the social web. International Journal on Artificial Intelligence Tools, 24(5). Ran Levy, Yonatan Bilu, Daniel Hershcovich,Ehud Aha- roni, and Noam Slonim. 2014. Context depen-dent claim detection. In Proceedings of the 25th In- ternational Conference on ComputationalLinguistics, pages 1489–1500. Marco Lippi and Paolo Torroni. 2015. Context- independent claim detection for argument mining. In Proceedings of the 24th International Joint Confer- ence on Artificial Intelligence, pages 185–191. Marco Lippi and Paolo Torroni. 2016. Argument mining from speech: Detecting claims in political debates. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Proceedingsof 52nd Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 55–60. Andrew Kachites McCallum. 2002. MALLET: A Ma- chine Learning for Language Toolkit. http:// mallet.cs.umass.edu. David Milne and Ian Witten. 2008. An effective, low- cost measure of semantic relatedness obtained from Wikipedia links. In Proceedings ofAAAI Workshop on Wikipedia and Artificial Intelligence: an Evolving Synergy, pages 25–30. Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. Automatic detection of arguments in legal texts. In Proceedings of the 11th International Conference on Artificial Intelligence and Law, pages 225–230. Nathan Ong, Diane Litman, and Alexandra Brusilovsky. 2014. Ontology-based argument mining and auto- matic essay scoring. In Proceedings ofthe First Work- shop on Argumentation Mining, pages 24–28. Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: The detection, classifi- cation and structure of arguments in text. In Proceed- ings of the 12th International Conference on Artificial Intelligence and Law, pages 98–107. Joonsuk Park and Claire Cardie. 2014. Identifying ap- propriate support for propositions in online user com- ments. In Proceedings of the First Workshop on Argu- mentation Mining, pages 29–38. Andreas Peldszus and Manfred Stede. 2015. Joint pre- diction in mst-style discourse parsing for argumenta- tion mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Process- ing, pages 938–948. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence - an automatic method for context dependentevidencedetection. In Proceed-ings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450. Niall Rooney, Hui Wang, and Fiona Browne. 2012. Ap- plying kernel methods to argumentation mining. In Proceedings of the 21st International Florida Artifi- cial Intelligence Research Society Conference. Dan Roth and Wen-tau Yih. 2004. A linear program- ming formulation for global inference in natural lan- guage tasks. In Proceedings of the Eighth Conference on Computational Natural Language Learning,pages 1–8. Christos Sardianos, Ioannis Manousos Katakis,Georgios Petasis, and Vangelis Karkaletsis. 2015. Argument extraction from news. In Proceedings of the Second Workshop on Argumentation Mining, pages 56–66. Swapna Somasundaran and Janyce Wiebe. 2010. Recog- nizing stances in ideological on-line debates. In Pro- ceedings of the NAACL HLT 2010 Workshop on Com- putational Approaches to Analysis and Generation of Emotion in Text, pages 116–124. Yi Song, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation
- 13. 1396 schemes for essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 69–78. Christian Stab and Iryna Gurevych. 2014a. Annotating argument components and relations in persuasive es- says.In Proceedingsof the 25th International Confer- ence on Computational Linguistics,pages 1501–1510. Christian Stab and Iryna Gurevych. 2014b. Identify- ing argumentative discourse structures in persuasive essays.In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 46–56. Reid Swanson, Brian Ecker, and Marilyn Walker. 2015. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 217–226. Simone Teufel. 1999. Argumentative Zoning: Informa- tion Extraction from Scientific Text. Ph.D. thesis,Uni- versity of Edinburgh. Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings ofthe 51st Annual Meeting of the Association for Com- putationalLinguistics (Volume 1:Long Papers),pages 1640–1649.