![# Problems Changes Modifications of
ConfigurationFile
Modificationsof
Source Code
1 Clauses/Subclausesstarted
with‘section#.#’cannot be
detected
Addregularexpressionin
configfile forsection#.#
3.1.TAB
Punctuation
3.3. Section#.#
2 Clauses/Subclauses started
with ‘Article
one/two/three’ cannot be
detected
Addregularexpressionin
configfile forArticle
One/Two/Three
3.2.Article
one/two/three…
4.1.Recognize
LetterNumeric
3 Title of Clauses/Subclauses
endedwithcoloncannotbe
detected
Turn off the filteringrule
whichexclude Clause
whose title endedwith
colon
3.4. ColonTitles 4.2.Colon Titles
Filter
4 Missing Clauses/Subclauses
if there isgap in numbering
Setcontinuouskey 4.3.Numbering
Gap
3. ModificationsofConfigFile
3.1. TAB Punctuation
Motivation:Indocumentparsing,titleswhichfollowatabpunctuationwillbe setaspriority3.Butactually
theyare importanttitlesaswe don’twantthembe priority3.
Solution:AddTAB intoimportantpunctuationinthe configfile.
3.2. Article one/two/three…
Motivation:Titleswith “letternumber”likearticle one/two/three can’tbe detectedbythe tool.
Solution: Add a new regular expression in config file to extract such titles. Besides, to make them
recognizable, amappingfunctionisalso neededtomap the “letternumber”like one/two/three to“key
value” like 1,2,3. This mapping function is added in the source code. Please refer to 4.1 for more
information.
Regular Expression:
sectionRegexp b((?:[Aa][Rr][Tt][Ii][Cc][Ll][Ee])s+(([TtWwEeNnHhIiRrFfOoGg]{4,5}[Yy]){0,1}-
{0,1}[OoNnTtWwRrEeFfSsVvXxGgHhLlIiUuYy]{3,9}))b
Explanation:Thisregularexpressioniscombinedbytwoparts. The firstpart isa prefix of “article”while
the secondpart isa lettersetof all possible combinationof letternumbersfromone tofifty-nine.
3.3. Section#. # (# refersto numeral symbols)
Motivation:Whenextractingmultileveltitles,the systemgiveslow prioritytothose titles whichfollowa
wordlike ‘section’ or‘article’etc.
Solution:Addnewregularexpressionswhichcontainthese wordsasprefixestoavoidthissituation.
Figure 1 - TAB Punctuation](https://image.slidesharecdn.com/de3b6541-5920-49ce-9674-9c40c282a1b1-161107094612/85/DocumentationofchangesondocumentparsingofBlueHOUND-2-320.jpg)
![Regular Expression:
multilevelRegexp (([Ss][Ee][Cc][Tt][Ii][Oo][Nn])s(d{1,3}|[a-z]{1,2}|[A-
Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5}).)((d{1,3}|[a-z]{1,2}|[A-
Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})(.))*((d{1,3}|[a-z]{1,2}|[A-
Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})(.?))s+
multilevelRegexp(([Ss][Ee][Cc][Tt][Ii][Oo][Nn])s(d{1,3}|[a-z]{1,2}|[A-
Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})-)((d{1,3}|[a-z]{1,2}|[A-
Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})(-))*(d{1,3}|[a-z]{1,2}|[A-
Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})s+
3.4. ColonTitles
Motivation:Whenextractingitems,the systemexclude titlesfollowedbycolonswhen
“LongTitleAndBlankDescription”filterisopen.
Solution: “LongTitleAndBlankDescription” filter contains a set of rules, like exclude clauses/subclauses
withlongtitle;excludeclause/subclauseswithblankdescription;andexcludeclauses/subclausesifitstitle
isendedwithcolonorsemicolon. If we simplyclosethisfilter,itwillhave badeffectsonotherdocuments.
Sowe separate the colonfunctionfrom“LongTitleAndBlankDescription”filterand addanew filternamed
“ColonTitles” inconfigfile. We alsoadjustedthe source code toseparate the rule aboutcolon/semicolon
from the set of rules of “LongTitleAndBlankDescription” filter. Please refer to 4.2 for more information.
4. ModificationsofSource Code
4.1. Recognize LetterNumeric
Motivation: As we’ve added a new regular expression in config file in 3.1, the system now can extract
titleswithletternumeric.Butthe systemcan’t give a numerickeyto themas there’snosuch functionin
the source code to achieve so.
Solution:Buildafunctiontogive keysto suchtitles.We simplyuse a“decode”functiontoachieve this.
Figure 3 shows twolistswhichare pre-builtforthe decoding function.
Figure 2 - ColonTitles Filter
Figure 3 – Preparation for Decoding](https://image.slidesharecdn.com/de3b6541-5920-49ce-9674-9c40c282a1b1-161107094612/85/DocumentationofchangesondocumentparsingofBlueHOUND-3-320.jpg)
![Figure 6 shows the code of Givekeys function. This function will check the priority and prefix of
neighboringtitles.If theyare the same,the systemwill change the keyof the secondtitle toone plusthe
keyof the firsttitle. Notedthatlists(e.g.:LastKn[],LastPrfx[]..) inthisfunctionwill be resetwhenusinga
newregularexpression.
This function now can’t identify English character and roman letter perfectly. When they are used at a
same article,it will be confused.Inthe attached code,thisfunctionnow is justused fordigital titles.For
letternumeric,Englishcharacterandromannumber,itwon’tbe usednow.(However,if onlyone of them
isusedin a givenarticle,thisfunction isstill agoodsolution).Youcansimplysearchby “givekeys” to see
where toopen thisfunctionforthese kindsof titles (relatedsentencesare commented inthe code).
Figure 6 – Givekeys Function](https://image.slidesharecdn.com/de3b6541-5920-49ce-9674-9c40c282a1b1-161107094612/85/DocumentationofchangesondocumentparsingofBlueHOUND-6-320.jpg)
DocumentationofchangesondocumentparsingofBlueHOUND
- 1. Documentation of changes on document parsing of BlueHOUND Version 1.2 Date 2016-06-02 Mentor: Long Yin Interns: Wan Xulang, Zhang Yuxi Author: Wan Xulang 1. Files To improve the performance of structure detection & clause extraction in BlueHOUND, we identified a lists of issues and tested the solutions for 4 of them. Changes have been made in two files. One is the configurationfile (.txt) whichcontainsthe parametersettingsof structure detection& clause extraction module.The otherfile (.java)the structuredetection&clause extractionmodulewhichcontainsthe detail rulesof structure detection&clause extraction. These are 6filesinthe documentfolder. ConfigFile-Original.txt - the original configuration file ConfigFile-Updated.txt - the updatedconfigurationfile DetectStructure-Original.java - the original source code of structure detection&clause extractionmodule DetectStructure-Updated.java - the updated source code of structure detection&clause extractionmodule StructureDetection&ClauseExtractionModule.pptx - Flow charts to describe the procedures in structure detection & clause extraction module. This file canbe usedas a reference toquicklyunderstandDetectStructure-Original.java DocumentationofchangesondocumentparsingofBlueHOUND.docx - Document the changes we made in configuration file (.txt) and the structure detection& clause extractionmodule (.java) 2. Problemsand solutions We testedthe performanceof documentparsing function by reviewing31documents. We foundthatnot all documents could be parsed 100% correctly. A list of issues has been identified, and we proposed 4 changes to fix some of these issues. Note that in problem 2 and 3, we need to modify both the configurationfileandthe source code.
- 2. # Problems Changes Modifications of ConfigurationFile Modificationsof Source Code 1 Clauses/Subclausesstarted with‘section#.#’cannot be detected Addregularexpressionin configfile forsection#.# 3.1.TAB Punctuation 3.3. Section#.# 2 Clauses/Subclauses started with ‘Article one/two/three’ cannot be detected Addregularexpressionin configfile forArticle One/Two/Three 3.2.Article one/two/three… 4.1.Recognize LetterNumeric 3 Title of Clauses/Subclauses endedwithcoloncannotbe detected Turn off the filteringrule whichexclude Clause whose title endedwith colon 3.4. ColonTitles 4.2.Colon Titles Filter 4 Missing Clauses/Subclauses if there isgap in numbering Setcontinuouskey 4.3.Numbering Gap 3. ModificationsofConfigFile 3.1. TAB Punctuation Motivation:Indocumentparsing,titleswhichfollowatabpunctuationwillbe setaspriority3.Butactually theyare importanttitlesaswe don’twantthembe priority3. Solution:AddTAB intoimportantpunctuationinthe configfile. 3.2. Article one/two/three… Motivation:Titleswith “letternumber”likearticle one/two/three can’tbe detectedbythe tool. Solution: Add a new regular expression in config file to extract such titles. Besides, to make them recognizable, amappingfunctionisalso neededtomap the “letternumber”like one/two/three to“key value” like 1,2,3. This mapping function is added in the source code. Please refer to 4.1 for more information. Regular Expression: sectionRegexp b((?:[Aa][Rr][Tt][Ii][Cc][Ll][Ee])s+(([TtWwEeNnHhIiRrFfOoGg]{4,5}[Yy]){0,1}- {0,1}[OoNnTtWwRrEeFfSsVvXxGgHhLlIiUuYy]{3,9}))b Explanation:Thisregularexpressioniscombinedbytwoparts. The firstpart isa prefix of “article”while the secondpart isa lettersetof all possible combinationof letternumbersfromone tofifty-nine. 3.3. Section#. # (# refersto numeral symbols) Motivation:Whenextractingmultileveltitles,the systemgiveslow prioritytothose titles whichfollowa wordlike ‘section’ or‘article’etc. Solution:Addnewregularexpressionswhichcontainthese wordsasprefixestoavoidthissituation. Figure 1 - TAB Punctuation
- 3. Regular Expression: multilevelRegexp (([Ss][Ee][Cc][Tt][Ii][Oo][Nn])s(d{1,3}|[a-z]{1,2}|[A- Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5}).)((d{1,3}|[a-z]{1,2}|[A- Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})(.))*((d{1,3}|[a-z]{1,2}|[A- Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})(.?))s+ multilevelRegexp(([Ss][Ee][Cc][Tt][Ii][Oo][Nn])s(d{1,3}|[a-z]{1,2}|[A- Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})-)((d{1,3}|[a-z]{1,2}|[A- Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})(-))*(d{1,3}|[a-z]{1,2}|[A- Z]{1,2}|[MDCLXVI]{1,5}|[mdclxvi]{1,5})s+ 3.4. ColonTitles Motivation:Whenextractingitems,the systemexclude titlesfollowedbycolonswhen “LongTitleAndBlankDescription”filterisopen. Solution: “LongTitleAndBlankDescription” filter contains a set of rules, like exclude clauses/subclauses withlongtitle;excludeclause/subclauseswithblankdescription;andexcludeclauses/subclausesifitstitle isendedwithcolonorsemicolon. If we simplyclosethisfilter,itwillhave badeffectsonotherdocuments. Sowe separate the colonfunctionfrom“LongTitleAndBlankDescription”filterand addanew filternamed “ColonTitles” inconfigfile. We alsoadjustedthe source code toseparate the rule aboutcolon/semicolon from the set of rules of “LongTitleAndBlankDescription” filter. Please refer to 4.2 for more information. 4. ModificationsofSource Code 4.1. Recognize LetterNumeric Motivation: As we’ve added a new regular expression in config file in 3.1, the system now can extract titleswithletternumeric.Butthe systemcan’t give a numerickeyto themas there’snosuch functionin the source code to achieve so. Solution:Buildafunctiontogive keysto suchtitles.We simplyuse a“decode”functiontoachieve this. Figure 3 shows twolistswhichare pre-builtforthe decoding function. Figure 2 - ColonTitles Filter Figure 3 – Preparation for Decoding
- 4. Figure 4 showsthe code of decodingfunction.Thisfunctionwill divide the numericsymbol intodifferent parts by “-” and convert each part into a numeric value by going through the two lists we set before. Finally,sumupthe numericvalue of eachpart to give the key. Before thisfunction,anotherfunctionnamed”isLetterNumeral”isusedtoidentifywhetheratitleisletter numberor not. 4.2. ColonTitlesFilter Motivation: To separate the colon titles excluding function from “LongTitleAndBlankDescription” filter. We make a newfilternamed “ColonTitles”. Thus, “LongTitleAndBlankDescription”filterwill onlyexclude clauseswhichhave longtitlesandblankdescription. Solution:Adda newfunctioninfilteringmodule tobecome anew filter. Figure 4 – Convert Function
- 5. Figure 5 showsthe code of howto identifyatitle followedbycolonandgive it a mark. Whenthisfilteris setto “true”.These markedtitleswill be excludedfromthe final output. Atthe same time,suchfunctions inlongtitle filterhave beenremoved. 4.3. NumberingGap Motivation: In item extraction part, system will do a fast filtering to check the continuous of extracted titles.However,somedocumentshave missingtitleswillbe effectbadlybythis.Alsothe pruningfunction is basedon continuouskeysof extractedtitles.Sowe can’t justclose that fast filteringfunctiontoavoid this. Solution: Build a tricky function to across this numbering gap. Note that this change may have negative impacton parsingotherdocumentsandthusit needsmore validationbeforeimplementation. Figure 5 – ColonTitles Filter
- 6. Figure 6 shows the code of Givekeys function. This function will check the priority and prefix of neighboringtitles.If theyare the same,the systemwill change the keyof the secondtitle toone plusthe keyof the firsttitle. Notedthatlists(e.g.:LastKn[],LastPrfx[]..) inthisfunctionwill be resetwhenusinga newregularexpression. This function now can’t identify English character and roman letter perfectly. When they are used at a same article,it will be confused.Inthe attached code,thisfunctionnow is justused fordigital titles.For letternumeric,Englishcharacterandromannumber,itwon’tbe usednow.(However,if onlyone of them isusedin a givenarticle,thisfunction isstill agoodsolution).Youcansimplysearchby “givekeys” to see where toopen thisfunctionforthese kindsof titles (relatedsentencesare commented inthe code). Figure 6 – Givekeys Function
- 7. 5. Outcome and Defects Figure 7 shows performance of documentparsingafterapplychange 1-4. It isshownthat change 1&2&3 can improve the overall performance of documentparsing. Change 4 has negative impactonseveral documents,especiallyfor“Google PlayTermsof Service”.More validation is needed before we implement this change to production. After improving change 4, some negative impactshave beensolved. Figure 7 – Performance on Test Documents