AACL 2018 - Going Beyond Simple Word-list Creation Using CasualConc
- 1. Going beyond simple word-list creation using CasualConc Yasuhiro IMAO Osaka University, Japan casualconc@gmail.com AACL 2018 at Georgia State University, Atlanta GA
- 2. A few questions How many of you are Mac users? How many of you have used CasualConc?
- 3. A few observations Through attending presentations / reading papers Methods of analysis Use of statistics Hugely depend on the access to the resources the tools one uses specialized application programing skills someone who can write scripts
- 4. To advance the field more easy-to-use and accessible tools are necessary
- 5. Current Situation AntConc and Antxxx and other more small specialized application WordSmith Tools / Monoconc Pro The gold standard?
- 7. A little bit of background I started developing a concordancer around 2005 I released the first, limited version around 2008 It is a Mac native app! KWIC, Word/n-gram Lists, Collocation
- 9. Before going into more detail
- 10. Small Scale Corpus Research Building your own specialized corpus Possibly adding annotations (POS, syntactic, etc.) Which tools to use?
- 11. A suggestion (not the answer) I have developed few companion apps CasualTranscriber (transcription helper) CasualTextractor (text extractor/editor) CasualTagger (tagging helper) CasualPConc (parallel concordancer)
- 12. CasualTagger KWIC search + short-cut tag insertion Batch Processing Tagging (TreeTagger, STF CoreNLP, MeCab) Tokenizing (macOS built-in tagger) Sentence Splitting (macOS built-in tagger, STF CoreNLP) Misspelling detection (non-dictionary words) compound-2-word variation detection
- 13. CasualTagger
- 16. Finally, the main point!
- 17. Today’s highlights Corpus file management - going beyond loading files More informative word/n-gram lists Individual file word/n-gram lists More keyword extraction functions - going beyond LL/χ2 Visualizing frequency data - incl. multivariate analyses Some other niche functions
- 19. Corpus File Management Accept Drag & Drop!
- 20. Where were those files I want to use? Of course, you can use Spotlight to search them, but… If the application remembers where they are…
- 23. More Informative Word/n-gram List
- 24. Proportion, # of Files, % of Files
- 25. Proportion, # of Files, % of Files
- 26. 3-grams / 4-grams comparison
- 27. p-frames
- 28. p-frames
- 29. Lemma / Spelling variations/ multi-word
- 30. Lemma / Spelling Variation
- 32. Multi Filters
- 33. Search Word List
- 34. Individual File Word/n-gram Lists
- 35. Individual File/Corpus Word/n-gram Lists
- 36. More keyword extraction functions
- 37. Sample ICNALE - Writing Written learner English corpus College students in Asian countries/regions JPN, CHN, HKG, IDN, KOR, PAK, PHI, SIN, THA, TWN Two topics Ave. 220-230 words
- 38. Simple Keyword Extraction (LL)
- 42. Going Beyond Simple Keyness
- 44. TF-IDF (term-frequency / inverse document frequency)
- 45. TF-IDF (term-frequency / inverse document frequency)
- 46. Mann-Whitney-U (using multiple files)
- 47. Random Forest (1000 words)
- 50. Word Cloud
- 51. Let’s go back to keyword analysis
- 52. Simple Keyword Extraction (LL)
- 53. Keyword analysis is done How can you check the validity? Let’s see how well they separate the groups Use the LL result above 6.63 (p < .01)
- 54. Principle Component Analysis (PCA)
- 55. Cluster Analysis
- 56. What about different ability levels?
- 57. Principle Component Analysis (PCA)
- 58. Cluster Analysis
- 59. Heat Map
- 60. By the way, do you check raw data? Not a lot of people really look at the data… Let’s check the use of “I think”
- 61. Box Plot
- 62. Histogram
- 63. What about a word that are too common to be looked at?
- 64. Box Plot
- 65. Histogram
- 66. Some other misc features…
- 67. Relative Position Plot (paragraph) therefore / however Japanese Non-Japanese ENS
- 68. Relative Position Polot (sentence) therefore / however Japanese Non-Japanese ENS
- 69. Vocab Profiler
- 70. Tokenizer
- 71. Tokenizer
- 73. The Way Forward Utilizing Stanford CoreNLP - dependency information
- 75. CasualConc I just released version 2.1.0 with the updated manual The manual is full of screenshots with over 250 pages It is a FREEWARE Downloadable from https://sites.google.com/site/casualconc Or just google ‘casualconc’