![Aside: Why KS as noun associations?
• Deese and word net
• Burgess and HAL
• AI models (Elmann… Rumelhart…)
• Because I started this with concept maps that
link concepts (nouns, see above)
• It doesn’t make much sense to compare
different parts of speech, [red ------- water]
14](https://image.slidesharecdn.com/tlt2016-160502021856/85/Teaching-Learning-with-Technology-TLT-2016-14-320.jpg)
Teaching & Learning with Technology TLT 2016
- 1. Automatic Knowledge Structure Measures in Online Courses Roy B. Clariana, Kyung Kim, JooYoung Seo Dept. of Learning and Performance Systems, College of Education 1
- 2. Presenter: Dr. Roy B. Clariana Professor of Learning, Design, & Technology 2 The Story of “Knowledge Structure”
- 3. Iterate measurement approach Iterate my sandbox model This story begins with concept maps 3
- 4. Ellen Taricani’s (2002) dissertation • Undergraduates read Frank Dwyer’s heart text • Then draw concept maps • Treatment group receives an expert map as ‘feedback’, control group no feedback • Posttest measures terminology, comprehension, drawing • My interest – do the concept maps relate to the posttests? But first, a few notes about concept maps… 4
- 5. Definition of concept map • Novak (1972) concept maps are diagrammatic representations of propositions arranged hierarchically • Mind maps (*Buzan, 1980s??)) • Semantic maps (Aly 1944) 5 (reflect specific epistemology or belief about what knowledge is) Theories embody or reflect an epistemology, so I want to use terminology that are ‘theory free’
- 6. Concept maps: What can be measured? “Regardless of the domain within which structural knowledge has been investigated, the concept of structural knowledge itself is thought to consist of three components. Those three components are (1) relevant domain concepts, (2) the presence and/or nature of relationships between those concepts, and (3) the strength of those relationships. These components then become critical to the definition and measurement of the construct.” p.2 6 Suen, Hoi.K., & Murphy, L.C.R. (1999). Validating Measures of Structural Knowledge through the Multitrait-Multimethod matrix. Presented at AERA.
- 7. Recording link and distance data of a concept map 7 of 34 lungs oxygenateddeoxygenated pulmonary artery pulmonary vein left atrium right ventricle Link Array (linear, proposition specific) a b c d e f g a left atrium - b lungs 0 - c oxygenate 0 1 - d pulmonary artery 0 1 0 - e pulmonary vein 1 1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle 0 0 0 1 0 0 - Distance Array (relational) a b c d e f g a left atrium - b lungs 120 - c oxygenate 150 36 - d pulmonary artery 108 84 120 - e pulmonary vein 73 102 114 138 - f deoxgenate 156 42 54 84 144 - g right ventricle 66 102 138 42 114 120 - moves through to the passes into to the Student’s concept map raw data: (n2-n)/2 pair-wise comparisons We noticed that participants’ spend a lot of time making small moves
- 8. Distance raw data reduction by Pathfinder KNOT 8 of 34 Pathfinder Network a b c d e f g a left atrium - b lungs 0 - c oxygenate 0 1 - d pulmonary artery 0 1 0 - e pulmonary vein 1 1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle 0 0 0 1 0 0 - Distance Array a b c d e f g a left atrium - b lungs 120 - c oxygenate 150 36 - d pulmonary artery 108 84 120 - e pulmonary vein 73 102 114 138 - f deoxgenate 156 42 54 84 144 - g right ventricle 66 102 138 42 114 120 - lungs oxygenateddeoxygenated pulmonary artery pulmonary vein left atrium right ventricle Pathfinder network (based on distances) PFnet analysis: 21 distance data points reduced to 6 link data points Pathfinder algorithm reduces the proximity raw data using triangle inequality to define the shortest path between all of the terms
- 9. In the NO feedback control, concept map link and distance data contain different KS information Taricani, E. M. & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), 61-78. Poindexter, M. T., & Clariana, R.B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184. Taricani & Clariana (2006) Term Comp Link data (linear props) 0.78 0.54 Distance data (relational) 0.48 0.61 9 of 34 Poindexter & Clariana (2006)Term Comp Link data (linear props) 0.77 0.53 Distance data (relational) 0.69 0.71 KS to posttest correlations
- 10. What we learned here • Open ended concept maps are difficult to score, so constrain the activity by providing a list for students • Link data – verbatim propositional knowledge • Distance data – relational knowledge, inferences and comprehension • Giving students an expert map after making their own map wrecks posttest performance (note: we saw similar damage for refutation texts in Ntshalintshali's dissertation) 10 We sensed that we were on to something. So what other ways of measuring KS are there?
- 11. A bigger picture … 11 Reading Texting TV Radio Conversations Sign language needs concerns feelings empowerment relationship motivation individual productivity pay plan contingency classical efficiency humanistic measure leadership managementsuccess focus company TQM quality customers goal work situation employee Knowledge structure
- 12. The notion of ‘above the line’ and ‘below the line’ 12 The line dialog Church School Home Pastor Teacher People Meals Auditorium Podium HymnalSongs Lectern Read Concept mapping Write an essay Take IQ test Production tasks Implicit/tacit Knowledge structure Artifact structure
- 13. apposite KS specific cognitive tasks “… over the course of learning, students knowledge structures become more similar to this expert structure, and students who have acquired better knowledge structures tend to perform better on traditional performance assessments (Jonassen, Beissner, & Yacci, 1993; Wilman, 1996). There is also some evidence that knowledge structures, as reflected by conceptual knowledge, may play a more direct causal role in enabling good performance rather than simply reflecting expertise.” (p. 427) knowledge structure mental function 13 Trumpower, D.L., & Goldsmith, T.E. (2004). Structural enhancement of learning. Contemporary Educational Psychology, 29, 426–446.
- 14. Aside: Why KS as noun associations? • Deese and word net • Burgess and HAL • AI models (Elmann… Rumelhart…) • Because I started this with concept maps that link concepts (nouns, see above) • It doesn’t make much sense to compare different parts of speech, [red ------- water] 14
- 15. Application of word co-occurrences as a visual wordnet • Deese free recall lists WordNet: An on-line lexical database http://kylescholz.com/projects/wordnet/ • Visual wordnet: Many applications: • Semantic web • Word predictors • Spelling apps • Google 15 Fellbaum, Christiane (2005). WordNet and wordnets. In: Brown, Keith et al. (eds.), Encyclopedia of Language and Linguistics, Second Edition, Oxford: Elsevier, 665-670 Deese–Roediger–McDermott (DRM) paradigm
- 16. Burgess: Large vector model of L1 • 70, 000 x 70, 000 terms matrix • “The corpus that served as input to the HAL model is approximately 300 million words of English text gathered from Usenet newsgroups that contained English text. Properties of Usenet text that were appealing were both its conversational nature and its diverse nature, making it closer in form to everyday speech.” Burgess, C. (1998). From simple associations to the building blocks of language: Modeling meaning in memory with the HAL model. Behavior Research Methods. Instruments. & Computers, 30 (2), 188-198. p.194 Generic L1 lexicon 16
- 17. Trained neural networks exhibit the same sort of ‘categorization’ Elman, J.L. (2004). An alternative view of the mental lexicon. TRENDS in Cognitive Sciences, 8 (7), 301-306. 17
- 18. Dave Jonassen’s summary … graph building similarity ratings semantic proximity word associations card sort ordered recall free recall additive trees hierarchical clustering ordered trees minimum spanning trees link weighted Pathfinder nets Networks Dimensional principal components MDS – multidimensional scaling cluster analysis expert/ novice qualitative graph comparisons quantitative graph comparisons relatedness coefficients scaling solutions C of PFNets Trees Knowledge representation Knowledge comparison Knowledge elicitation Jonassen, Beissner, & Yacci (1993), page 22 18 of 34 concept maps written text Social network analysis
- 19. Measuring KS in essays: started with how reading seems to work 19 of 34 Hyona, J., & Lorch, R.F. (2004). Effects of topic headings on text processing: evidence from adult readers’ eye fixation patterns. Learning and Instruction, 14, 131–152. Figure 1, p.136
- 20. Knowledge structure (KS) 20 of 34 Imminent extinction pandas the climate today exclusively in the wildlive Imminent extinction pandas the climate Retrieval function A B (propositional knowledge): Where do pandas live? In the wild A B,C,D (relational knowledge): What do we know about pandas today? Pandas are heading towards extinction in the wild due to climate change Retrieval structure linear Notice for this reader: “imminent extinction”, “pandas” and “the climate” enter the attentional sequence twice, the linear sequence begins to fold
- 21. Read KS Retrieval function 21 of 34 Relational Retrieval structure Retrieval function A B (propositional knowledge): Where do pandas live? In the wild A B,C,D (relational knowledge): What do we know about pandas today? Pandas are heading towards extinction in the wild due to climate change Thus the linear sequence has propositional and relational information in the same trace (fuzzy trace theory) a b c d a 1 0.2 0.1 0.1 b 0.3 1 0 0.2 c 0.3 0 1 0.1 d 0 0.4 0.1 1
- 22. ALA-Reader: term links and distances in students essays 22 Clariana, R.B., Wolfe, M. B., & Kim, K. (2014). The influence of narrative and expository text lesson text structures on knowledge structures: alternate measures of knowledge structure. Educational Technology Research and Development, 62 (4), 601-616. Linear link data better than word distance data
- 23. Example: read Supertanker text How can engineers (1) help prevent spills of oil (2) from supertankers (3)? Supertankers (4) are huge ships (5) that carry oil (6) over the oceans (7). A supertanker (8) can contain about a half-million tons of oil (9). Such huge tankers (10) are each the size of five football fields (11). A tanker's (12) cargo area could easily hold the Empire State Building (13). Most of the world's oil (14) is transported by these supertankers (15). Disasters (16) occur when wrecked tankers (17) spill oil (18) into the ocean (19). As a result of these oil spills (20), the environment is damaged (21). In 1967, a supertanker (22), named Torrey Canyon, crashed near England (23). This crash resulted in washing ashore 200,000 dead seabirds (24). In 1989, the tanker (25) Exxon Valdez (25) spilled oil (26) into Alaska's (27) coast. As a result of it the 11 million gallons spilled (infer oil spill 28), 1,000 otters (29) died. Oil spills (30) from tankers (31) also kill drifting microscopic (32) plant life. These plants provide food for sea life (33), such as whales and shrimp (34). They also produce 70 percent of the world's oxygen (35) supply. Oil spills (36) result partly from limitations in supertankers' (37) engineering (38). Supertankers (39) lack double bottom hulls (40) for extra protection. They also lack extra power and steering equipment for safety (41). They have only one boiler (42) to provide the ship power (43). They have only one propeller (44) to steer (45) the huge ship (46). Lack of such backup (47) components causes problems (48) during emergencies (49). Emergency (49) situations include ocean storms (50) and coastal reefs (51). . Solutions (52) to these problems (53) with oil spills (54) include three tactics. · Supertankers (55) must be built with added hulls (56), .boilers (57), and propellers (58). These provide extra safety (59), control, and backup in emergencies (60). Also, officers need top training (61) to run and maneuver their ships (62). Supertanker (63) simulators (64) at some facilities provide top training (65). Finally, ground control (66) stations should be installed near the shore. Ground control (66) stations would act like airplane control (67) towers. They would guide supertankers (68) safely on the oceans (69) along coasts. This ensures safety (70) in shipping lanes and dangerous channels. Manually create cmap Manually select terms 23
- 24. ALA-Reader (an MS Excel file) • ALA-Reader onetime setup (contrast with LSA) – Add selected terms to the MS Excel file – Correct terms for synonyms and metonyms and likely misspelling • Run lesson text and each student essay (e.g., copy text from word file paste into excel file copy excel file prx data paste into a new notepad file save notepad prx file; repeat for each essay) • Run Pathfinder KNOT on the notepad prx files • See the next two slides of the pathfinder networks (Pfnets, using the symmetric-undirected network form) of the 18 essays that Bonnie collected in her graduate class Symmetric above, asymmetric below 24(you can see why this needs to be automated)
- 25. Ngram aside: why include Exxon Valdez and not Torrey Canyon? 25 The answer to every question is ‘ngrams’
- 26. 26 Pathfinder networks of two students essays Students’ essays Networks
- 27. 27
- 28. Comparison of Pfnets: common links 28 symmetric PFNETS asymmetric PFNETS Full prob sol Full prob sol Full_lesson_text 63 41 22 74 51 23 lesson_problem 41 41 2 51 51 2 lesson_solution 22 2 22 23 2 23 pf_page 02 PS 7 5 3 5 5 1 pf_page 03 PS 4 4 1 5 5 1 pf_page 04 PS 6 5 2 3 3 0 pf_page 05 PS 8 8 1 9 9 1 pf_page 06 P 7 5 3 6 4 2 pf_page 07 PS 5 4 2 3 3 1 pf_page 08 noPS 8 8 2 10 10 2 pf_page 09 list 4 2 2 2 1 1 pf_page 10 list 3 3 1 3 3 1 pf_page 11 na 7 6 2 5 5 0 pf_page 12 na 6 5 2 6 6 1 pf_page 13 na 14 9 7 14 9 7 pf_page 14 na 8 7 3 7 6 3 pf_page 15 PS 7 7 2 7 7 2 pf_page 16 list 5 5 1 3 3 1 pf_page 17 na 3 3 1 3 3 0 pf_page 18 PS 11 10 2 9 9 1 pf_page 19 PS 6 5 3 6 6 2 Scores: the students’ networks derived from their essays are compared to different expert referents Also, it is possible to report as feedback where the students networks ‘agreed’ and ‘disagreed’ with the expert referents
- 29. Pfnet (sym) node degree as MDS 29
- 30. Published investigations on this KS essay approach 30 Koul, R., Clariana, R.B., & Salehi, R. (2005). Comparing several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32 (3), 261-273. link Taricani, E. M. & Clariana, R.B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 54, 61-78. Poindexter, M. T., & Clariana, R.B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184. Clariana, R.B., & Wallace, P. E. (2007). A computer-based approach for deriving and measuring individual and team knowledge structure from essay questions. Journal of Educational Computing Research, 37 (3), 209-225. link Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229- 236. link Clariana, R.B., Wallace, P.E., & Godshalk, V.M. (2009). Deriving and measuring group knowledge structure from essays: The effects of anaphoric reference. Educational Technology Research and Development, 57, 725-737. Clariana, R.B. (2010). Deriving group knowledge structure from semantic maps and from essays. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 7, pp. 117-130). New York, NY: Springer.
- 31. Published investigations on this KS essay approach 31 Clariana, R.B., Wolfe, M. B., & Kim, K. (2014). The influence of narrative and expository text lesson text structures on knowledge structures: alternate measures of knowledge structure. Educational Technology Research and Development, 62 (4), in press. doi: 10.1007/s11423-014-9348-3 Kim, K., & Clariana, R.B. (2015). Knowledge structure measures of reader’s situation models across languages: Translation engenders richer structure. Technology, Knowledge and Learning, 20, 249-268. (L1L2) Conference Presentations Kim, K., Clariana, R., & Mun, Y. (2014). Using Pathfinder Network as a measure of lexical structure of bilingual learners. Proceedings (full paper) of the 2014 IEEE International Conference on Advanced Learning Technologies (ICALT), Athens, Greece: IEEE Computer Society (L1L2) Kim, K., & Clariana, R. B. (2014). Concept centrality: A useful and usable analysis method to reveal mental representation of bilingual readers. Proceedings of Selected Research and Development paper of the 2014 Association for Educational Communication and Technology (AECT), Jacksonville, FL (pp. 117-124) (L1L2) Dissertations Outside PSU: Vera Chen (2012, University of Missouri), Min Kyu Kim (2012?, Georgia), Sabine Klois (2013, Radboud University Nijmegen), Ginger Howell (2014, Capella)
- 32. Published investigations on this KS essay approach 32 Thesis/dissertations at PSU Fanella, D. (2015). The effects of changing the number of terms used to create proximity files on the predictive ability of scoring essay-derived network graphs via the ALA-Reader approach. PhD dissertation, https://etda.libraries.psu.edu/paper/26367/ (KS foundations) Houston, V.C. (2014). Consequences of team charter quality: teamwork mental model similarity and team viability in engineering design student teams. PhD dissertation, https://etda.libraries.psu.edu/paper/20503/ (team collaboration) Journal, submitted, under review Kim, K. (under review). How the relationship between a heading and underline influences second language reading comprehension: Knowledge structure analysis. Manuscript submitted to the Instructional Science (L1L2) Kim, K. (under review). The influence of first language in reading a second language expository text: Knowledge structure analysis. Manuscript submitted to the Reading and Writing (L1L2) Kim, K, & Clariana, R. B. (under review). Text signals influence knowledge structure complexity of readers: Knowledge structure analysis. Manuscript submitted to the Educational Technology Research and Development (L1L2)
- 33. Presenter: Kyung Kim Doctoral Candidate in Learning, Design, & Technology 33 GISK: Graphical Interface of Structural Knowledge
- 34. I will demonstrate GISK 34
- 35. 35 Knowledge Structure for Blind Learners Presenter: JooYoung Seo Master Candidate in Learning, Design, & Technology
- 36. Project Aims 1. To extend the accessibility of the ALA-Reader. 2. To ensure equal access to KS feedback for visually impaired learners. 3. To help blind learners improve their readings and writings.
- 37. Traditional Ways MCCC :: - Montgomery County Community College banner visited Link Graphic MCCC visited Link navigation region list of 7 items Link ABOUT US Link ACADEMICS Link ADMISSIONS Link STUDENT RESOURCES Link CAMPUS LIFE Link ALUMNI AND DONORS Link ARTS list end navigation region end navigation region list of 1 items Edit search Button list end navigation region end Link Graphic login-button banner end 123456 clickable Link Graphic slideshow/251b00471f1c74a44efeb45a46eb4d84 Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1 Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1 heading level 3 Upcoming Events Blind access vs. sighted access Sequential/Linear navigation using screen reader
- 38. Traditional Ways MCCC :: - Montgomery County Community College banner visited Link Graphic MCCC visited Link navigation region list of 7 items Link ABOUT US Link ACADEMICS Link ADMISSIONS Link STUDENT RESOURCES Link CAMPUS LIFE Link ALUMNI AND DONORS Link ARTS list end navigation region end navigation region list of 1 items Edit search Button list end navigation region end Link Graphic login-button banner end 123456 clickable Link Graphic slideshow/251b00471f1c74a44efeb45a46eb4d84 Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1 Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1 heading level 3 Upcoming Events Blind access vs. sighted access Sequential/Linear navigation using screen reader
- 39. Beyond the traditions: Touch-Navigation Paper-Based Tactile Feedback Tablet-Based Haptic Feedback
- 40. How to Make it Possible? SVG Scalable Vector Graphics Tablet Based Haptic Output Paper Based Tactile Output Android haptic Feedback Sonification Swell Machine or Braille Embosser
- 41. WHY SVG? • Scalability • Accessibility • Tangibility
- 42. Scalability Retrieved from: https://www.w3.org Small SVG and PNG: Enlarged SVG and PNG:
- 43. Accessibility W3C’s recommended vector markup language. Structured images. Alternative equivalents.
- 44. Tangibility Paper-Based Tactile Feedback. Swell Touch Paper Braille Embosser/IVEO Source: www.americanthermoform.com Source: http://www.nelowvision.com Source: www.viewplus.com
- 45. Tangibility Tablet-Based Haptic Feedback Using haptic API Android
- 49. THANK YOU 49