PolyCAFe and Social Learning Support for CSCL in LTfLL
1 like488 views
This document describes PolyCAFe, a system for analyzing collaborative learning conversations. PolyCAFe was developed as part of the LTfLL project to provide automated feedback and summaries of chat discussions. It is based on theories of collaborative learning and uses natural language processing techniques like part-of-speech tagging and semantic analysis. The system was validated through experiments with students and teachers who found that it helped reduce the time to provide feedback and improved the quality of feedback for collaborative assignments.
PolyCAFe and Social Learning Support for CSCL in LTfLL
- 1. PolyCAFe and Social Learning Support for CSCL in LTfLL Stefan Trausan-Matu Traian Rebedea Mihai Dascalu Vlad Posea “Politehnica" University of Bucharest Computer Science Department 9th December 2010
- 2. General data 2 9 December 2010
- 3. The LTfLL - EU FP7 STREP Project (2008-2011) • Open Universiteit Nederland (coordinator) • The University of Manchester • Open University UK • Universiteit Utrecht • Eberhard Karls Universität Tübingen • Wirtschaftsuniversität Wien • Université Pierre-Mendès France, Grenoble • Politehnica University of Bucharest (PUB) • Institute for parallel processing of the Bulgarian Academy • Aurus Kennis- en Trainingssystemen BV • BIT MEDIA E-learning solution GMBH and CO KG 3 9 December 2010
- 4. Past International Research Projects Related to LTfLL The cognitive paradigm (e.g. Web semantic) Portable AI Lab (PAIL) – IDSIA Lugano (1994-1995) PeKADS - EU Copernicus (1994-1996) LARFLAST – EU Copernicus (1998-2001) IKF – EU EUREKA (2002-2003) Towntology – EU COST Action (2005-2008) The socio-cultural paradigm (e.g. Web2.0) VMT –USA NSF (2005-2007) EU-NCIT – EU FP6 (2005-2008) Cooper – EU FP6 (2005-2007) 4 9 December 2010
- 5. The Outcomes of the LTfLL Project Prototypes of next-generation services built on advanced research on the application of language technologies in education. http://www.ltfll-project.org 5 9 December 2010
- 6. LTfLL Services 6 9 December 2010
- 7. LTfLL Development Cycles 7 9 December 2010
- 8. Workpackages Work WP1 WP2 WP3 WP4 WP5 WP6 WP7 WP8 WP9 TOTAL per package Beneficiary OUNL 19 7 10 11 11 3 2 63 UU 3 5 5 2 18 7 3 1 44 UTU 7 6 18 6 3 1 41 WUW 10 3 4 7 4 3 1 32 UPMF 1 1 19 3 2 1 27 PUB-NCIT 1 3 19 17 8 3 1 52 AURUS 3 6 1 6 3 1 20 KTS UNIMAN 5 12 6 14 3 1 41 IPP-BAS 3 6 11 15 9 3 1 48 BIT 15 3 5 6 1 30 MEDIA OU 8 3 6 3 1 1 22 TOTAL 19 58 51 68 53 56 70 33 12 420 8 9 December 2010
- 9. PUB Implementation Team Prof.dr.ing. Stefan Trausan-Matu Prof.dr.ing. Valentin Cristea As.drd.ing. Traian Rebedea As.drd.ing. Vlad Posea As.drd.ing. Mihai Dascalu As.drd.ing. Costin Chiru As.drd.ing. Dan Mihaila Ing. Alexandru Gartner Ing. Erol Chioasca Students which helped at the implementation: Dan Banica Mihai Nicolae Iulia Pasov Ionela Voinescu Iulia Moscalenco Oana Mihai Alexandru Georgescu 9 9 December 2010
- 10. PUB Publications on LTfLL 29 papers, 12 in Proceedings ISI, 23 in international databases, 2 NLPSL Workshops with Proceedings (Eds. S. Trausan-Matu, P. Dessus) 4 book chapters (including published at Springer and Hershey) 10 9 December 2010
- 11. The Problems Solved by PolyCAFe (a Polyphony-Based System for Collaboration Analysis and Feedback Generation) 11 9 December 2010
- 12. Chat Conversations with Multiple Participants Multiple participants (≥3), conferencing style Particular features – multiple, parallel discussion chains !!! There is a need for Determining important utterances Contributions of the participants Degree of collaboration - inter-animation analysis 12 9 December 2010
- 13. Example: CSCL assignment Students had to debate in chat sessions in groups ranging from 3 to 8 In the first part of the conversation, each student had to defend a technology by presenting its features and advantages and criticize the others by invoking their flaws and drawbacks In the final part of the chat, they had to discuss on how they could integrate all these technologies in a single online collaboration platform 13 9 December 2010
- 14. CSCL assignment: Problems How to assist teachers in evaluating students’ work in chats? Offer assistance to students Abstraction tools Automatic feedback 14 9 December 2010
- 15. Experiments with Chat-based CSCL K-12 students solving mathematics problems both individually and collaboratively in the VMT project at Drexel University, Philadelphia, USA Computer Science students at Bucharest “Politehnica” University, Romania at Human-Computer Interaction course in Romanian and French – role playing and debate Natural Language Processing - role playing and debate Algorithm Design – problem solving 15 9 December 2010
- 16. The VMT chat environment 16 9 December 2010
- 17. VMT Referencing facility 17 9 December 2010
- 18. Theoretical Basis of PolyCAFe 18 9 December 2010
- 19. Paradigms about Knowledge Cognitive Socio-cultural Newell, Simon Vygotsky, Bakhtin “Knowledge is in the head” “Knowledge is in the community” Artificial Intelligence, Theory of Activity, Natural Language Processing Collaborative systems Ontologies Folksonomies Semantic Web Social Web (Web2.0) Intelligent Tutoring Systems Computer-Supported Collaborative Learning 19 9 December 2010
- 20. Computer Supported Collaborative Learning A new paradigm in learning with computers (Koshmann, 1999): Knowledge is constructed socially (Vygotsky) Induced by the spread of forums, chats, blogs, wikis and folksonomies learning in (on-line) virtual teams and/or communities 20 9 December 2010
- 21. Dialogism – Mikhail Bakhtin • Basis for the CSCL paradigm (Koschman, 1999) • “… Any true understanding is dialogic in nature” (Voloshinov-Bakhtin, 1973) • Opposed to de Saussure ideas, which are the basis for Natural Language Processing Polyphony Inter-animation of voices 21 9 December 2010
- 22. The Polyphonic Model of CSCL (Trausan-Matu, Stahl and Zemel, 2005, http://mathforum.org/wikis/uploads/Stefan_Interanimation.doc) A polyphony of voices characterizes any linguistic phenomenon (Bakhtin) including CSCL chats Inter-animation (Bakhtin, Wegerif) may be detected in interactions and it may be used for analyzing collaboration and assessing learners Integrating NLP techniques with polyphony identification and Social Network Analysis may provide a way for analyzing the contributions of each participant and their collaboration. Inter-animation and polyphony appears also in non-verbal interactions Consider threads (voices which last) rather than analyzing pairs of utterances 22 9 December 2010
- 23. Polyphonic analysis (Trausan-Matu & Stahl, 2007, http://gerrystahl.net/vmtwiki/stefan.pdf) 23 9 December 2010
- 24. 24 9 December 2010 24
- 25. Words, voices and threads Different positions assigned to participants – different voices Additional voices – frequent concepts – repeated words become voices, stronger or weaker Voices continue and influence each other through explicit or implicit links. Voices correspond to chains or threads of utterances: repeated words lexical chains co-references reasoning or argumentation rhetorical schemas 25 9 December 2010
- 26. Analysis Units in the Polyphonic Model Words Utterances Pairs of utterances (links) Threads Voices Participants 26 9 December 2010
- 27. Units of Interaction Echoes of voices Polyphonic-contrapuntal weaving Inter-animation Links between utterances and between words Links may be: implicit explicit 27 9 December 2010
- 28. Inter-animation Patterns (Trausan-Matu, Stahl & Sarmiento, 2007) Longitudinal Adjacency pairs Repetitions Elaboration Cumulative talk (collaborative utterances) Repair Transversal Convergence Differential Dissonance 28 9 December 2010
- 29. Implementation Details of PolyCAFe 29 9 December 2010
- 30. Architecture of PolyCAFE 30 9 December 2010 30
- 31. 31 9 December 2010
- 32. NLP pipe spelling correction, stemmer, tokenizer, Named Entity Recognizer, POS tagger and parser, and NP-chunker. Stanford NLP software (http://nlp.stanford.edu/software) Spellchecker : Jazzy http://www.ibm.com/developerworks/java/library/j- jazzy/ Alternative NLP pipes are under development, GATE (http://gate.ac.uk) LingPipe (http://aliasi.com/lingpipe/). 32 9 December 2010
- 33. Functional architecture Surface Analysis Readability & Page essay grading Morphological Analysis Social Network and POS Tagging Analysis Semantic Evaluation (LSA) Grading Process 33 9 December 2010
- 34. Tagged LSA • Corpus of chats focused on collaborative technologies • POS Tagging • Stemming • Segmentation – Participants – Fixed Window • Cosine similarity Term-Doc Matrix + Tf – Idf 34 9 December 2010
- 35. Vector space visualization Radial Model Physical Model 35 9 December 2010
- 36. Social Network Analysis • Degree • Centrality – Closeness – Graph – Eigen Value • User Ranking – Google Page Ranking 36 9 December 2010
- 37. Utterance evaluation Social • Degree • Semantic similarity Qualitative • Predefined topics • Overall discourse • NLP Pipe Quantitative • No of occurrences mark(u) = ∑ words ( stem ) × (1 + log( no _ occurences )) × emphasis (u ) × social (u ) length remaining emphasis (u ) = Sim (u , whole _ document ) × Sim (u , predefined _ keywords ) social (u ) = ∏ (1 + log( f (u )) all social factors f ( quantitative and qualitative ) 37 9 December 2010
- 38. Collaboration (1) • Utterance graph – Explicit links – Attenuation – Implicit links – Trust • Social cohesion • Quantitative collaboration ∑ all links l with different speakers attenuatio n(l) * trust(l) quantitati ve collaborat ion = total number of links (implicit/ explicit) 38 9 December 2010
- 39. Collaboration (2) Qualitative - Gain based collaboration = ECHO Personal – individual knowledge building Links to previous utterances with same speaker Collaborative – collaborative knowledge building Links to previous utterances with different speaker personal gain(u) = ∑ ((mark(v) + gain(v) ) * similarity(u, v) * attenuation(l) * trust(l)) link l exists between u and v, v is an earlier utterance and u and v have same speaker collaborative gain(u) = ∑ ((mark(v) + gain(v) ) * similarity(u, v) * attenuatio n(l) * trust(l)) link l exists between u and v, v is an earlier utterance and u and v have different speakers 39 9 December 2010
- 40. Collaboration (3) 40 9 December 2010
- 41. Services & Widgets PolyCAFe is an online platform: Web services (Java and PHP-based) Web widgets using W3C Widgets1.0 standard The widgets can be integrated into any web platform that has a W3C widget container (e.g. Wookie) There are services for maintenance tasks and analysis tasks (process discussion, search, etc.) The widgets have been deployed in Elgg (PLE) 41 9 December 2010
- 42. Presentation of Widgets • Conversation Feedback (content + collaboration) • Conversation Visualization (collaboration) • Utterance Feedback (content + collaboration) • Participant Feedback (content + collaboration) • Search Conversation (content) • Prepare chat/forum assignment • Add conversations for assignment 42 9 December 2010
- 43. Conversation Feedback 43 9 December 2010
- 44. Conversation Visualization 44 9 December 2010
- 45. 45 9 December 2010
- 46. 46 9 December 2010
- 47. Search Conversation 47 9 December 2010
- 48. Validation and Verification of PolyCAFe 48 9 December 2010
- 49. Validation Experiment 1.0 • Validation of PolyCAFe 1.0 – 9x students – 5x tutors, 1x teacher • Two chats with 4-5 students in a team • All the students used PolyCAFe for the first time • Only two tutors have previously used the system • Students used the automatic feedback to get insight about their discussion • Tutors used PolyCAFe to offer manual feedback 49 9 December 2010
- 50. Validation Results – Tutors The tutors validated all the instruments with suggestions to change part of them Quiz with 35 questions – all of them passed with an average score between 3.50-5.00 / 5.00 The system is relevant and useful for their activity The time for providing final feedback to the students is definitively reduced (between 30- 50%) The quality of the feedback is improved 50 9 December 2010
- 51. Validation Results – Tutors (2) Category Average Perc. agree Pedagogic effectiveness 4.11 83% Efficiency 5.00 100% Cognitive load 4.60 100% Usability 4.36 93% Satisfaction 4.57 91% Total 4.53 93% 1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree 51 9 December 2010
- 52. Validation Results – Students The students have validated most of the instruments Quiz with 32 questions – 5 where not validated; all others have scores over 3.66/5.00 In the focus group, they reported that several misleads have been found using this widgets These errors or misleads were reported to be only minor without influencing the overall feedback It has been suggested to try and fix them in order to gain the full trust of the users 52 9 December 2010
- 53. Validation Results – Students (2) For example, the students did not validate: “Overall, I believe that the support for my learning PolyCAFe (Chat Analysis and Feedback Service) provides is close enough to the current support provided by humans. ” Average = 3.11 Perc. agree = 33% However, we do not want to provide a substitute for human evaluation! 53 9 December 2010
- 54. Validation Results – Students (3) Category Average Perc. agree Pedagogic effectiveness 3.94 77% Efficiency 4.22 78% Cognitive load 3.56 56% Usability 4.11 81% Satisfaction 3.89 72% Total 3.94 73% 1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree 54 9 December 2010
- 55. Validation Results - Widgets Learners Tutors Validation Statement Agreement Agreement PolyCAFe feedback is useful 100% 100% PolyCAFe feedback is relevant 63% 80% Conversation feedback is useful 78% 80% Conversation visualisation is 89% 100% useful Utterance feedback is useful 83% 100% Participant feedback is useful 78% 100% Search conversation is useful 61% 100% 55 9 December 2010
- 56. Validation Experiment 2.0 Validation of PolyCAFe 1.5 25x students in experimental group 10x students in control group 6x tutors in experimental group 2x 7 chats with 5 students in a team Most of the students used PolyCAFe for the first time All the tutors have previously used the system Experiment is still under-way Preliminary results are very encouraging: very high correlation between ranking of participants by the system and by the participants themselves 56 9 December 2010
- 57. Verification Accuracy of system ranking of participants Accuracy of grading utterances Accuracy of determining implicit links, collaboration areas, discussion threads Accuracy for determining speech acts Accuracy for determining Model of Inquiry classes 57 9 December 2010
- 58. Example – Speech Acts Speech act - label Precision Recall Continuation 93% 92% Statement 94% 93% Greeting 100% 80% Accept 92% 80% Partial accept 71% 55% Agreement 90% 51% Understanding 96% 58% Negative 97% 78% Reject 73% 82% Partial reject 35% 27% Action directive 75% 90% Info request 100% 71% Thanks 100% 100% Maybe 100% 69% Conventional 66% 50% Personal opinion 100% 36% Sorry 66% 75% 58 9 December 2010
- 59. Conclusions and Future Work (PolyCAFe) The system is working well The system passed the validation with students and tutors The visualization widget proved to be the most useful The interface might be improved 59 9 December 2010
- 60. Transferability Issues • Domain – The topic of the conversation should be easily solved using discussions, no graphics or formulas • Language – Need for the components of the NLP pipe – Corpus for training the LSA – Maybe, a domain ontology • Activity – Collaborative activity – Teams of 4-15 students (in the current design) 60 9 December 2010
- 61. Learning Support Based on Social Networks Search and Recommendation 61 9 December 2010
- 62. Learning Scenario Learner just uses social networking web sites He connects to friends and possible tutors Our application indexes the user’s friends and resources and their peers friends and resources Based on the data acquired we offer search and recommendation services 62 9 December 2010
- 63. Tag and Social Network-based Search Learner Community Knowledge = = = Web 2.0 user Social networking application Relevant content 63 9 December 2010
- 64. Find Relevant Resources Understand the learner’s request and lead him to a resource that is both relevant and trusted as it is recommended by a member of the community 64 9 December 2010
- 65. Find Relevant Peers We search for the right person to offer feedback and guidance 65 9 December 2010
- 66. Semantic Indexing of Learning Objects • Crawled data converted in semantic formats •Vocabularies used: SIOC, Neuman’s tagging ontology, SCOT, MOAT, SKOS •*using Amazon EC2 for crawling 66 9 December 2010
- 67. Search and Recommendations Search • Search algorithm based on PageRank R (v ) R(u ) = c ∑ • Folkrank - a user is important if he annotates important (v) v∈B ( u ) N resources with relevant (well ranked) tags. • Fokrank works on a folksonomy - hypergraph - F=(V,E) where V=U ∪R ∪T and E⊂UxRxT Recommendation • Uses clustering of resources, users and tags to suggest relevant resources 67 9 December 2010
- 68. Architecture of the Application 68 9 December 2010
- 69. Using the Learning Services in a PLE Search for Search for resources users username Social networking platform Recommended Check the readings learner’s Relevant tags profile and their importance 69 9 December 2010
- 70. Conclusions and Future Work (Search and recommendations service) We use social data to provide feedback for learners We offer relevant resources to the user from his social network We help learners to find peers or tutors The work was embedded in Moodle/WebCT and validated in Bucharest and Utrecht We’re centralizing the results 70 9 December 2010