SlideShare a Scribd company logo
Computer-Based Support and Feedback for
  Collaborative Chat Conversations and
           Discussion Forums

        PhD Candidate: Traian Rebedea

     Scientific Advisor: Ștefan Trăușan-Matu
Contents
• Introduction
• Inter-animation in Online Conversations
      – Types of Interactions between Utterances
• Discourse Models for Online Conversations
      – Conversation Graph
• PolyCAFe – A Chat & Forum Analysis and Feedback
  System
• PolyCAFe Validation Experiments
• Using Semantics for Improving Relevance of Text
  Documents
• Conclusions

14 December 2012,                    PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                     Collaborative Chat Conversations and Discussion Forums
Introduction
 “Time changes all things; there is no reason why
     language should escape this universal law” (
                        de Saussure, 1915, p. 77)




14 December 2012,           PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania            Collaborative Chat Conversations and Discussion Forums
Introduction
• Evolution of online communication (and the
  linguistic artifacts): chat conversation, online
  forums, blogs, social networks
• New type of discourse
      – Highly collaborative
      – Many participants
      – Different from narratives, but also from dialogues
• Parallel discussion threads, topic shifts, new
  language, etc.
• Not investigated enough, there is no linguistic
  theory built for this type of discourse
14 December 2012,                   PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                    Collaborative Chat Conversations and Discussion Forums
Introduction
• Inter-disciplinary research topic
      – Computer Supported Collaborative Learning
        (CSCL)
      – Natural Language Processing
      – Collaborative Knowledge Building


• CSCL
      – Uses lots of online communication
      – Massive Open Online Courses (MOOCs)
14 December 2012,               PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                Collaborative Chat Conversations and Discussion Forums
Research Question
• How to design and implement a system to provide
  automatic feedback and support for learners and
  tutors involved in online discussions?
      – How to (best) analyze collaborative online discussions
        using chat and forums?
      – What elements of the proposed theoretical framework
        can be used to design a system for automatic analysis of
        chats and forums in a CSCL situation?
      – How accurate is the feedback offered by such a system?
      – Is the system efficient for the learners and tutors that are
        using it?
      – Can semantics be used in other domains as well?


14 December 2012,                       PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                        Collaborative Chat Conversations and Discussion Forums
Why?
• Which of the previous two messages does
  reply 68 continue?
      – Does Loredana continue her own message?
      – Does she continue the previous message (67)?
    66        yes, i think chat is more for brainstorming                          Loredana

    67        chat's greatest advantage is the simplicity it proposes              Alex

    68        or maybe finding out faster and direct ways of dealing               Loredana
              with a certain domain



14 December 2012,                                    PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                     Collaborative Chat Conversations and Discussion Forums
Why?




14 December 2012,      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania       Collaborative Chat Conversations and Discussion Forums
Why?
• Collaboration & parallel discussion threads




14 December 2012,          PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania           Collaborative Chat Conversations and Discussion Forums
Novelty
• Inter-animation framework for online conversation
  analysis
• Study for detecting implicit links in discussions
• Building a conversation graph using natural language
  processing (NLP) techniques
• Development of PolyCAFe
      – POLYphony-based system for Collaboration Analysis and
        Feedback generation
• Validation experiments and results
• Using Wikipedia to extract semantics that may be used
  to improve the relevance of online text documents

14 December 2012,                    PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                     Collaborative Chat Conversations and Discussion Forums
Inter-animation in Collaborative
            Chat Conversations
   “The text lives only by coming into contact with
     another text. Only at the point of this contact
      between texts does a light flash…” (Bakhtin,
                                               1986)




14 December 2012,             PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania              Collaborative Chat Conversations and Discussion Forums
Scope
• Inter-animation framework of analysis
      – Developed for multi-party online conversations
        (especially chat and other short text streams)
      – Focus on discovering interactions/links/references
        among utterances
      – Determining discussion threads
      – Compute interactions between threads and the
        degree of inter-animation or collaborative
        discourse
14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
The Utterance
• “unit of speech communication […] determined by a
  change of speaking subjects, that is, a change of
  speakers.” (Bakhtin)

• A unit of speech communication – not be related to
  existing boundaries in written text, such as phrases,
  sentences, paragraphs, etc.
• An utterance does not have a specific length (may vary
  from a word to a novel or a treaty).
• Any utterance should be meaningful on its own.
• “Responsive understanding” – other utterances are
  linked to previous ones in the same discourse.
14 December 2012,               PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                Collaborative Chat Conversations and Discussion Forums
Inter-animation
• Inter-animation: “the meaning of an utterance is
  not reducible to the intentions of the speaker or
  to the response of the addressee but emerges
  between these two.” (Wegerif, 2008)

•   Polyphony, heteroglossia, inter-animation
•   Intertextuality
•   Interdiscourse
•   Ideal text, memex, hypertext, WWW
14 December 2012,             PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania              Collaborative Chat Conversations and Discussion Forums
In Online Conversations
• Interactions or links among utterances and
  among different discussion topics
• Explicit links
      – #tag
      – @username
      – Other means for linking utterances
• Implicit links
      – Must be determined automatically

14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
Problems in Online Conversations
• Multiple parallel discussion floors
• Different discussion threads, running in parallel,
  inter-animating or competing

• “competition for attention or control of the
  discourse” (Cherny, 1999)

• Patterns that have been identified in specific
  conversations: speaker-and-supporter,
  collaborative and multiple conversational floors
14 December 2012,             PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania              Collaborative Chat Conversations and Discussion Forums
Similar Problems
• Identifying implicit links in
      • Newsgroups
      • Blog posts
      • Chats
• Thread detection in chat and chat-like
  conversations
• Thread reconstruction in discussion forums
• Uptake analysis framework
14 December 2012,             PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania              Collaborative Chat Conversations and Discussion Forums
Classification of Utterances
•   Speech acts
•   Dialog acts
•   Argumentation acts
•   Rhetoric structures
•   Transacts




14 December 2012,           PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania            Collaborative Chat Conversations and Discussion Forums
Example: Argumentation in Chat
34 Mona
              Conversation
              Ok! So... main the reason I sustain this technology is it's accesibility         CLAIM
35 Mona       Every person that knows how to type can succesfully be part of a chat            DATA
                                                                                               (LINK TO 34)
36 Cristi     indeed .... chat is everywhere                                                   DATA
                                                                                               (LINK TO 34)
37 Cristi     nowadays almost everyone uses one chat client                                    DATA
                                                                                               QUALIFIER
                                                                                               (LINK TO 34)
38 Stefan     Why do you say that? Every other techology is everywhere.                        CLAIM2
                                                                                               REBUTTAL
                                                                                               (LINK TO 34, 36)


39 Corina     true....                                                                          (LINK TO 39)
40 Mona       By accesibility I was also referring to the way you use it... chat is one of the WARRANT
              most simple means of communication nowadays                                       (LINK TO 34)
41 Stefan     that's true.                                                                     (LINK TO 40)
42 Cristi     is not very complex ... mainly is just typing                                    BACKING
                                                                                               (LINK TO 40)
43 Mona       Even a 7 year old kid can use messenger for example                              BACKING
                                                                                               (LINK TO 40)
14 December 2012,                                               PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                                Collaborative Chat Conversations and Discussion Forums
Classification of Interactions between
                  Utterances
• Interactions based on Lexical Cohesion
      – Reiterations
      – Collocations
      – Lexical chains
•   Adjacency pairs
•   Coherence relations
•   Interactions based on Semantic Similarity
•   Anaphora and Coreference Resolution
14 December 2012,            PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania             Collaborative Chat Conversations and Discussion Forums
Lexical Chains in Chats
64       and if you do not have a good internet connection, clasic chat chould also be a solution...       Mona


65       agree...videochat requires some bandwidth                                                         Cristi
66       ok...but what about information, what do you think it is the most reliable source of              Diana
         information?

67       chat is fast, simple to use and it does not require such a good internet connection               Mona
68       if it is an inside network i don't think this should be a problem                                 Stefan
69       well depends on the type of the meeting...                                                        Corina
70       what do you mean by inside network?                                                               Mona
71       in the same building?                                                                             Mona
72       LAN                                                                                               Stefan
73       nowadays a fast internet connection is everywhere :)                                              Cristi
74       or VPN                                                                                            Stefan
75       true.... internet connection is not really a problem nowadays...                                  Mona
76       so we all agree that videochat is the best solution for a "virtual" meeting                       Mona
77       undoubtedly                                                                                       Cristi
78       yes!                                                                                              Corina
79       so.... one activity solved....                                                                    Mona
80 December 2012, meetings we have video chat
14      ok so for                                                 PhD Defense - Computer-Based Support and Feedback for
                                                                                                            Stefan
Bucharest Romania                                                  Collaborative Chat Conversations and Discussion Forums
Semantic Similarity
• Knowledge-based methods
      – Dictionaries, thesauri, ontologies, other semantic
        networks (DOLCE, SUMO, Cyc)

• The length of the path between the two concepts
• The depth of the node in the IS-A taxonomy
• The lowest common ancestor (LCA or lowest super-
  ordinate – LSO)
• The information content of a concept
• The commonality and difference between the two
  concepts in the IS-A taxonomy

14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Semantic Similarity
• Corpus-based methods
      – Statistical analysis of occurrences and co-occurrences of
        words in a large collections of documents

•   PMI (Point-wise mutual information)
•   GND (Google Normalized Distance)  WWW-based
•   LSA (Latent Semantic Analysis)
•   LDA (Latent Dirichlet Allocation)
•   ESA (Explicit Semantic Analysis)  Wikipedia-based


14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Discourse Models for Web
Conversations like Chats & Forums
                    “Practically everything is a graph”

            XKCD comics:




14 December 2012,                PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                 Collaborative Chat Conversations and Discussion Forums
Types of Discourses
•   OLAC (Open Language Archives Community)

•   Dialogue: “conversations, interviews, correspondence, consultations, greetings
    and leave-takings”
•   Drama: “plays, skits, and enacted scenes”
•   Formulaic: “prayers, curses, blessings, charms, curing rituals, marriage vows, and
    oaths”
•   Ludic: “play languages, jokes, secret languages, and speech disguises”
•   Oratory: “Public speaking, or of speaking eloquently according to rules or
    conventions”
•   Narrative: “historical, traditional, and personal narratives, myths, folktales, fables,
    and humorous stories”
•   Procedural: “recipes, instructions, and plans”
•   Report: “news reports, essays, and commentaries”
•   Singing
•   Unintelligible speech


14 December 2012,                                   PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                    Collaborative Chat Conversations and Discussion Forums
Interactions/Links among Utterances
• “The base condition governing the operation
  of meaning in any utterance. It is that which
  insures the primacy of context over text. At
  any given time, in any given place, there will
  be a set of conditions – social, historical,
  meteorological, physiological – that will insure
  that a word uttered in that place and at that
  time will have a meaning different than it
  would have under any other conditions”
  (Bakhtin, 1981, p. 428)
14 December 2012,           PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania            Collaborative Chat Conversations and Discussion Forums
Modelling Conversations as a
                    Graph
• Many discourse theories define links between
  discourse units
      – Rhetorical structure
      – Coherence relations
      – Adjacency pairs
• Stavrianou: post-opinion graph for online
  forums
• Suthers: contingency graphs, uptake graph
14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
Conversation Graphs




14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
Conversation Graphs




14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
Explicit Links
• Corpus with 8 chats, 2800+ utterances,
  1900+ explicit links
              0.25



               0.2



              0.15
                                                           Distribution of explicit links


                                                           Poly. (Distribution of explicit
               0.1
                                                           links)


              0.05

                                          2
                                         R = 0.9927
                0
                     0   5     10   15            20


14 December 2012,                         PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                          Collaborative Chat Conversations and Discussion Forums
Shallow Links
• Broken utterances / split turns

• Reasons:
      – Limitations in the length of the utterance caused by the
        environment
      – Incomplete turns due to haste
      – Trying to grab the discussion floor and the attention of the
        other participants although an utterance is still not finalized
      – Lengthy turns are seldom broken into multiple shorter ones
      – Allowing the other participants to continue the utterance that is
        not finalized or even incomplete ideas


14 December 2012,                         PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                          Collaborative Chat Conversations and Discussion Forums
29      how could we forget the                                                Michael      Example 1
     30      blog                                                                   Michael      LINK TO 29
     …
     36      let's try to focus on the subject matter                               Michael      Example 2
     37      if you had to choose ONE reason                                        Michael
     38      for choosing your technology                                           Michael      LINK TO 37
     39      what would that one reason be ?                                        Michael      LINK TO 37
     …
     66      yes, i think chat is more for brainstorming                            Loredana     Example 3

     67      chat's greatest advantage is the simplicity it proposes                Alex
     68      or maybe finding out faster and direct ways of dealing with a          Loredana     LINK TO 67
             certain domain
     …
     70      true, however, in most projects you need a data format with            Michael      Example 4
             more structure
     71      and that is why i think Wave offers a good trade-off between           Michael      LINK TO 70
             structure and interactivity
     …
     75      A chat or a chat room has a more private nature than blogs or          Alex         Example 5
             waves or wikis
     …
     78      but privacy isn't really the best choice if you have an urgent         Cristi       LINK TO 75
             matter and you're looking for answers
     …
     123     of course                                                              Michael      Example 6
     124     but to make things interesting                                         Michael      LINK TO 123
     125     we're in different cities/countries/timezones                          Michael      LINK TO 124
     126     i.e. we can't be in the same room                                      Michael      LINK TO 125
14 December 2012,                                                PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                                 Collaborative Chat Conversations and Discussion Forums
Lexical Links




14 December 2012,            PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania             Collaborative Chat Conversations and Discussion Forums
Semantic Links
         0.12



          0.1



         0.08



         0.06                                                                               LSA similarity score
                                                                                            between utterances

         0.04



         0.02



           0
                1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16 17 18 19 20




14 December 2012,                                                    PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                                     Collaborative Chat Conversations and Discussion Forums
WordNet – Resnik

   0.25



    0.2



   0.15

                                                                                          WordNet similarity score
                                                                                          between utterances
    0.1



   0.05



     0
          1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16 17 18 19 20




14 December 2012,                                                PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                                 Collaborative Chat Conversations and Discussion Forums
148     ok.... so what we were talking about before the             Mona
           connection issue?
   149

   150     meeting board
                        LSA for implicit links
           as long as it depends on an internet connection...          Mona

                                                                       Cristi
                                                                                   LINK TO
                                                                                   147
                                                                                   LINK TO    score(150, 148) =
                                                                                   148        0.18 = max
   151     about different stages of a project... a client must        Corina      LINK TO
           know about them..so wiki is a good solution                             148
   152     meeting?                                                    Corina      LINK TO    score(152, 150) =
                                                                                   150        1.00 = max
   153     ok                                                          Stefan      LINK TO
                                                                                   151
   154     so we agree that wiki is a good solution when we want Corina                       score(154, 151) =
           to present a product/the evolution of a project to a                               0.5 = max
           client ?
   155     what about selling our products..what technologies we Diana
           should use for this?
   …
   166     to sell our products blog is the best solution              Corina      LINK TO score(166, 155) =
                                                                                   155     0.38 = max
   …
   169     A blog would be a good way to advertise our products        Mona        LINK TO score(169, 166) =
                                                                                   166     0.48 = max
   …
   180     A forum would be useful for offering solutions to some Mona
           problems that our customers have
   181     I agree...also other people can offer solutions, not only Corina        LINK TO score(181, 180) =
           us                                                                      181     0.27 = max
14 December 2012,                                                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                                  Collaborative Chat Conversations and Discussion Forums
How useful are semantic links for
       implicit links’ detection?
 
                             131   132   133   134      135      136      143      An5       Tot


References  greater  than 
                             0.44 0.47 0.41 0.45 0.42 0.43 0.45 0.43 0.44
10-turn median


References  greater  than 
                             0.40 0.47 0.41 0.47 0.42 0.45 0.46 0.45 0.44
20-turn median


Pearson          correlation 
between  number  of 
                              0.83 0.90 0.81 0.88 0.76 0.92 0.78 0.83 0.84
references  and  average 
similarity score



14 December 2012,                              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                               Collaborative Chat Conversations and Discussion Forums
Open Issues in Building the
               Conversation Graph
• Automatic Identification of Utterance Types
      – Not just speech acts, but also argumentation
      – Machine learning vs rule-based
• Questions and Answers in Online
  Conversations
      – Questions are easier to identify
      – Pairing answers to questions is much more
        difficult
• Conversation-Focused Coreference Resolution
14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
Developing PolyCAFe
   “In theory, theory and practice are the same. In
  practice, they are not.” (attributed to various, in
                                              press)




14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
Problems
• P1. Need to integrate feedback on different
  levels: conversation, utterance, participant
• S1. Mixture of techniques:
      – Natural Language Processing
      – Social Network Analysis
      – Information Retrieval
• P2. Supplementary, focus on measuring
  collaborative discourse as a key characteristic of
  successful online textual interaction
• S2. New theory of discourse for chat
  conversations with multiple participants
14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
Proposed Solution - PolyCAFe
• Chat and Forum Analysis and Feedback System
• PolyCAFe = Polyphony-based Collaboration Analysis
  and Feedback generation
• Provides feedback to learners, tutors and teachers
  related to the interaction of students in online
  discussions
• Takes into account both:
      – Content of the conversation (related to a domain or topic)
      – Collaboration (related to the conversation, participation,
        etc.)
14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Implicit Links in PolyCAFe
•   Repetitions (expanded using ontologies)
•   Lexical chains (using WordNet, LSA and ESA)
•   Semantic similarity (including lexical chains)
•   Adjacency pairs (using speech acts)
•   Cue phrases (“that’s a good idea”, “as <x> said”, …)




14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
Technical architecture




14 December 2012,                PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                 Collaborative Chat Conversations and Discussion Forums
Utterance Evaluation & Threads

                    Thread Coherence    Content              Future Impact
                                        Relevance

                                       Completeness

                                        Utterance



                                        Centrality

                       In-Degree         Social              Out-Degree


                       Past              Current                          Future

14 December 2012,                                    PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                     Collaborative Chat Conversations and Discussion Forums
Utterance evaluation




14 December 2012,               PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                Collaborative Chat Conversations and Discussion Forums
Lexical Chains Using ESA
   employees employe employee employer employers
   audit auditing statistics audits business
   provides provide service communication
   executives state world constitution
   companies company corporate market
   redundant information technology
   chats clients irc chat client
   comes google come news refers
   friends live life status work
   policies prfit public policy
   real complex analysis number
   programs computer software
   marketing profitability
   discussing forums forum
   personal inter-personal
   users interface window
   question ask questions
   room rooms chat-rooms
   research university
   brainstorming mind
   music song artist
14 December 2012,                                  PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                   Collaborative Chat Conversations and Discussion Forums
Conversation Visualization




14 December 2012,            PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania             Collaborative Chat Conversations and Discussion Forums
Utterance Feedback




14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
PolyCAFe Validation Experiments
          «Validation ensures that “you built the right
          thing”. Verification ensures that “you built it
             right”.» (in press and in project meetings)




14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
Validation Experiment
• 35 students part of the HCI course
      – Experimental group: 25
      – Control group: 10
• Divided in teams of 5
• Had two distinct assignments that were correlated
• 6 tutors had to provide manual feedback for the
  students: using and not using PolyCAFe



14 December 2012,                PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                 Collaborative Chat Conversations and Discussion Forums
Tutor Efficiency
• VT1: Tutors/facilitators spend less time preparing
  feedback for learners compared with traditional means
• Likert questionnaire: everyone agreed that “they find
  the information needed to write the feedback for the
  learners more quickly using PolyCAFe than without it”
  (average=4.7, agree=100%)
• Comparison between average time needed to prepare
  feedback for a conversation with and without the
  system:
      – without PolyCAFe: 84 minutes
      – with PolyCAFe: 55 minutes
      – Improvement: 35%

14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Quality and Consistency of Feedback
• VT2: Learners perceive that the feedback
  received from the system contributes to
  informing their study activities

• Logging: 285 visits to PolyCAFe and 1447
  page-views, that results in more than 40 page-
  views in average per student.


14 December 2012,          PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania           Collaborative Chat Conversations and Discussion Forums
Quality and Consistency of Feedback

                                                                      % Agree /
                    Validation statement           Average
                                                                    Strongly agree
   The information the system provides me is
   accurate enough for helping me perform                  3.7             60%
   my learning tasks.
   PolyCAFe's      feedback      is sufficiently
   accurate to inform my study activities.                 3.8             64%
   PolyCAFe provides feedback that is useful
   to my study activities.                                 3.8             72%
   PolyCAFe provides feedback that is
   relevant to my study activities.                        3.9             72%
   I trust PolyCAFe to provide helpful
   feedback.                                               4.0             80%


14 December 2012,                              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                               Collaborative Chat Conversations and Discussion Forums
Quality of Educational Output
• VT3: Learner performance in online
  discussions is improved in the areas of content
  coverage and collaboration when using
  PolyCAFe
• Measurements computed for the second chat
  assignment, by comparing experimental with
  control groups


14 December 2012,          PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania           Collaborative Chat Conversations and Discussion Forums
Quality of Educational Output
                            Experimental      Control          Improvement over
                               group           group             control group
Average score for a chat
conversation
(collaboration + content)           6.80              6.37               6.8%

Average importance of
the most important 20              0.194            0.192                1.2%
concepts
Average     number     of
utterances                          351                338               3.8%
Average distribution of
(implicit and explicit)             1.12              0.87               29%
links between utterances


14 December 2012,                          PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                           Collaborative Chat Conversations and Discussion Forums
Verification Experiments
•   Utterance scoring
•   Participant ranking
•   Speech acts classification
•   Evaluation of collaboration score
•   & more




14 December 2012,                 PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                  Collaborative Chat Conversations and Discussion Forums
Utterance Scoring
• Chat 1 (331 utterances):
• Scores: 1 (not important) – 4 (very important)

• Tutor 1–Tutor 2 (inter-rater) correlation:
  61%
• Tutor 1 – PolyCAFe correlation:            60%
• Tutor 2 – PolyCAFe correlation:            51%
• Tutor average – PolyCAFe correlation:      57%
14 December 2012,             PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania              Collaborative Chat Conversations and Discussion Forums
Participant Ranking
  Rank                Student 1 Student 2 Student 3 Student 4 Student 5

  Student 1                   -         2                  3                 1               4
  Student 2                  2          -                  3                 1               4
  Student 3                  2          3                   -                1               4
  Student 4                  1          2                  3                 -               4
  Student 5                  1          2                  4                 3                -
  Student average            2          3                  4                 1               5
  Tutor 1                    4          1                  5                 2               3
  Tutor 2                    4          2                  5                 1               3
  Tutor average              4        1-2                  5              1-2                3
  PolyCAFe                   4          2                  5                 1               3

14 December 2012,                           PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                            Collaborative Chat Conversations and Discussion Forums
Participant Ranking

          Rankings                                                    Average
                        Correlation       Precision
         compared                                                     distance

    Tutors – System        94%                77%                         0.23


   Students – System       84%                66%                         0.43


   Tutors – Students       84%                71%                         0.40



14 December 2012,                     PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                      Collaborative Chat Conversations and Discussion Forums
Using Semantics for Improving
       Relevance of Text Documents
         “All our work, our whole life is a matter of
      semantics, because words are the tools with
     which we work, the material out of which laws
          are made […]. Everything depends on our
        understanding of them.” (Felix Frankfurter)




14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
Wikipedia-Based Semantics for
        Improving Search Relevance
                Refined Rank = Rank(Google) + p * S
             score




14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Improved Semantics
    Source Concept     Destination Concept                          Score
    avl tree           self balanced binary tree                    0.703385
    avl tree           amortization analysis                        0.605471
    avl tree           donald knuth                                 0.557814
    avl tree           b tree                                       0.546726
    avl tree           binary tree                                  0.527878
    avl tree           computer science                             0.527694
    avl tree           persistent data structure                    0.500780
    avl tree           red black tree                               0.441281
    avl tree           associative array                            0.437683

14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Relevance for Ranking Video
            Comments on YouTube
• The Beatles- Here Comes The Sun
• On YouTube we have two options
1. Top comments
      – Really? Who would've thought?
      – i think the beatles are better then one derection
1. Chronologic
      – Pathetic attempt to get thumbs up
      – Why are people comparing The Beatles to all these other
        musicians? one direction and Justin Bieber are musicians
        too, but we prefer The Beatles.
      – yes of course they are?!
      –…
14 December 2012,                      PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                       Collaborative Chat Conversations and Discussion Forums
Solution




14 December 2012,         PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania          Collaborative Chat Conversations and Discussion Forums
Our results
 Comment                                                                                      Relevance
 maybe your friend should know that being english, have a picture in abbey road               662
 and "sing" all you need is love" won't make one direction? a group like the
 beatles...
 my mom said she doesn't like the beatles and she said that john was only good                968
 to look at? not to hear. my dad said, " haha so true!." i'm an orphan now.
 you shouldn't be listening to the beatles since these seem to turn your friends              983
 into enemies! beatles are all about peace!? you are not getting their message!
 please read this ! hey i know u just wanna listen to the song but i still have to            1309
 write this hoping someone will see it and that someone will care .i'm a? young
 musician from croatia so this spam is my only chance to get noticed.please check
 out my channel and i promise u won't be sorry.i appreciate your time because
 music means everything to me, thank you! ?
 i didn't mean fight other places. i meant focus on the hurt people in your own               1639
 country first, then expand to the others. if people don't agree with peace that's
 an opinion. not a fact, and people often take offense to opinions. there isn't?
 anything to take offense to, they say something that's all it is. they said it, don't
 put meaning to it. world peace - i meant the whole world having peace there

14 December 2012,                                          PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                                           Collaborative Chat Conversations and Discussion Forums
Overall Conclusions
   “I'm sorry I wrote you such a long letter; I didn't
     have time to write a short one.” (attributed to
                             Blaise Pascal, in press)

           Inter-animation, implicit links,
    theoretical framework, conversation graph,
                  PolyCAFe system,
      validation experiments, measurements,
          semantics for improved relevance
14 December 2012,              PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania               Collaborative Chat Conversations and Discussion Forums
•
•
     Conference proceedings and journal articles
                                                             Publications
     Trausan-Matu S., Dascalu M., Rebedea T. - A System for the Automatic Analysis of Computer-Supported Collaborative Learning Chats, accepted at ICALT 2012 (ISI Web of Knowledge,
     Category B in ARC Conference Ranking)
•    Rebedea T., Dascalu M., Trausan-Matu S., Armitt G., Chiru C.G. - Automatic Assessment of Collaborative Chat Conversations with PolyCAFe. EC-TEL 2011, LNCS 6964, Springer, 2011, pp.
     299-312 (ISI Web of Knowledge)
•    Dascalu M., Rebedea T., Trausan-Matu S., Armitt G. - PolyCAFe: Collaboration and Utterance Assessment for Online CSCL Conversations, in CSCL 2011 Conference Proceedings, Hong Kong,
     vol 2, pp. 781-785 (ISI Web of Knowledge, Category A in ARC Conference Ranking)
•    Banica D., Trausan-Matu S., Rebedea T. - Detecting Collaboration Regions in a Chat Session, in CSCL 2011 Conference Proceedings, vol 2, Hong Kong, pp. 771-775 (ISI Web of Knowledge,
     Category A in ARC Conference Ranking)
•    Dessus P., Trausan-Matu S., Wild F., Dupré D., Loiseau M., Rebedea T., Zampa V. - Un environnement personnel d'apprentissage évaluant des distances épistémiques et
     dialogiques. Distances & Savoirs, 9(4), 473–492
•    Chiru C., Cojocaru V., Trausan-Matu S, Rebedea T., Mihaila D. - Repetition and Rhythmicity Based Assessment for Chat Conversation. ISMIS 2011, LNAI 6804, Springer, 2011, pp 513-523.
•    (ISI Web of Knowledge, Category C in ARC Conference Ranking)
•    Wild F., Ullmann T., Scott P., Rebedea T., Hoisl, B. (2011) - Applicability of the technology acceptance model for widget-based personal learning environments. In: 1st Workshop on
     Exploring Fitness and Evolvability of Personal Learning Environments (EFEPLE'11), 2011, France, http://www.ceur-ws.org/Vol-773/EFEPLE-10-Wild.pdf
•    Trausan-Matu S., Rebedea T. - A Polyphonic Model and System for Inter-Animation Analysis in Chat Conversations with Multiple Participants, CICLING 2010, LNCS 6008, Springer, 2010,
     pp.354-363 (ISI Web of Knowledge, Category B in ARC Conference Ranking)
•    Rebedea T. et al. - Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation, EC-TEL 2010, LNCS 6383, Springer, 2010, pp. 420-425 (ISI Web
     of Knowledge)
•    Rebedea T., Posea V., Dascalu M., Trausan-Matu S. - Web Services for Supporting the Interactions of Learners in the Social Web, accepted at 9th International Conference Roedunet 2010
     (ISI Web of Knowledge)
•    Scheau C., Rebedea T., Costin C., Trausan-Matu S. - Improving the Relevance of Search Engine Results by Using Semantic Information from Wikipedia, 9th International Conference
     Roedunet 2010, pp. 151-156 (ISI Web of Knowledge)
•    Rebedea T., Trausan-Matu S. - Automatic Feedback System for Collaborative Learning Using Chats and Forums, CSEDU 2010 (ISI Web of Knowledge)
•    Chiru C.G., Cojocaru V., Rebedea T., Trausan-Matu S. - Malapropisms Detection and Correction Using a Paronyms Dictionary, a Search Engine and WordNet, ICSOFT 2010 (ISI Web of
     Knowledge, Category B in ARC Conference Ranking)
•    Chiru C.G., Hanganu A., Rebedea T., Trausan-Matu S. - Filling the Gaps Using Google 5-Grams Corpus, ICSOFT 2010 (ISI Web of Knowledge, Category B in ARC Conference Ranking)
•    Trausan-Matu S., Posea V., Rebedea T., Chiru C. - Using the Social Web to Supplement Classical Learning. In Advances in Web Based Learning – ICWL 2009, LNCS 5686, Springer, 2009, pp.
     386-389 (ISI Web of Knowledge)
•    Rebedea T., Trausan-Matu S. - Computer-Assisted Evaluation of CSCL Chat Conversations, CSCL 2009 (poster) (ISI Web of Knowledge, Category A in ARC Conference Ranking)
•    Dessus P., Trausan-Matu S., Zampa V., Rebedea T., Dascalu M. - Vers un environnement-tuteur d’apprentissage dialogique, 2e Colloque Echanger pour Apprendre en Ligne (EPAL’09),
     Grenoble : France, http://hal.archives-ouvertes.fr/hal-00404842/fr/
•    Trausan-Matu S., Rebedea T. - Ontology-Based Analyze of Chat Conversations. An Urban Development Case, Towntology Conference 2009. In Teller (ed.) Proceedings of Towntology
     Conference, Liege, 2009
•    Rebedea T., Trausan-Matu S., Chiru C.G. - Extraction of Socio-semantic Data from Chat Conversations in Collaborative Learning Communities, EC-TEL 2008. In P. Dillenbourg and M. Specht
     (Eds.): EC-TEL 2008, LNCS 5192, Springer, 2008, pp. 366–377 (ISI Web of Knowledge)
•    Rebedea T., Trausan-Matu S. - Autonomous News Clustering and Classification for an Intelligent Web Portal, ISMIS 2008. In A. An et al. (Eds.): ISMIS 2008, LNAI 4994, Springer, 2008, pp.
     477–486 (ISI Web of Knowledge, Category C in ARC Conference Ranking)
•    Trausan–Matu S., Rebedea T., Stahl G. - Polyphonic Analysis of CSCL Chats, ICLS Workshop: A Common Framework for CSCL Interaction Analysis, Utrecht, The Netherlands, 2008
14 December 2012,
•    Trausan-Matu S., Rebedea T., Dragan A., Alexandru C. - Visualisation of Learners’ Contributions in Chat Conversations, Workshop on Blended Learning at ICWL 2007. and Feedback for
                                                                                                               PhD Defense - Computer-Based Support In Fong., J., Wang, P.
BucharestBlended Learning, Pearson Prentice Hall, 2007, pp. 215-224
     (Eds.), Romania
                                                                                                            Collaborative Chat Conversations and Discussion Forums
Publications (2)
•   Book Chapters
•   Rebedea T., Trausan-Matu S., Chiru C. - Inter-animation between Utterances in Collaborative
    Chat Conversations, submitted for Innovative Methods for Electronic Discourse Analysis, IGI
    Global, 2013, to appear
•   Trausan-Matu S., Rebedea T., Dascalu M. - Analysis of discourse in collaborative Learning
    Chat Conversations with Multiple Participants, ỉn Tufis D., Forascu C. (eds.), Multilinguality
    and Interoperability in Language Processing with Emphasis on Romanian, Editura Academiei,
    2010
•   Trausan-Matu S., Rebedea T. - Polyphonic Inter-Animation of Voices in VMT, in Stahl G. (ed.),
    Studying Virtual Math Teams, Springer, 2009
•   Rebedea T., Trausan-Matu S. - Identification of Discussion Threads in Chat Conversations, in
    Trausan-Matu S. (ed.), Conversational Analysis in Online Collaborative Systems, Matrix-Rom,
    2008 (in Romanian)

•   More than 20 papers co-authored in international workshops, Romanian conference
    proceedings and journals

•   11 deliverables co-authored within the FP7 ICT Language Technologies for Lifelong Learning
14 December 2012,                                       PhD Defense - Computer-Based Support and Feedback for
     (LTfLL) project
Bucharest Romania                                        Collaborative Chat Conversations and Discussion Forums
THANK YOU!


                    Questions
                           &
                           Feedback



14 December 2012,               PhD Defense - Computer-Based Support and Feedback for
Bucharest Romania                Collaborative Chat Conversations and Discussion Forums

More Related Content

PPT
Relevance based ranking of video comments on YouTube
PPT
How useful are semantic links for the detection of implicit references in csc...
PPTX
Web services for supporting the interactions of learners in the social web - ...
PPT
Opinion mining for social media and news items in Romanian
PPTX
Detecting and Describing Historical Periods in a Large Corpora
PPTX
An Evolution of Deep Learning Models for AI2 Reasoning Challenge
PDF
A Polyphonic Model and System for Inter-animation Analysis in Chat Conversat...
PPTX
Analysis of Metadata and Topic Modeling for
Relevance based ranking of video comments on YouTube
How useful are semantic links for the detection of implicit references in csc...
Web services for supporting the interactions of learners in the social web - ...
Opinion mining for social media and news items in Romanian
Detecting and Describing Historical Periods in a Large Corpora
An Evolution of Deep Learning Models for AI2 Reasoning Challenge
A Polyphonic Model and System for Inter-animation Analysis in Chat Conversat...
Analysis of Metadata and Topic Modeling for

What's hot (20)

PDF
Lecture 2: Computational Semantics
PDF
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
PDF
Its2016 presentations-stefan-trausan
PPTX
What is word2vec?
POTX
LDA Beginner's Tutorial
PDF
Practical machine learning - Part 1
PDF
Frontiers of Natural Language Processing
PPTX
Question answering
PDF
Nlp research presentation
PPTX
Lecture 1: Semantic Analysis in Language Technology
PDF
Semantics and Computational Semantics
PPS
Extraction of Socio-Semantic Data from Chat Conversations in Collaborative Le...
PDF
Chinese Character Decomposition for Neural MT with Multi-Word Expressions
PDF
ADAPT Centre and My NLP journey: MT, MTE, QE, MWE, NER, Treebanks, Parsing.
PPTX
Word Tagging with Foundational Ontology Classes
PDF
Supervised Approach to Extract Sentiments from Unstructured Text
PDF
Question Answering - Application and Challenges
PDF
Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction...
ODP
The P4 of Networkacy
PDF
Successes and Frontiers of Deep Learning
Lecture 2: Computational Semantics
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
Its2016 presentations-stefan-trausan
What is word2vec?
LDA Beginner's Tutorial
Practical machine learning - Part 1
Frontiers of Natural Language Processing
Question answering
Nlp research presentation
Lecture 1: Semantic Analysis in Language Technology
Semantics and Computational Semantics
Extraction of Socio-Semantic Data from Chat Conversations in Collaborative Le...
Chinese Character Decomposition for Neural MT with Multi-Word Expressions
ADAPT Centre and My NLP journey: MT, MTE, QE, MWE, NER, Treebanks, Parsing.
Word Tagging with Foundational Ontology Classes
Supervised Approach to Extract Sentiments from Unstructured Text
Question Answering - Application and Challenges
Unsupervised Does Not Mean Uninterpretable: The Case for Word Sense Induction...
The P4 of Networkacy
Successes and Frontiers of Deep Learning
Ad

Viewers also liked (20)

PDF
Public PhD defense
PPTX
Languages and frameworks for specifying test artifacts
PPTX
Mobile Learning Unit - Communicating with Your Mobile Device
PPTX
Egységtesztek automatikus generálása forráskódból
PDF
Public defense PhD
PPT
PhD Defense - A Context Management Framework based on Wisdom of Crowds for So...
PPTX
Narratives of Learning: The Portfolio Approach. PhD - Public Defense Presenta...
PPTX
Single SignOn and Context Management Solutions
PPTX
Engagement in Teaching Though Teaching Presence: 2016
PPTX
PhD Public Presentation
PDF
Balanced Diversity A Portfolio Approach to Organisational Change (2012)
PDF
Reflections on the Trial of the "L'Aquila Seven"
PPTX
APPLICATIONS OF COMPUTER
PDF
Social Media and Local Governments
PPTX
Computer Security in Banking and bookkeeping
PDF
PhD Thesis Defense - Enhancing Software Quality and Quality of Experience thr...
PPTX
PPTX
Applications of computers in law enforcement and government
PPTX
Unit 4 - Referring Expressions
PPTX
Cohesive devices
Public PhD defense
Languages and frameworks for specifying test artifacts
Mobile Learning Unit - Communicating with Your Mobile Device
Egységtesztek automatikus generálása forráskódból
Public defense PhD
PhD Defense - A Context Management Framework based on Wisdom of Crowds for So...
Narratives of Learning: The Portfolio Approach. PhD - Public Defense Presenta...
Single SignOn and Context Management Solutions
Engagement in Teaching Though Teaching Presence: 2016
PhD Public Presentation
Balanced Diversity A Portfolio Approach to Organisational Change (2012)
Reflections on the Trial of the "L'Aquila Seven"
APPLICATIONS OF COMPUTER
Social Media and Local Governments
Computer Security in Banking and bookkeeping
PhD Thesis Defense - Enhancing Software Quality and Quality of Experience thr...
Applications of computers in law enforcement and government
Unit 4 - Referring Expressions
Cohesive devices
Ad

Similar to PhD Defense: Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums (20)

PPT
Detecting discourse creativity in chat conversations
PDF
DeLiddo&BuckinghamShum-e-Part2014
PPTX
Understanding Critical Elements of E-books: The Social Reading Experience of ...
PDF
Conversations in Context: A Twitter Case for Social Media Systems Design
PDF
Crowdsourcing Documentation in Software Engineering
PPTX
Thesis summary-arguments-about-deleting-wikipedia-content-paris-2013-04-19
PPTX
Restorative Practice & Educational Technologies
PPTX
The Social Semantic Server: A Flexible Framework to Support Informal Learning...
PPTX
The Social Semantic Server - A Flexible Framework to Support Informal Learnin...
PDF
Niso Annotation Webinar
PDF
Natural Language Processing
PPTX
How communities curate knowledge & how ontologists can help -Eurecom--2015-01-19
PPTX
Digital Identity and Personal Learning Networks
PDF
National Digital Learning Resources DSpace Repository Presentation
PPTX
Talking is (virtual) work -supporting online argumentation--2013-09-18 Malta ...
PDF
Lecture Polimi April2021
PDF
"It's the Conversation, Stupid!" - Social media systems design for open innov...
PPT
Leb08talksept17
PDF
Beginners chatbotai workshopdbb_bitfusion
PPT
Making Your Classes, Sing, Dance, Talk, and Talk Back!
Detecting discourse creativity in chat conversations
DeLiddo&BuckinghamShum-e-Part2014
Understanding Critical Elements of E-books: The Social Reading Experience of ...
Conversations in Context: A Twitter Case for Social Media Systems Design
Crowdsourcing Documentation in Software Engineering
Thesis summary-arguments-about-deleting-wikipedia-content-paris-2013-04-19
Restorative Practice & Educational Technologies
The Social Semantic Server: A Flexible Framework to Support Informal Learning...
The Social Semantic Server - A Flexible Framework to Support Informal Learnin...
Niso Annotation Webinar
Natural Language Processing
How communities curate knowledge & how ontologists can help -Eurecom--2015-01-19
Digital Identity and Personal Learning Networks
National Digital Learning Resources DSpace Repository Presentation
Talking is (virtual) work -supporting online argumentation--2013-09-18 Malta ...
Lecture Polimi April2021
"It's the Conversation, Stupid!" - Social media systems design for open innov...
Leb08talksept17
Beginners chatbotai workshopdbb_bitfusion
Making Your Classes, Sing, Dance, Talk, and Talk Back!

More from Traian Rebedea (20)

PDF
AI @ Wholi - Bucharest.AI Meetup #5
PDF
Deep neural networks for matching online social networking profiles
PDF
Intro to Deep Learning for Question Answering
PPT
A focused crawler for romanian words discovery
PPT
Propunere de dezvoltare a carierei universitare
PPT
Automatic plagiarism detection system for specialized corpora
PPT
Importanța algoritmilor pentru problemele de la interviuri
PPT
Automatic assessment of collaborative chat conversations with PolyCAFe - EC-T...
PPT
Conclusions and Recommendations of the Romanian ICT RTD Survey
PPT
Istoria Web-ului - part 2 - tentativ How to Web 2009
PPT
Istoria Web-ului - part 1 (2) - tentativ How to Web 2009
PPT
Istoria Web-ului - part 1 - tentativ How to Web 2009
PDF
Algorithm Design and Complexity - Course 12
PDF
Algorithm Design and Complexity - Course 11
PPT
Algorithm Design and Complexity - Course 10
PPT
Algorithm Design and Complexity - Course 9
PPT
Algorithm Design and Complexity - Course 8
PPTX
Algorithm Design and Complexity - Course 7
PPTX
Algorithm Design and Complexity - Course 6
PPTX
Algorithm Design and Complexity - Course 5
AI @ Wholi - Bucharest.AI Meetup #5
Deep neural networks for matching online social networking profiles
Intro to Deep Learning for Question Answering
A focused crawler for romanian words discovery
Propunere de dezvoltare a carierei universitare
Automatic plagiarism detection system for specialized corpora
Importanța algoritmilor pentru problemele de la interviuri
Automatic assessment of collaborative chat conversations with PolyCAFe - EC-T...
Conclusions and Recommendations of the Romanian ICT RTD Survey
Istoria Web-ului - part 2 - tentativ How to Web 2009
Istoria Web-ului - part 1 (2) - tentativ How to Web 2009
Istoria Web-ului - part 1 - tentativ How to Web 2009
Algorithm Design and Complexity - Course 12
Algorithm Design and Complexity - Course 11
Algorithm Design and Complexity - Course 10
Algorithm Design and Complexity - Course 9
Algorithm Design and Complexity - Course 8
Algorithm Design and Complexity - Course 7
Algorithm Design and Complexity - Course 6
Algorithm Design and Complexity - Course 5

Recently uploaded (20)

PPTX
master seminar digital applications in india
PDF
Module 4: Burden of Disease Tutorial Slides S2 2025
PDF
Insiders guide to clinical Medicine.pdf
PDF
Chapter 2 Heredity, Prenatal Development, and Birth.pdf
PPTX
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
PPTX
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
PPTX
Institutional Correction lecture only . . .
PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
PDF
O7-L3 Supply Chain Operations - ICLT Program
PDF
2.FourierTransform-ShortQuestionswithAnswers.pdf
PPTX
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
PDF
VCE English Exam - Section C Student Revision Booklet
PDF
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
PDF
Complications of Minimal Access Surgery at WLH
PDF
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
PDF
Black Hat USA 2025 - Micro ICS Summit - ICS/OT Threat Landscape
PDF
ANTIBIOTICS.pptx.pdf………………… xxxxxxxxxxxxx
PDF
RMMM.pdf make it easy to upload and study
PDF
Basic Mud Logging Guide for educational purpose
PDF
Anesthesia in Laparoscopic Surgery in India
master seminar digital applications in india
Module 4: Burden of Disease Tutorial Slides S2 2025
Insiders guide to clinical Medicine.pdf
Chapter 2 Heredity, Prenatal Development, and Birth.pdf
Introduction_to_Human_Anatomy_and_Physiology_for_B.Pharm.pptx
1st Inaugural Professorial Lecture held on 19th February 2020 (Governance and...
Institutional Correction lecture only . . .
102 student loan defaulters named and shamed – Is someone you know on the list?
O7-L3 Supply Chain Operations - ICLT Program
2.FourierTransform-ShortQuestionswithAnswers.pdf
PPT- ENG7_QUARTER1_LESSON1_WEEK1. IMAGERY -DESCRIPTIONS pptx.pptx
VCE English Exam - Section C Student Revision Booklet
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
Complications of Minimal Access Surgery at WLH
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
Black Hat USA 2025 - Micro ICS Summit - ICS/OT Threat Landscape
ANTIBIOTICS.pptx.pdf………………… xxxxxxxxxxxxx
RMMM.pdf make it easy to upload and study
Basic Mud Logging Guide for educational purpose
Anesthesia in Laparoscopic Surgery in India

PhD Defense: Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums

  • 1. Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums PhD Candidate: Traian Rebedea Scientific Advisor: Ștefan Trăușan-Matu
  • 2. Contents • Introduction • Inter-animation in Online Conversations – Types of Interactions between Utterances • Discourse Models for Online Conversations – Conversation Graph • PolyCAFe – A Chat & Forum Analysis and Feedback System • PolyCAFe Validation Experiments • Using Semantics for Improving Relevance of Text Documents • Conclusions 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 3. Introduction “Time changes all things; there is no reason why language should escape this universal law” ( de Saussure, 1915, p. 77) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 4. Introduction • Evolution of online communication (and the linguistic artifacts): chat conversation, online forums, blogs, social networks • New type of discourse – Highly collaborative – Many participants – Different from narratives, but also from dialogues • Parallel discussion threads, topic shifts, new language, etc. • Not investigated enough, there is no linguistic theory built for this type of discourse 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 5. Introduction • Inter-disciplinary research topic – Computer Supported Collaborative Learning (CSCL) – Natural Language Processing – Collaborative Knowledge Building • CSCL – Uses lots of online communication – Massive Open Online Courses (MOOCs) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 6. Research Question • How to design and implement a system to provide automatic feedback and support for learners and tutors involved in online discussions? – How to (best) analyze collaborative online discussions using chat and forums? – What elements of the proposed theoretical framework can be used to design a system for automatic analysis of chats and forums in a CSCL situation? – How accurate is the feedback offered by such a system? – Is the system efficient for the learners and tutors that are using it? – Can semantics be used in other domains as well? 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 7. Why? • Which of the previous two messages does reply 68 continue? – Does Loredana continue her own message? – Does she continue the previous message (67)? 66 yes, i think chat is more for brainstorming Loredana 67 chat's greatest advantage is the simplicity it proposes Alex 68 or maybe finding out faster and direct ways of dealing Loredana with a certain domain 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 8. Why? 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 9. Why? • Collaboration & parallel discussion threads 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 10. Novelty • Inter-animation framework for online conversation analysis • Study for detecting implicit links in discussions • Building a conversation graph using natural language processing (NLP) techniques • Development of PolyCAFe – POLYphony-based system for Collaboration Analysis and Feedback generation • Validation experiments and results • Using Wikipedia to extract semantics that may be used to improve the relevance of online text documents 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 11. Inter-animation in Collaborative Chat Conversations “The text lives only by coming into contact with another text. Only at the point of this contact between texts does a light flash…” (Bakhtin, 1986) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 12. Scope • Inter-animation framework of analysis – Developed for multi-party online conversations (especially chat and other short text streams) – Focus on discovering interactions/links/references among utterances – Determining discussion threads – Compute interactions between threads and the degree of inter-animation or collaborative discourse 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 13. The Utterance • “unit of speech communication […] determined by a change of speaking subjects, that is, a change of speakers.” (Bakhtin) • A unit of speech communication – not be related to existing boundaries in written text, such as phrases, sentences, paragraphs, etc. • An utterance does not have a specific length (may vary from a word to a novel or a treaty). • Any utterance should be meaningful on its own. • “Responsive understanding” – other utterances are linked to previous ones in the same discourse. 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 14. Inter-animation • Inter-animation: “the meaning of an utterance is not reducible to the intentions of the speaker or to the response of the addressee but emerges between these two.” (Wegerif, 2008) • Polyphony, heteroglossia, inter-animation • Intertextuality • Interdiscourse • Ideal text, memex, hypertext, WWW 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 15. In Online Conversations • Interactions or links among utterances and among different discussion topics • Explicit links – #tag – @username – Other means for linking utterances • Implicit links – Must be determined automatically 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 16. Problems in Online Conversations • Multiple parallel discussion floors • Different discussion threads, running in parallel, inter-animating or competing • “competition for attention or control of the discourse” (Cherny, 1999) • Patterns that have been identified in specific conversations: speaker-and-supporter, collaborative and multiple conversational floors 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 17. Similar Problems • Identifying implicit links in • Newsgroups • Blog posts • Chats • Thread detection in chat and chat-like conversations • Thread reconstruction in discussion forums • Uptake analysis framework 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 18. Classification of Utterances • Speech acts • Dialog acts • Argumentation acts • Rhetoric structures • Transacts 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 19. Example: Argumentation in Chat 34 Mona Conversation Ok! So... main the reason I sustain this technology is it's accesibility CLAIM 35 Mona Every person that knows how to type can succesfully be part of a chat DATA (LINK TO 34) 36 Cristi indeed .... chat is everywhere DATA (LINK TO 34) 37 Cristi nowadays almost everyone uses one chat client DATA QUALIFIER (LINK TO 34) 38 Stefan Why do you say that? Every other techology is everywhere. CLAIM2 REBUTTAL (LINK TO 34, 36) 39 Corina true.... (LINK TO 39) 40 Mona By accesibility I was also referring to the way you use it... chat is one of the WARRANT most simple means of communication nowadays (LINK TO 34) 41 Stefan that's true. (LINK TO 40) 42 Cristi is not very complex ... mainly is just typing BACKING (LINK TO 40) 43 Mona Even a 7 year old kid can use messenger for example BACKING (LINK TO 40) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 20. Classification of Interactions between Utterances • Interactions based on Lexical Cohesion – Reiterations – Collocations – Lexical chains • Adjacency pairs • Coherence relations • Interactions based on Semantic Similarity • Anaphora and Coreference Resolution 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 21. Lexical Chains in Chats 64 and if you do not have a good internet connection, clasic chat chould also be a solution... Mona 65 agree...videochat requires some bandwidth Cristi 66 ok...but what about information, what do you think it is the most reliable source of Diana information? 67 chat is fast, simple to use and it does not require such a good internet connection Mona 68 if it is an inside network i don't think this should be a problem Stefan 69 well depends on the type of the meeting... Corina 70 what do you mean by inside network? Mona 71 in the same building? Mona 72 LAN Stefan 73 nowadays a fast internet connection is everywhere :) Cristi 74 or VPN Stefan 75 true.... internet connection is not really a problem nowadays... Mona 76 so we all agree that videochat is the best solution for a "virtual" meeting Mona 77 undoubtedly Cristi 78 yes! Corina 79 so.... one activity solved.... Mona 80 December 2012, meetings we have video chat 14 ok so for PhD Defense - Computer-Based Support and Feedback for Stefan Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 22. Semantic Similarity • Knowledge-based methods – Dictionaries, thesauri, ontologies, other semantic networks (DOLCE, SUMO, Cyc) • The length of the path between the two concepts • The depth of the node in the IS-A taxonomy • The lowest common ancestor (LCA or lowest super- ordinate – LSO) • The information content of a concept • The commonality and difference between the two concepts in the IS-A taxonomy 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 23. Semantic Similarity • Corpus-based methods – Statistical analysis of occurrences and co-occurrences of words in a large collections of documents • PMI (Point-wise mutual information) • GND (Google Normalized Distance)  WWW-based • LSA (Latent Semantic Analysis) • LDA (Latent Dirichlet Allocation) • ESA (Explicit Semantic Analysis)  Wikipedia-based 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 24. Discourse Models for Web Conversations like Chats & Forums “Practically everything is a graph” XKCD comics: 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 25. Types of Discourses • OLAC (Open Language Archives Community) • Dialogue: “conversations, interviews, correspondence, consultations, greetings and leave-takings” • Drama: “plays, skits, and enacted scenes” • Formulaic: “prayers, curses, blessings, charms, curing rituals, marriage vows, and oaths” • Ludic: “play languages, jokes, secret languages, and speech disguises” • Oratory: “Public speaking, or of speaking eloquently according to rules or conventions” • Narrative: “historical, traditional, and personal narratives, myths, folktales, fables, and humorous stories” • Procedural: “recipes, instructions, and plans” • Report: “news reports, essays, and commentaries” • Singing • Unintelligible speech 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 26. Interactions/Links among Utterances • “The base condition governing the operation of meaning in any utterance. It is that which insures the primacy of context over text. At any given time, in any given place, there will be a set of conditions – social, historical, meteorological, physiological – that will insure that a word uttered in that place and at that time will have a meaning different than it would have under any other conditions” (Bakhtin, 1981, p. 428) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 27. Modelling Conversations as a Graph • Many discourse theories define links between discourse units – Rhetorical structure – Coherence relations – Adjacency pairs • Stavrianou: post-opinion graph for online forums • Suthers: contingency graphs, uptake graph 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 28. Conversation Graphs 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 29. Conversation Graphs 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 30. Explicit Links • Corpus with 8 chats, 2800+ utterances, 1900+ explicit links 0.25 0.2 0.15 Distribution of explicit links Poly. (Distribution of explicit 0.1 links) 0.05 2 R = 0.9927 0 0 5 10 15 20 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 31. Shallow Links • Broken utterances / split turns • Reasons: – Limitations in the length of the utterance caused by the environment – Incomplete turns due to haste – Trying to grab the discussion floor and the attention of the other participants although an utterance is still not finalized – Lengthy turns are seldom broken into multiple shorter ones – Allowing the other participants to continue the utterance that is not finalized or even incomplete ideas 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 32. 29 how could we forget the Michael Example 1 30 blog Michael LINK TO 29 … 36 let's try to focus on the subject matter Michael Example 2 37 if you had to choose ONE reason Michael 38 for choosing your technology Michael LINK TO 37 39 what would that one reason be ? Michael LINK TO 37 … 66 yes, i think chat is more for brainstorming Loredana Example 3 67 chat's greatest advantage is the simplicity it proposes Alex 68 or maybe finding out faster and direct ways of dealing with a Loredana LINK TO 67 certain domain … 70 true, however, in most projects you need a data format with Michael Example 4 more structure 71 and that is why i think Wave offers a good trade-off between Michael LINK TO 70 structure and interactivity … 75 A chat or a chat room has a more private nature than blogs or Alex Example 5 waves or wikis … 78 but privacy isn't really the best choice if you have an urgent Cristi LINK TO 75 matter and you're looking for answers … 123 of course Michael Example 6 124 but to make things interesting Michael LINK TO 123 125 we're in different cities/countries/timezones Michael LINK TO 124 126 i.e. we can't be in the same room Michael LINK TO 125 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 33. Lexical Links 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 34. Semantic Links 0.12 0.1 0.08 0.06 LSA similarity score between utterances 0.04 0.02 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 35. WordNet – Resnik 0.25 0.2 0.15 WordNet similarity score between utterances 0.1 0.05 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 36. 148 ok.... so what we were talking about before the Mona connection issue? 149 150 meeting board LSA for implicit links as long as it depends on an internet connection... Mona Cristi LINK TO 147 LINK TO score(150, 148) = 148 0.18 = max 151 about different stages of a project... a client must Corina LINK TO know about them..so wiki is a good solution 148 152 meeting? Corina LINK TO score(152, 150) = 150 1.00 = max 153 ok Stefan LINK TO 151 154 so we agree that wiki is a good solution when we want Corina score(154, 151) = to present a product/the evolution of a project to a 0.5 = max client ? 155 what about selling our products..what technologies we Diana should use for this? … 166 to sell our products blog is the best solution Corina LINK TO score(166, 155) = 155 0.38 = max … 169 A blog would be a good way to advertise our products Mona LINK TO score(169, 166) = 166 0.48 = max … 180 A forum would be useful for offering solutions to some Mona problems that our customers have 181 I agree...also other people can offer solutions, not only Corina LINK TO score(181, 180) = us 181 0.27 = max 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 37. How useful are semantic links for implicit links’ detection?   131 132 133 134 135 136 143 An5 Tot References  greater  than  0.44 0.47 0.41 0.45 0.42 0.43 0.45 0.43 0.44 10-turn median References  greater  than  0.40 0.47 0.41 0.47 0.42 0.45 0.46 0.45 0.44 20-turn median Pearson  correlation  between  number  of  0.83 0.90 0.81 0.88 0.76 0.92 0.78 0.83 0.84 references  and  average  similarity score 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 38. Open Issues in Building the Conversation Graph • Automatic Identification of Utterance Types – Not just speech acts, but also argumentation – Machine learning vs rule-based • Questions and Answers in Online Conversations – Questions are easier to identify – Pairing answers to questions is much more difficult • Conversation-Focused Coreference Resolution 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 39. Developing PolyCAFe “In theory, theory and practice are the same. In practice, they are not.” (attributed to various, in press) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 40. Problems • P1. Need to integrate feedback on different levels: conversation, utterance, participant • S1. Mixture of techniques: – Natural Language Processing – Social Network Analysis – Information Retrieval • P2. Supplementary, focus on measuring collaborative discourse as a key characteristic of successful online textual interaction • S2. New theory of discourse for chat conversations with multiple participants 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 41. Proposed Solution - PolyCAFe • Chat and Forum Analysis and Feedback System • PolyCAFe = Polyphony-based Collaboration Analysis and Feedback generation • Provides feedback to learners, tutors and teachers related to the interaction of students in online discussions • Takes into account both: – Content of the conversation (related to a domain or topic) – Collaboration (related to the conversation, participation, etc.) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 42. Implicit Links in PolyCAFe • Repetitions (expanded using ontologies) • Lexical chains (using WordNet, LSA and ESA) • Semantic similarity (including lexical chains) • Adjacency pairs (using speech acts) • Cue phrases (“that’s a good idea”, “as <x> said”, …) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 43. Technical architecture 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 44. Utterance Evaluation & Threads Thread Coherence Content Future Impact Relevance Completeness Utterance Centrality In-Degree Social Out-Degree Past Current Future 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 45. Utterance evaluation 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 46. Lexical Chains Using ESA employees employe employee employer employers audit auditing statistics audits business provides provide service communication executives state world constitution companies company corporate market redundant information technology chats clients irc chat client comes google come news refers friends live life status work policies prfit public policy real complex analysis number programs computer software marketing profitability discussing forums forum personal inter-personal users interface window question ask questions room rooms chat-rooms research university brainstorming mind music song artist 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 47. Conversation Visualization 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 48. Utterance Feedback 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 49. PolyCAFe Validation Experiments «Validation ensures that “you built the right thing”. Verification ensures that “you built it right”.» (in press and in project meetings) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 50. Validation Experiment • 35 students part of the HCI course – Experimental group: 25 – Control group: 10 • Divided in teams of 5 • Had two distinct assignments that were correlated • 6 tutors had to provide manual feedback for the students: using and not using PolyCAFe 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 51. Tutor Efficiency • VT1: Tutors/facilitators spend less time preparing feedback for learners compared with traditional means • Likert questionnaire: everyone agreed that “they find the information needed to write the feedback for the learners more quickly using PolyCAFe than without it” (average=4.7, agree=100%) • Comparison between average time needed to prepare feedback for a conversation with and without the system: – without PolyCAFe: 84 minutes – with PolyCAFe: 55 minutes – Improvement: 35% 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 52. Quality and Consistency of Feedback • VT2: Learners perceive that the feedback received from the system contributes to informing their study activities • Logging: 285 visits to PolyCAFe and 1447 page-views, that results in more than 40 page- views in average per student. 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 53. Quality and Consistency of Feedback % Agree / Validation statement Average Strongly agree The information the system provides me is accurate enough for helping me perform 3.7 60% my learning tasks. PolyCAFe's feedback is sufficiently accurate to inform my study activities. 3.8 64% PolyCAFe provides feedback that is useful to my study activities. 3.8 72% PolyCAFe provides feedback that is relevant to my study activities. 3.9 72% I trust PolyCAFe to provide helpful feedback. 4.0 80% 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 54. Quality of Educational Output • VT3: Learner performance in online discussions is improved in the areas of content coverage and collaboration when using PolyCAFe • Measurements computed for the second chat assignment, by comparing experimental with control groups 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 55. Quality of Educational Output Experimental Control Improvement over group group control group Average score for a chat conversation (collaboration + content) 6.80 6.37 6.8% Average importance of the most important 20 0.194 0.192 1.2% concepts Average number of utterances 351 338 3.8% Average distribution of (implicit and explicit) 1.12 0.87 29% links between utterances 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 56. Verification Experiments • Utterance scoring • Participant ranking • Speech acts classification • Evaluation of collaboration score • & more 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 57. Utterance Scoring • Chat 1 (331 utterances): • Scores: 1 (not important) – 4 (very important) • Tutor 1–Tutor 2 (inter-rater) correlation: 61% • Tutor 1 – PolyCAFe correlation: 60% • Tutor 2 – PolyCAFe correlation: 51% • Tutor average – PolyCAFe correlation: 57% 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 58. Participant Ranking Rank Student 1 Student 2 Student 3 Student 4 Student 5 Student 1 - 2 3 1 4 Student 2 2 - 3 1 4 Student 3 2 3 - 1 4 Student 4 1 2 3 - 4 Student 5 1 2 4 3 - Student average 2 3 4 1 5 Tutor 1 4 1 5 2 3 Tutor 2 4 2 5 1 3 Tutor average 4 1-2 5 1-2 3 PolyCAFe 4 2 5 1 3 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 59. Participant Ranking Rankings Average Correlation Precision compared distance Tutors – System 94% 77% 0.23 Students – System 84% 66% 0.43 Tutors – Students 84% 71% 0.40 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 60. Using Semantics for Improving Relevance of Text Documents “All our work, our whole life is a matter of semantics, because words are the tools with which we work, the material out of which laws are made […]. Everything depends on our understanding of them.” (Felix Frankfurter) 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 61. Wikipedia-Based Semantics for Improving Search Relevance Refined Rank = Rank(Google) + p * S score 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 62. Improved Semantics Source Concept Destination Concept Score avl tree self balanced binary tree 0.703385 avl tree amortization analysis 0.605471 avl tree donald knuth 0.557814 avl tree b tree 0.546726 avl tree binary tree 0.527878 avl tree computer science 0.527694 avl tree persistent data structure 0.500780 avl tree red black tree 0.441281 avl tree associative array 0.437683 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 63. Relevance for Ranking Video Comments on YouTube • The Beatles- Here Comes The Sun • On YouTube we have two options 1. Top comments – Really? Who would've thought? – i think the beatles are better then one derection 1. Chronologic – Pathetic attempt to get thumbs up – Why are people comparing The Beatles to all these other musicians? one direction and Justin Bieber are musicians too, but we prefer The Beatles. – yes of course they are?! –… 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 64. Solution 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 65. Our results Comment Relevance maybe your friend should know that being english, have a picture in abbey road 662 and "sing" all you need is love" won't make one direction? a group like the beatles... my mom said she doesn't like the beatles and she said that john was only good 968 to look at? not to hear. my dad said, " haha so true!." i'm an orphan now. you shouldn't be listening to the beatles since these seem to turn your friends 983 into enemies! beatles are all about peace!? you are not getting their message! please read this ! hey i know u just wanna listen to the song but i still have to 1309 write this hoping someone will see it and that someone will care .i'm a? young musician from croatia so this spam is my only chance to get noticed.please check out my channel and i promise u won't be sorry.i appreciate your time because music means everything to me, thank you! ? i didn't mean fight other places. i meant focus on the hurt people in your own 1639 country first, then expand to the others. if people don't agree with peace that's an opinion. not a fact, and people often take offense to opinions. there isn't? anything to take offense to, they say something that's all it is. they said it, don't put meaning to it. world peace - i meant the whole world having peace there 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 66. Overall Conclusions “I'm sorry I wrote you such a long letter; I didn't have time to write a short one.” (attributed to Blaise Pascal, in press) Inter-animation, implicit links, theoretical framework, conversation graph, PolyCAFe system, validation experiments, measurements, semantics for improved relevance 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 67. • • Conference proceedings and journal articles Publications Trausan-Matu S., Dascalu M., Rebedea T. - A System for the Automatic Analysis of Computer-Supported Collaborative Learning Chats, accepted at ICALT 2012 (ISI Web of Knowledge, Category B in ARC Conference Ranking) • Rebedea T., Dascalu M., Trausan-Matu S., Armitt G., Chiru C.G. - Automatic Assessment of Collaborative Chat Conversations with PolyCAFe. EC-TEL 2011, LNCS 6964, Springer, 2011, pp. 299-312 (ISI Web of Knowledge) • Dascalu M., Rebedea T., Trausan-Matu S., Armitt G. - PolyCAFe: Collaboration and Utterance Assessment for Online CSCL Conversations, in CSCL 2011 Conference Proceedings, Hong Kong, vol 2, pp. 781-785 (ISI Web of Knowledge, Category A in ARC Conference Ranking) • Banica D., Trausan-Matu S., Rebedea T. - Detecting Collaboration Regions in a Chat Session, in CSCL 2011 Conference Proceedings, vol 2, Hong Kong, pp. 771-775 (ISI Web of Knowledge, Category A in ARC Conference Ranking) • Dessus P., Trausan-Matu S., Wild F., Dupré D., Loiseau M., Rebedea T., Zampa V. - Un environnement personnel d'apprentissage évaluant des distances épistémiques et dialogiques. Distances & Savoirs, 9(4), 473–492 • Chiru C., Cojocaru V., Trausan-Matu S, Rebedea T., Mihaila D. - Repetition and Rhythmicity Based Assessment for Chat Conversation. ISMIS 2011, LNAI 6804, Springer, 2011, pp 513-523. • (ISI Web of Knowledge, Category C in ARC Conference Ranking) • Wild F., Ullmann T., Scott P., Rebedea T., Hoisl, B. (2011) - Applicability of the technology acceptance model for widget-based personal learning environments. In: 1st Workshop on Exploring Fitness and Evolvability of Personal Learning Environments (EFEPLE'11), 2011, France, http://www.ceur-ws.org/Vol-773/EFEPLE-10-Wild.pdf • Trausan-Matu S., Rebedea T. - A Polyphonic Model and System for Inter-Animation Analysis in Chat Conversations with Multiple Participants, CICLING 2010, LNCS 6008, Springer, 2010, pp.354-363 (ISI Web of Knowledge, Category B in ARC Conference Ranking) • Rebedea T. et al. - Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation, EC-TEL 2010, LNCS 6383, Springer, 2010, pp. 420-425 (ISI Web of Knowledge) • Rebedea T., Posea V., Dascalu M., Trausan-Matu S. - Web Services for Supporting the Interactions of Learners in the Social Web, accepted at 9th International Conference Roedunet 2010 (ISI Web of Knowledge) • Scheau C., Rebedea T., Costin C., Trausan-Matu S. - Improving the Relevance of Search Engine Results by Using Semantic Information from Wikipedia, 9th International Conference Roedunet 2010, pp. 151-156 (ISI Web of Knowledge) • Rebedea T., Trausan-Matu S. - Automatic Feedback System for Collaborative Learning Using Chats and Forums, CSEDU 2010 (ISI Web of Knowledge) • Chiru C.G., Cojocaru V., Rebedea T., Trausan-Matu S. - Malapropisms Detection and Correction Using a Paronyms Dictionary, a Search Engine and WordNet, ICSOFT 2010 (ISI Web of Knowledge, Category B in ARC Conference Ranking) • Chiru C.G., Hanganu A., Rebedea T., Trausan-Matu S. - Filling the Gaps Using Google 5-Grams Corpus, ICSOFT 2010 (ISI Web of Knowledge, Category B in ARC Conference Ranking) • Trausan-Matu S., Posea V., Rebedea T., Chiru C. - Using the Social Web to Supplement Classical Learning. In Advances in Web Based Learning – ICWL 2009, LNCS 5686, Springer, 2009, pp. 386-389 (ISI Web of Knowledge) • Rebedea T., Trausan-Matu S. - Computer-Assisted Evaluation of CSCL Chat Conversations, CSCL 2009 (poster) (ISI Web of Knowledge, Category A in ARC Conference Ranking) • Dessus P., Trausan-Matu S., Zampa V., Rebedea T., Dascalu M. - Vers un environnement-tuteur d’apprentissage dialogique, 2e Colloque Echanger pour Apprendre en Ligne (EPAL’09), Grenoble : France, http://hal.archives-ouvertes.fr/hal-00404842/fr/ • Trausan-Matu S., Rebedea T. - Ontology-Based Analyze of Chat Conversations. An Urban Development Case, Towntology Conference 2009. In Teller (ed.) Proceedings of Towntology Conference, Liege, 2009 • Rebedea T., Trausan-Matu S., Chiru C.G. - Extraction of Socio-semantic Data from Chat Conversations in Collaborative Learning Communities, EC-TEL 2008. In P. Dillenbourg and M. Specht (Eds.): EC-TEL 2008, LNCS 5192, Springer, 2008, pp. 366–377 (ISI Web of Knowledge) • Rebedea T., Trausan-Matu S. - Autonomous News Clustering and Classification for an Intelligent Web Portal, ISMIS 2008. In A. An et al. (Eds.): ISMIS 2008, LNAI 4994, Springer, 2008, pp. 477–486 (ISI Web of Knowledge, Category C in ARC Conference Ranking) • Trausan–Matu S., Rebedea T., Stahl G. - Polyphonic Analysis of CSCL Chats, ICLS Workshop: A Common Framework for CSCL Interaction Analysis, Utrecht, The Netherlands, 2008 14 December 2012, • Trausan-Matu S., Rebedea T., Dragan A., Alexandru C. - Visualisation of Learners’ Contributions in Chat Conversations, Workshop on Blended Learning at ICWL 2007. and Feedback for PhD Defense - Computer-Based Support In Fong., J., Wang, P. BucharestBlended Learning, Pearson Prentice Hall, 2007, pp. 215-224 (Eds.), Romania Collaborative Chat Conversations and Discussion Forums
  • 68. Publications (2) • Book Chapters • Rebedea T., Trausan-Matu S., Chiru C. - Inter-animation between Utterances in Collaborative Chat Conversations, submitted for Innovative Methods for Electronic Discourse Analysis, IGI Global, 2013, to appear • Trausan-Matu S., Rebedea T., Dascalu M. - Analysis of discourse in collaborative Learning Chat Conversations with Multiple Participants, ỉn Tufis D., Forascu C. (eds.), Multilinguality and Interoperability in Language Processing with Emphasis on Romanian, Editura Academiei, 2010 • Trausan-Matu S., Rebedea T. - Polyphonic Inter-Animation of Voices in VMT, in Stahl G. (ed.), Studying Virtual Math Teams, Springer, 2009 • Rebedea T., Trausan-Matu S. - Identification of Discussion Threads in Chat Conversations, in Trausan-Matu S. (ed.), Conversational Analysis in Online Collaborative Systems, Matrix-Rom, 2008 (in Romanian) • More than 20 papers co-authored in international workshops, Romanian conference proceedings and journals • 11 deliverables co-authored within the FP7 ICT Language Technologies for Lifelong Learning 14 December 2012, PhD Defense - Computer-Based Support and Feedback for (LTfLL) project Bucharest Romania Collaborative Chat Conversations and Discussion Forums
  • 69. THANK YOU! Questions & Feedback 14 December 2012, PhD Defense - Computer-Based Support and Feedback for Bucharest Romania Collaborative Chat Conversations and Discussion Forums

Editor's Notes

  • #14: Utterance = interventia
  • #15: “ Within the arena of almost every utterance an intense interaction and struggle between one&apos;s own and another&apos;s word is being waged, a process in which they oppose or dialogically interanimate each other.” (Bakhtin, 1981)
  • #72: A fost un drum lung, interesant si cateodata anevoios, Insa am cunoscut multi oameni extraordinari, cu care am mers mai departe – colegi din facultate, din proiecte, studenti Lor trebuie sa le multumesc, dar si familiei si prietenilor, dl.prof. Trausan pentru sfaturile primite in ultimii multi ani, precum si comisiei de doctorat pentru efortul intreprins in a intelege si imbunatati anumite aspecte ale tezei.