Java parser a fine grained indexing tool and its application
Download as PPTX, PDF2 likes596 views
1) The document discusses an intelligent tutoring system called Knowledge Maximizer (KM) that uses fine-grained modeling of programming concepts to provide personalized exam preparation for students. 2) An evaluation of KM compared to other course tutoring systems found that students using KM attempted more complex questions, had a higher success rate, and achieved greater knowledge gains. 3) An automatic tool called JavaParser was developed to index programming questions by the underlying concepts, achieving over 98% accuracy compared to manual indexing. Future work aims to improve modeling and provide more adaptive support.
Java parser a fine grained indexing tool and its application
- 1. University of Pittsburgh Intelligent Systems Program
- 2. Outline • Introduction • Hypothesis testing (pre-study) • Findings and challenges • JavaParser • Conclusion &Future works 2AIEDCS 2013
- 3. Introduction There are two possible ways for modeling student’s knowledge: • Coarse-grained knowledge modeling • Fine-grained knowledge modeling 3AIEDCS 2013
- 4. Motivation Fine-grained indexing? Do we really need it ? ... 4AIEDCS 2013
- 5. • An important aspect of task sequencing in Adaptive Hypermedia is the granularity of the domain model and the task indexing. • In general, the sequencing algorithm can better determine the appropriate task if the granularity of the domain model and the task indexing is finer. • However, fine-grained domain models that dissect a domain into dozens or hundreds of knowledge units are much harder to develop and to use for indexing. 5AIEDCS 2013
- 6. • Typical approach to present programming knowledge which uses coarse-grained topics like “loops” and “increment” allows reasonable sequencing during course. (Brusilovsky et al. 2009, Hsiao et al. 2010, Kavcic 2004, Vesin et al. 2012) • However, this approach fails in providing advance sequencing such as providing support for exam preparation or remediation. 6AIEDCS 2013
- 7. Outline • Introduction • Hypothesis testing (pre-study) • Findings and challenges • JavaParser • Conclusion &Future works 7AIEDCS 2013
- 8. Pre-study Knowledge Maximizer: • An exam preparation tool for Java programming • based on a fine-grained concept model of Java knowledge • assumes a student already completed a considerable amount of work • goal is to help her define gaps in knowledge and try to redress them as soon as possible. 8AIEDCS 2013
- 10. KM Parameters: • How prepared is the student to do the activity? 12 r r M i i M i ii w wk K K: user knowledge level Mr :prerequisite concepts ki :knowledge in Ci wi : weight of Ci for activity AIEDCS 2013
- 11. • How prepared is the student to do the activity? • What is the impact of the activity? 13 o o M i i M i ii w wk I )1( I : activity impact Mo : outcome concepts AIEDCS 2013
- 12. • How prepared is the student to do the activity? • What is the impact of the activity? • What is the value of repeating the activity again? 14 1 1 t s S : inverse success rate for the activityS S : number of success in the activity t : number of times the activity is done AIEDCS 2013
- 13. • Determining the sequence of top 10 activity with the highest rank using: 15 SIKR R : activity rank K : knowledge level in prerequisites of activity I : activity Impact : Inverse of success rateS AIEDCS 2013
- 14. Evaluation • We conducted a classroom study for the Java Programming undergraduate course. • Study started on Dec. 4th 2012 about a week before the final exam. • The course also used QuizGuide (QG), and Progressor+ (P+) systems to access Java questions (available from the beginning of the semester). • All these systems used same 103 parameterized Java questions. 16AIEDCS 2013
- 15. Evaluation Measures We grouped participants into two groups: • KM: those who made at least ten attempt using KM (n = 9) • QG/P+: those who made no attempt using KM and at least 10 attempt with QG/P+ (n = 16) For each group we measured: • Number of questions (attempts) done using each system • Success Rate 17AIEDCS 2013
- 16. Outline • Introduction • Hypothesis testing (pre-study) • Findings and challenges • JavaParser • Conclusion &Future works 18AIEDCS 2013
- 17. System Usage Summary 19 6.20% 43.50% 50.20% 34.60% 45.30% 20.10% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Easy Moderate Complex Attempts per Question Compelixty KM QG,P+ AIEDCS 2013
- 18. Results 20 58% 64% 0% 50% 100% KM QG,P+ Success rate AIEDCS 2013 -12.80% -13.70% 11.53% -15.00% -10.00% -5.00% 0.00% 5.00% 10.00% 15.00% Class QG/P+ KM Average Relative Progress Percentage
- 19. • Fine-grained knowledge modeling will push students toward appropriate complex questions which does not lead to miserable fails in those questions . • During exam preparation, complex questions are more useful, since they target more concepts at once. • This helps students fill the gaps in their knowledge in a more efficient way. (Eg. : 6 easy question must be done to get the same outcome as only 1 complex question!) 22AIEDCS 2013
- 20. JAVA Ontology http://www.sis.pitt.edu/~paws/ont/java.owl 23AIEDCS 2013 ~ 400 node in ontology ~ 160 concept (leaf nodes)
- 21. 24AIEDCS 2013 103 JAVA Parameterized Question 1-5 classes per question 5-52 concepts per question ~ 40% of questions has more than 20 concepts
- 22. 25AIEDCS 2013 52 concepts or more in a question ?
- 23. 26AIEDCS 2013 JavaParser: A tool for automatic indexing of Java Problems
- 24. Outline • Introduction • Hypothesis testing (pre-study) • Findings and challenges • JavaParser • Conclusion &Future works 27AIEDCS 2013
- 25. Java Parser • Developed using Eclipse AST Tree API • The AST tree is semantically analyzed using the information in each of its nodes. 28 1 2 3 AIEDCS 2013
- 26. Modifiers Name Parameters Body Structural properties of a method declaration 29 Exceptions Return Type
- 32. • Current version of JavaParser is able to extract 98.77% of the concepts in 103 manually indexed questions. • Average number of 8 extra concepts for each question indexed by automatic parsing. 35AIEDCS 2013
- 33. Missed concept • InheritanceBasedPolymorphism • SuperclassSubclassConversion • PolymorphicObjectCreationStatement • MethodInheritance • …. AIEDCS 2013 36
- 35. Outline • Introduction • Hypothesis testing (pre-study) • Findings and challenges • JavaParser • Conclusion &Future works 43AIEDCS 2013
- 36. Future work Use the results of fine-indexing for: • Improving the user modeling service • cross content sequencing and providing remediation in case of failing a question. • Predicting parts of code that might lead to student failure and provide hints accordingly. • Expand the parser to extract more elaborated concepts and programming patterns . 44AIEDCS 2013
Editor's Notes
- #2: 46
- #4: Coarse-grained knowledge modeling: modeling large chunks of knowledge (typically topics) Fine-grained knowledge modeling: modeling small piece of knowledge (typically concepts)
- #6: One of the oldest functions of adaptive educational systems is guiding students to the most appropriate educational problems at any time during their learning process. In classic ICAI and ITS systems, this function was known as task sequencing
- #7: In these special situations, students might have a reasonable overall understanding of the content (i.e., coarse-grained student model registers good level of knowledge), while still suffering some knowledge gaps and misconceptions that could only be registered using a finer-grained student model. In this situation, only a fine-grained indexing and sequencing tool is able to suggest learning tasks that can address these gaps and misconceptions.
- #8: To demonstrate the importance of fine-grained indexing, we look to a system called Knowledge Maximizer [3] that uses fine-grained concept-level problem indexing to identify gaps in user knowledge for exam preparation.
- #9: Comprises 103 parameterized questions (activity) indexed by ontology concepts.
- #11: Goal is to recommend sequence of questions with the highest gain to the learner which will help her to complete gaps in her knowledge.
- #20: Further Analyzing of number of attempts per questions complexity : Easy (15 or less concepts ) Moderate (16 to 90 concepts) Complex (more that 90 concepts) the total number of attempts made from JavaGuide/Progressor+ was much larger, which is natural since the students were familiar with JavaGuide and Progressor+ from the beginning of the class. Yet, it is quite remarkable that KZ, which was introduced just a week before the exam, was considerably used. the success rates across all systems were comparable. Despite a remarkable increase in complex questions, the success rates across all systems were comparable. Much larger fraction of students’ attempts in KZ has been done with complex questions. With complex questions students can check more concepts at once and move to their goal with much larger steps.
- #21: the total number of attempts made from JavaGuide/Progressor+ was much larger, which is natural since the students were familiar with JavaGuide and Progressor+ from the beginning of the class. Yet, it is quite remarkable that KZ, which was introduced just a week before the exam, was considerably used. the success rates across all systems were comparable. Despite a remarkable increase in complex questions, the success rates across all systems were comparable. Much larger fraction of students’ attempts in KZ has been done with complex questions. With complex questions students can check more concepts at once and move to their goal with much larger steps.
- #22: Then, progress percentage of students in each group were compared to the average progress percentage in whole class which also includes the student who did not use any system.
- #26: The challenge here is that manual indexing is a very hard and time consuming process. A single question may have more than 90 concepts which is very hard to be indexed manually!
- #29: AST tree consists of several nodes each containing some information known as structural properties.