Creating Session Data from eTextbook Event Streams
- 1. Creating Session Data from eTextbook Event Streams SAMNYEONG HEO, MOHAMMED FARGHALLY, MOSTAFA MOHAMMED, CLIFF SHAFFER DEPARTMENT OF COMPUTER SCIENCE, VIRGINIA TECH JULY 3, 2023 FIFTH WORKSHOP ON INTELLIGENT TEXTBOOKS ( ITEXTBOOKS)
- 2. PROBLEM Intelligent Textbooks typically provide text and interactive elements Easy to collect detailed event-level data Raw (event-level) data can be hard to interpret A large class can generate a million+ events
- 3. OBJECTIVES Group raw event data into study sessions Recognize behaviors Find correlations behavior and student performance
- 4. BACKGROUND - OpenDSA Open-source electronic textbook system Provides contents on CS-related topics • DSA, Formal Languages, Finite Automata Provides algorithm visualizations, interactive exercises, proficiency exercises
- 5. Module • A fundamental unit of OpenDSA • Resembles a specific topic or section in a traditional textbook • Groups of modules form “chapters” Structure of OpenDSA
- 7. Proficiency Exercises • A type of exercise used in FLA course • Student creates a state machine represented by a graph • Solutions are tested using test cases OpenDSA Exercises
- 8. PIFrameset: Programmed Instruction Frameset • Customized slideshows used in FLA course • Must answer questions on slides in order to move to the next • A credit will be awarded after completing the slide set OpenDSA PIFramesets
- 9. User Interaction Data Core information to analyze students’ behavior Records user activities in OpenDSA Window events Button clicks Credit events
- 11. Session • A session is a key organizing unit of behavior • Study sessions are a continuous stream of interactions without a “significant” gap • The time threshold for the gap is 10 minutes SESSION DATA
- 12. Abstracting User Interaction Data
- 13. Abstracting User Interaction Data
- 15. EXPERIMENT DATA: Courses Data collected from 4 classes FLA Fall 2020 / Spring 2021 DSA Fall 2020 / Spring 2021 (Data aggregated)
- 16. EXPERIMENT - Behaviors Session-level data is rather a lot of information for instructors to absorb “Behaviors” are a higher-level of abstraction
- 18. BEHAVIOR: Reading Prose A transition from “module opening” to “exercise state” (load-to-exercise) or leaving the page is made within the time threshold (15 to 120 seconds) Expect to see three clusters 1. Not reading – Less than 15 seconds 2. Reading – Between 15 and 120 seconds 3. Stepped Away – More than 120 seconds
- 19. SESSION STATES State • Series of events on the same content element • Such as Document State, Window State, PE State, FF State, Other State • Defining session states allows us to recognized multi-state behaviors.
- 20. BEHAVIOR: Credit Seeking A transition from “module opening” to “exercise state” (load-to-exercise) is made less than the time threshold (15 seconds). Students seeking credits from completing slideshows and solving exercises without reading the material.
- 21. BEHAVIOR: PI Credit Seeking A transition from FF state to the same FF state is triggered within the threshold (8 seconds) Student advances to the next slide within 8 seconds, indicating a possible credit seeking (guessing) Instead of raw counts of the behavior, we consider the ratio of the total number of PIFrame related events and the PI credit seeking events
- 22. SESSION STATES: Uses Instructor investigation: Present abstract session data to instructors or researchers to examine. Ex: Instructor can see from session data that student X is exhibiting high credit-seeking behavior. Useful to know when counseling student. Automated Analysis: Preprocessing step
- 23. BEHAVIOR: Automated Analysis 1. Define various behaviors 2. Recognize those behaviors from session data 3. Check correlation between behaviors and performance • Assign students into two groups: High vs Low performers • Investigate whether there are differences between the groups on candidate behaviors • If significant difference exists, then examine whether behavior affects performance
- 24. Reading Prose Behavior C S 5 0 4 0 H i g h P e r f o r m e r L o w p e r f o r m e r # of Students 35 21 Lowest occurrence 0 0 Highest occurrence 61 69 Average occurrence 18 15 Std. Deviation 15 16 P-value 0.54
- 25. PI Credit Seeking Behavior C S 4 1 1 4 F C S 4 1 1 4 S H i g h P e r f o r m e r L o w p e r f o r m e r H i g h P e r f o r m e r L o w p e r f o r m e r # of Students 33 25 24 23 Lowest Ratio 1.302 1.472 1.32 0 Highest Ratio 2.452 2.732 2.69 7.67 Average Ratio 1.790 1.883 1.67 1.82 Std. Deviation 0.265 0.310 0.32 1.32 P-value 0.24 0.60
- 26. • Only Activity Sessions behavior shows a significant relationship to the performance in terms of overall course grades • The HP group tends to generate more Activity Sessions behavior, and less PI Credit Seeking behavior ratio C S 4 1 1 4 F C S 4 1 1 4 S H P L P H P L P Activity Sessions Avg. occurrence 287 182 441 231 P-value 0.013 0.01 PI Credit Seeking Avg. Ratio 1.79 1.88 1.67 1.82 P-value 0.24 0.60 FLA Activity Sessions
- 27. CONCLUSIONS Main goal is to group the raw interaction data into sessions, then generate abstracted session-level data Suggested uses for abstracted data Defined possible behaviors that can be observed
- 28. FUTURE WORK Additional Learning Behaviors • Frequently jumping in and out of a module • Opens multiple modules simultaneously Time considerations for sessions • Tracking scroll events
- 29. Questions ?