Aspiring Minds | Automata
Download as PPTX, PDF1 like649 views
This document describes an approach to automatically grade programming assignments using machine learning. It discusses existing solutions like manual evaluation and test-case based evaluation that do not scale well. The proposed approach uses control and data dependency features extracted from programs to train a machine learning model on a small set of human-graded programs. Experimental results on sorting problems show the machine learning scores match human scores better than test-case based evaluation. Insights from features also help understand human grading criteria.
Aspiring Minds | Automata
- 1. Aspiring Minds www.aspiringminds.com Grading Programs using Machine Learning Varun Aggarwal Presented at KDD, 2014
- 2. Programming Assessments: Existing solutions • Manual evaluation: Can’t scale; not standardized • Test-case based evaluation: • High false-positives – hard code, inadvertent errors • High false-negatives – correct code but not efficient • Similarity metric between control flow graphs, syntax trees: • Need to handle multiple correct implementations – theoretically doesn’t fit in • No mapping of metric to an objective feedback
- 3. Automatic grading of programs– Why? • Widely performed - will help professors and TAs save a lot of time. • Companies can recruit efficiently • MOOCs - need automated open response assessments to really make it effective. True scaling of such system currently not achieved.
- 4. A model to predict the logical correctness of a program, given the control and data dependencies it possesses. Our Approach Automata – Automatic program evaluation engine Machine Learning based scoring engine Evaluation of programming best practices Asymptotic complexity evaluation Lint-styled rule-based system to detect programs not following programming best practices. Measures the run-time of the code for various input sizes and empirically derives the complexity.
- 5. Why programming modules give a better test- shortlist rate ? • Programming has more predictive power in identifying good performers than Logical ability. • Due to lower predictive power of Logical, a higher cut-off has to be applied to it as compared to Programming to get the same organizational efficiency. • Higher the Programming capability of the person, requirement on Logical score is lesser. • Given the person is lower than a given score on Programming, even having a higher logical ability does not help.
- 6. Evaluation Rubric ML based scoring Understanding the human process Problem and Language independent Features Machine learning model Ungraded programs Graded programs Predicted grades 1 2 3 4 5 6 7
- 7. Evaluation Rubric Our Approach Understanding the human process Problem and Language independent Features Machine learning model Ungraded programs Graded programs Predicted grades 1
- 8. Evaluation Rubric Our Approach Understanding the human process Problem and Language independent Features Machine learning model Ungraded programs Graded programs Predicted grades 2
- 9. Evaluation Rubric Our Approach Understanding the human process Problem and Language independent Features Machine learning model Ungraded programs Graded programs Predicted grades 3
- 10. void print_1(int N){ for(i =1 ; i<=N; i++){ print newline; count = i; for(j=0; j<i; j++) print count; count++; } } 1 2 3 3 4 5 4 5 6 7 OBJECTIVE To print the pattern of integers An implementation 1. Are there loops? Are there print statements? 3. Is the conditional in the inner loop dependent on - a variable modified in the outer loop? - a variable used in the conditional of the outer loop? What does a grader look for? 2. Is there a nested-loop structure?
- 11. Grammar for expressing features • Simple features • Keywords and Tokens (Counts): • Tokens like for, if, return, break; function calls like printf, strrev, strcat; declarations like int, char • Operators like various arithmetic, logical, relational operators used • Character constants like ‘0’, ‘ ’, ‘65’, ‘96’ • Capturing logical constructs (Interactions) • Control flow structure • Data-dependencies • Data-dependencies in context of control-flow
- 12. CONTROL FEATURES – COUNTS Counts of control-related keywords/tokens Ex. count(for) = 2 count(for-in-for) = 1 count(while) = 0 Control-context of these keywords - The Print command as loop(loop(print))) for(i =1 ; i<=N; i++){ print newline; count = i; CONTROL FLOW GRAPH i = 1 i <= N i++ j < i count = i j = 0 print(count) count++ j++ END Loop 1 Loop 2 Parent scope for(j=0; j<i; j++) print count; count++; void print(int N){ } } TARGET PROGRAM
- 13. DATA OPERATION FEATURES IN CONTROL-CONTEXT Counts of data-related tokens in context of the control structure Ex. count(block1 :loop(loop(++))) = 2 count(block1 :loop(loop_cond(<))) = 1 Capture control-context of data-dependencies in groups of expressions i++ j < i : var (i) related to var (j) : appearing in a loop(loop_cond) previously incremented : appearing in a loop The relation and the increment happen in the same block Loop 1 Loop 2 Loop 1 Loop 1 Loop 2 Loop 1 Loop 2 Loop 1 i = 1 i <= N print(count) count = ii++ j < i count = 0 count++ j= 0 j++ Parent scope Parent scope Loop 1 Loop 1 Loop 2 CONTROL FLOW INFORMATION ANNOTATED IN A-D-D GRAPH Loop 1
- 14. • Deployed Automata in a major product-based company’s recruitment • Analyzed the performance improvement in using Automata over test-case pass based selection criterion • 22.6% candidates who were not being shortlisted through test-case pass were now shortlisted using Automata. Case study
- 16. Doing it the one-class way! PROBLEM All features Basic features Mean Min25 Mean Min25 1 0.57 0.61 0.52 0.56 2 0.80 0.83 0.72 0.75 3 0.75 0.81 0.59 0.73 4 0.81 0.81 0.75 0.75 5 0.68 0.69 0.55 0.61 Betters test-case in all, but one
- 17. How good is the final ML-based score? Validation Correlation >= 0.79 Matches Inter-rater Correlation between two human raters PROBLEM # of features Cross-val correl Train correl Validation correl Test Case Score 1 80 0.61 0.85 0.79 0.54 2 68 0.77 0.93 0.91 0.80 3 193 0.91 0.98 0.90 0.64 4 66 0.90 0.94 0.90 0.80 5 87 0.81 0.92 0.84 0.84
- 18. Can we get insight? • The most contributing feature for Find Digit problem - int findDigit(int N, int digit){ … … LOOP (N != <constant value>){ … N = N / <constant value> … } } Features for FindDigit problem analyzed. Given a multi-digit number and a digit, one has to find the number of times the digit appears in the number
- 19. Yes, we can! • The most contributing feature for Find Digit problem - int findDigit(int N, int digit){ … LOOP (N != <constant value>){ … N = N / <constant value> … } … } int findDigit(int N, int digit){ ... while(N != 0){ d = N%10; if(d == digit) ... N = N / 10; } }
- 20. Evaluation Rubric Score Interpretation 5 Completely correct and efficient An efficient implementation of the problem using right control structures and data- dependencies. 4 Correct with some silly errors Correct control structures and closely matching data-dependencies. Some silly mistakes fail the code to pass test-cases. 3 Inconsistent logical structures Right control structures start exist with few correct data dependencies 2 Emerging basic structures Appropriate keywords and tokens present, showing some understanding of the problem 1 Gibberish code Seemingly unrelated to problem at hand.
- 21. Automata – Sample report Candidate’s source code Feedback on programming best practices Asymptotic complexity of the candidate’s solution Test case pass/fail information Problem summary
- 22. Do our fancy features help? Control and Data dependency features add around 0.15 correlation points above token information. PROBLEM Type of feature # of features Cross-val correl Train correl Validation correl 1 All, w/o test case 35 0.57 0.72 0.56 Basic 60 0.62 0.87 0.41 2 All, w/o test case 80 0.81 0.99 0.80 Basic 26 0.59 0.72 0.67 3 All, w/o test case 190 0.87 0.97 0.90 Basic 26 0.74 0.89 0.74 4 All, w/o test case 134 0.85 0.91 0.82 Basic 35 0.83 0.88 0.69 5 All, w/o test case 166 0.66 0.81 0.64 Basic 40 0.61 0.78 0.61
- 23. Conclusion • We propose the first machine learning based approach to automatically grade programs • An innovative feature grammar is proposed which matches human intuition of grading programs. • Models built for sample problems show promising results. • We propose and demonstrate machine learning techniques to lower the need of human-graded data to build models.