2020_04_30 «Computational Thinking Test for Beginners: Design and Content Validation»
0 likes330 views
The document describes the design, validation, and testing of the Beginners Computational Thinking Test (BCTt), which aims to assess computational thinking in primary school students ages 5-10. Experts validated version 1 of the BCTt and provided feedback, which informed version 2. Version 2 was administered to 299 primary students and showed high reliability. Results indicated that transitions between maze squares improved younger students' scores and the BCTt is best suited to assess computational thinking in students ages 5-8. Further research is still needed to validate the BCTt for other populations and countries.
2020_04_30 «Computational Thinking Test for Beginners: Design and Content Validation»
- 1. María Zapata Cáceres Estefanía Martín Barroso Marcos Román González EDUCON2020 – IEEE Global Engineering Education Conference "Engineering Education for the Future in a Multicultural and Smart World" Computational Thinking Test for Beginners Design and Content Validation 1maria.zapata@urjc.es - RRSS- @artbabygame - web - www.artbabygame.com/blue-ant-code
- 2. 2 Computational Thinking (CT) Fundamental skill required to adapt to the future Human problem-solving process that requires abstract thinking Many definitions- breakdowns / Broad debate Thinking skills that precede programming, and are applied in understanding a problem and formulating a solution like a computer scientist Should be taught at schools Programming exposes students to CT Consensus 3D Framework (Brenan & Resnick*) * K. Brennan, M. Resnick and MIT Media Lab, "New frameworks for studying and assessing the development of computational thinking," American Educational Research Association Meeting, Vancouver, BC, Canada, 2012. CT Concepts CT Practices CT Perspectives
- 3. 3 New Computational Thinking Test (BCTt) How to teach it? What to teach? How to assess it? When to teach it? Focused on middle school grades and specific programming environments * M. Román-González, J. Pérez-González and C. Jiménez-Fernández, "Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test," Computers in Human Behavior, vol. 72, pp. 678-691, 2017. Younger students (5 to 10 years old) Based on CTt Computational Thinking Test (CTt)* (Aimed for 10-16 year olds) Substancial improvements New Beginners Computational Thinking Test (BCTt) Assessment NOW Form / content adaptation Stand-alone assessment instrument
- 4. 4 BCTt v.1 Designv.1 25 items long 40 minutes Least possible text Self explanatory symbols Young studentsDesign Emotional connection 3 response alternatives Canvas and maze type Visual transitions in mazes Computational concepts
- 5. 5 BCTt Expert Judgement procedure & results 45 experts Different profiles Expert Judgement Content validation Item difficulty level v.1 Item relevance to measure CT Test length adequacy Graphic interface adequacy Improvements adequacy: e.g. transitions Other / suggestions / comments 66 questions form adequate too long too short y = 0,1063x + 1,4239 R² = 0,8883 0,00 0,50 1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 y = 0,0786x + 3,6857 R² = 0,5818 0,00 0,50 1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00 0 1 2 3 4 5 6 7 BCTt item difficulty perceived by experts (ordinate axis: Likert scale from 1 to 5), per BCTt item (abscissa axis). BCTt computational concept relevance to measure CT, perceived by experts (ordinate axis: Likert scale from 1 to 5) by computational concept (Abscissa axis: 1. Sequences, 2. Simple loop, 3. Nested loop, 4. If-then, 5. If-then-else, 6. While) Test length Adequacy to evaluate CT very good good intermediate bad very bad
- 6. 6 BCTt v.2 Design Based on experts’ Judgement results Modified both in form and content Final version (v.2) v.2 25 questions long 4 alternative responses If-else reformulation If-then-else reformulation Color blindness adaptation Oral explanation before the test BCTt v.2 item example (item number 18)
- 7. 7 BCTt Administration: participants and procedure 299 Primary School Students v.2 5 to 12 years old Educational stage Grade Identifier BCTt BCTt variation 1st 1 A A1: n=52 2 B B1: n=18 B2: n=18 2nd 4 C C1: n=54 4 D D1: n=28 D2: n=28 3rd 5 E E1: n=51 6 F F1: n=25 F2: n=25 School Educational stage Grades Students’ ages Colegio Público Carlos Ruiz 1st 1st and 2nd 5 - 8 Colegio Los Escolapios 2nd 3rd and 4th 7 -10 CEIP León Felipe 3rd 5th and 6th 9 -12 Action protocol Test printed in paper form BCTt BCTt variation transitions No-transitions Time 1 B2, D2, F2 A1, B1, C1, D1, E1, F1 D1 Time 2: 5 weeks later Procedure BCTtBCTt variation BCTt
- 8. 8 BCTt Administration: resultsv.2 Transitions Student’s t-test Significant difference in test scores (p=0.005 < 0.01) in lower grades Sample Entire sample A1 B1 C1 E1 F1 Grade 1-6 1 2 4 5 6 N 200 52 18 54 51 25 Mean 19.92 16.52 16.78 21.57 21.84 21.72 Median 20.00 16.00 18.00 23.00 23.00 22.00 Std. Deviation 3.79 3.31 2.49 3.044 2.61 2.62 Variance 14.36 10.96 6.183 9.268 6.815 6.88 Minimum 8.00 8.00 11.00 14.00 13.00 15.00 Maximum 25.00 24.00 20.00 25.00 25.00 25.00 Percentiles 25 17.00 14.00 15.75 19.00 20.00 19.50 50 20.00 16.00 18.00 23.00 23.00 22.00 75 23.00 19.00 18.00 24.00 24.00 24.00 Descriptive statistics
- 9. 9 BCTt Administration: resultsv.2 0,00 0,50 1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00 Sequences Simple loop Nested loop If-then If-then-else While 1st 2nd 4th 5th 6th Item analysis Abscissa axis: computational concept by grade. Ordinate axes: BCTt item score, normalized from 0 to 5: 5 maximum score). y = -0,0155x + 1,0083 R² = 0,6344 0 0,2 0,4 0,6 0,8 1 1,2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Item difficulty index (ordinate axis) for each BCTt item (abscissas axis). y = -0,0173x + 0,9249 R² = 0,3904 0 0,2 0,4 0,6 0,8 1 1,2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Item difficulty index (ordinate axis) for each BCTt item (abscissas axis). First Educational Stage
- 10. 10 BCTt Administration: resultsv.2 Reliability Sample Reliability Statistics Item Statistics N N of Items Cronbach's Alpha Cr. 's Alpha Based on Stand. Items Mean Min. Max. Variance 200 25 0.824 0.829 0.807 0.576 0.976 0.021 Subsamples Reliability Statistics Item Statistics Ed. stage Grade Id. n Cronbach's Alpha Cr. 's Alpha Based on Stand. Items Mean Variance 1st 1 A1 52 0.833 0.838 0.742 0.041 1st 2 B1 18 0.793 0.801 0.630 0.042 2nd 4 C1 54 0.771 0.735 0.837 0.022 3rd 5 E1 51 0.660 0.683 0.863 0.012 3rd 6 F1 25 0.657 0.648 0.844 0.015 very strong significant correlation (rs=0.93; p<0.01). Task and re-task method (D1 subsample) Non-parametric Spearman’s test
- 11. ✓BCTt is adequate for the assessment of CT in Primary School ✓Transitions between maze squares are a relevant improvement for young students ✓BCTt seems to be aimed at 1st to 4th grades (5 to 10 years old) ✓Reliability is high and higher in younger students ✓It focuses on computational concepts, partially in computational practices and ignores computational perspectives → recommended to use in parallel with other tools ✓Further research → BCTt lower limit, other countries / populations 11 Conclusions