What is the important data that is not being recorded in comparative international analyses on student success? Examining the challenges of addressing learner diversity proactively and of quantifying the impact of inclusive practices.
Download as PPTX, PDF0 likes229 views
The document discusses limitations of international standardized data collection in assessing learner diversity and inclusive practices. It acknowledges progress made through comparative analyses but notes significant resistance from teachers due to misuse of data to push political agendas. Key limitations of current data are the failure to record student challenges to learning, school environments' ability to be inclusive, and impacts of school culture. There is a need for large-scale qualitative tools to evaluate ecological factors influencing inclusion and better reflect realities on the ground. Key indicators like ableism, racism, and socioeconomic challenges remain largely ignored.
![References
Ainscow, M. (2015). Struggles for Equity in Education: The Selected Works of Mel Ainscow. Routledge. DOI:10.4324/978131156882213.
Booth, T., & Ainscow, M. (2002). Index for Inclusion: Developing Learning and Participation in Schools. Centre for Studies on Inclusive Education.
Breakspear, S. (2014) How does PISA shape education policy making? Why how we measure learning determines what counts in education. Centre for Strategic Education. https://allchildrenlearning.org/wp-content/uploads/2019/11/Breakspear-PISA-Paper.pdf
Brooks, C. (2021) The quality conundrum in initial teacher education. :Teachers and Teaching, 27(1-4), 131-146, DOI: 10.1080/13540602.2021.1933414
Cope, B., & Kalantzis, M. (2016). Big Data Comes to School: Implications for Learning, Assessment, and Research. AERA Open. https://doi.org/10.1177/2332858416641907
CSIE (2019) Breaking down the barriers: the Index for Inclusion. Centre for Studies on Inclusive Education supporting inclusion, challenging exclusion. http://www.csie.org.uk/resources/breaking-barriers.shtml
Cunningham, J. (2019). Missing the mark: Standardized testing as epistemological erasure in U.S. schooling. Power and Education, 11(1), 111–120. https://doi.org/10.1177/1757743818812093
Dickinson, K. (2019, February 15) Standardized tests: Finland’s education system vs. the U.S. Big Think. https://bigthink.com/politics-current-affairs/standardized-testing?rebelltitem=2#rebelltitem2
Interagency Working Group on Inclusion in STEM (2021) Best practices for diversity and inclusion in STEM Education and research: A guide by and for Federal Agencies. National Science and Technology Council. https://www.whitehouse.gov/wp-content/uploads/2021/09/091621-Best-Practices-for-Diversity-
Inclusion-in-STEM.pdf
Krammer, M., Gasteiger-Klicpera, B., Holzinger, A., & Wohlhart, D. (2021) Inclusion and standards achievement: the presence of pupils identified as having special needs as a moderating effect on the national mathematics standards achievements of their classmates. International Journal of Inclusive Education,
25(7), 795-811. DOI: 10.1080/13603116.2019.1573938
Krieg, J. (2019) International Student Assessments Performance and Spending. Fraser Institute. https://www.fraserinstitute.org/sites/default/files/international-student-assessment.pdf
Lundahl, C., & Serder, M. (2020) Is PISA more important to school reforms than educational research? The selective use of authoritative references in media and in parliamentary debates. Nordic Journal of Studies in Educational Policy, 6(3), 193-206,. DOI: 10.1080/20020317.2020.1831306
Organisation for Economic Co-operation and Development [OECD] (2000–2015). Program for International Student Assessment (PISA), 2000, 2003, 2006, 2009, 2012, and 2015 Mathematics, Reading, and Science Assessments. OECD
Organisation for Economic Co-operation and Development [OECD](2019) Strength through Diversity: Education for Inclusive Societies. Design and Implementation Plan. EDU/EDPC(2019)11/REV2. Organisation for Economic Co-operation and Development. https://www.oecd.org/education/strength-through-
diversity/Design-and-Implementation-Plan.pdf
Patrinos, H.A.., & Angrist, N. (2018) Global Dataset on Education Quality A Review and Update (2000–2017). World Bank Group. https://documents1.worldbank.org/curated/en/390321538076747773/pdf/WPS8592.pdf
Penninckx, M. (2020). Quality education without central examinations: The case of Flanders. Management in Education, 34(1), 40–43. https://doi.org/10.1177/0892020619881045
Pons, X. (2017) Fifteen years of research on PISA effects on education governance: A critical review. Eur J Educ., 52: 131– 144. https://doi.org/10.1111/ejed.12213
Poulsen, J., & Hewson, K. (2014) Standardized Testing: Fair or Not? Teaching Centre, University of Lethbridge. https://www.ulethbridge.ca/teachingcentre/standardized-testing-fair-or-not
Rozgonjuk, D., Täht, K. & Vassil, K. (2019) Internet use at and outside of school in relation to low- and high-stakes mathematics test scores across 3 years. International Journal of STEM Education, 8, 27. https://doi.org/10.1186/s40594-021-00287-y
Shelton, S., & Brooks, T. (2019) “We Need to Get These Scores Up”: A Narrative Examination of the Challenges of Teaching Literature in the Age of Standardized Testing. Journal of Language and Literacy Education. 19(2). https://files.eric.ed.gov/fulltext/EJ1235207.pdf
Simon, S. (2012, June 12) Parents Protest Surge in Standardized Testing. Reuters, U.S. edition . http://www.reuters.com/article/us-usa-education-testing-idUSBRE85B0EO20120612.
Tveit, S., & Lundahl, C. (2018). New modes of policy legitimation in education: (Mis)using comparative data to effectuate assessment reform. European Educational Research Journal, 17(5), 631–655. https://doi.org/10.1177/1474904117728846
Wells, A.S., Fox, L., & Cordova-Cobo, D. (2016) How Racially Diverse Schools and Classrooms Can Benefit All Students. The Century Foundation. https://tcf.org/content/report/how-racially-diverse-schools-and-classrooms-can-benefit-all-students/?agreed=1
Wu, X., Wu, R., Chang, H. H., Kong, Q., & Zhang, Y. (2020). International Comparative Study on PISA Mathematics Achievement Test Based on Cognitive Diagnostic Models. Frontiers in psychology, 11, 2230. https://doi.org/10.3389/fpsyg.2020.02230](https://image.slidesharecdn.com/fredericfovetfridaynovember19th-211119183336/85/What-is-the-important-data-that-is-not-being-recorded-in-comparative-international-analyses-on-student-success-Examining-the-challenges-of-addressing-learner-diversity-proactively-and-of-quantifying-the-impact-of-inclusive-practices-14-320.jpg)
What is the important data that is not being recorded in comparative international analyses on student success? Examining the challenges of addressing learner diversity proactively and of quantifying the impact of inclusive practices.
- 1. What is the importantdata that is not beingrecordedin comparativeinternationalanalyses on studentsuccess? Examiningthechallengesof addressinglearnerdiversity proactivelyand of quantifyingthe impactof inclusivepractices. CIMQUSEF18: 18th Edition of the International Conference on the Quality of Education The main determinants of improving learning outcomes: Analysis of PISA, TIMSS & PIRLS data Frederic Fovet, School of Education and Technology, RRU
- 2. Objectives of the Session • Acknowledge the global push back against national and international data analysis in education in the current neo-liberal landscape • Explore some of the motivations behind that push back • Examine the extent to which inclusive practices currently fail to be recorded by comparative international data collection • Explore avenues for reform and for the development of pertinent large scale qualitative tools for the evaluation of ecological factors that impact inclusion
- 3. Phenomenal progress • There have been giant steps made in the last decade with regards to the ways data on student performance is collected, analyzed and used for school improvement (Breakspear, 2014; Rozgonjuk et al., 2019; Wu et al., 2020). • Much of the impact of the analysis of this data lies in the fact that it has allowed for large international comparative studies that yield important conclusions on the effectiveness of teaching practices, curriculum, and modes of assessment (Dickinson, 2019; OECD 2000-2015). • The PISA framework and annual PISA results have in particular allowed for revealing reflections, at international level, in relation to the objectives, ethos and performance of national educational structures (Krieg, 2019; Patrinos & Angrist, 2018).
- 4. But also unprecedented resistance • There is widespread resistance, misunderstanding and fear among teachers and administrators against all forms of standardized testing. • Continued disquiet - Pons (2017) talks about the “ongoing PISA shock” • Lundahl & Serder (2020) discuss the dangerous precedence of PISA stats over educational research. Tveit & Lundahl (2018) also discuss the misuse of international comparative data on education to push political reform • Explicit push back in the field: Tension with regards to purpose of assessment (Penninckx, 2020); tension with regards to goals and values of teacher training (Brooks, 2021); clash with learning outcomes and teaching philosophies (Shelton & Brooks, 2019) • Also considerable evidence of en masse push-back from parents (Simon, 2012)
- 5. Causes of this resistance • Much of the push back emerges from fears of the Neoliberal agenda that underlies much of the discourse on quality indicators in education. • Though the essential objective is to obtain a snapshot of practices and standards for comparative purposes, the objective is now widely seen as simply ‘scoring high’ in standardized testing. • Has led to a process where this creates tension on school leaders to ‘dribble down’ pressure on teachers rather than on examining the root cause of low performance. • In many contexts, this pressure is then passed on to the student to ‘over- perform’ in standardized screenings. • It can be said that at this stage, on the ground, international efforts to gather standardized data are misunderstood and misused.
- 7. A disquiet that is not going away… And should it? • International comparative studies are carried out on the data collected in these large surveys/ standardized exercises. • However, these studies have yet to examine learner diversity or an educational system’s ability to develop, grow and sustain inclusive practices in schools (Krammer et al., 2021). • As a result, a significant gap exists in the quantitative data that is emerging from international comparative studies (Ainscow, 2015; Booth & Ainscow, 2002; Poulsen & Hewson, 2014). • Much of the disquiet felt by teachers relates the increasing chasm between awareness of/ policy around learner diversity and inclusion vs. global quantitative outcomes.
- 8. Limitationsof international, comparativestandardizeddata on the issuesof learner diversityand inclusivepractices • Three angles that are under-reported or not reported at all: (1) Students’ challenges with access to learning (acknowledging student diversity) In the simplest of ways, learner diversity is entirely erased from the tools for data collection and analysis. In most contexts, students with disabilities for example are not given access to accommodations. Nor are students allowed to use assessment strategies they may have used for years to achieve positive outcomes. (2) Environments’ ability to design inclusively (social model perspective) Current international comparative tools dismiss the importance of the design of the learning environment and experience/ the role of the educator as designer of an inclusive experience. Tools quantify learners as though they remain unaffected by the practices of educators beyond competencies themselves. (3) School culture and its impact on learning outcomes (systemic dimension) There is no provision currently to assess school culture and its impact on learning outcomes, even of the literature constantly reiterates its key role in student success
- 9. The COVID Crisis and its impact on this tension • The school closures and the pivot to online teaching and assessment has led to a cathartic understanding of the limitations of current national and international data practices in education. • Inequities have increased, the gap has widened. Some learners and communities have been disastrously left out. • Current quality insurance tools in the K-12 sector have been unable to reflect these challenges and these realities.
- 10. Quantitative tools that do exist but are currently under-utilized in terms of data mining • Strength through Diversity: Education for Inclusive Societies. Design and Implementation Plan (OECD, 2019) is promising but mostly focused on refugees and migrants – somewhat limited concept of learner diversity • Some large scale projects produced by private foundations, such as the Century Foundation’s How Racially Diverse Schools and Classrooms Can Benefit All Students (Wells, Fox & Cordova- Cobo, 2016). There are, however, inherent issues with work from the private sector in this area: scope, lens, international and comparative flavour. • Some key national reports on inclusive practices in specific sectors, such as the Interagency Working Group on Inclusion in STEM’s (2021) Best practices for diversity and inclusion in STEM Education and research: A guide by and for Federal Agencies. However, many of these reports are proactive recommendations rather than the analysis of existing outcomes/ data. • All in all, credible efforts are observed but we are still very far from functional, sustainable, quantitative, international and comparative tools that have the ability to offer a pertinent and valid snapshot of inclusive practices in the K-12 sector and of their impact on outcomes.
- 11. Need for large scale qualitative tools that go beyond existing tools • The Index for Inclusion (Booth & Ainscow, 2002) is a set of materials to guide schools through a process of inclusive school development. It is about building supportive communities and fostering high achievement for all staff and students. • “The Index is a set of materials to support schools in a process of inclusive school development, drawing on the views of staff, governors, school students, parents/carers and other community members. It is concerned with improving educational attainments through inclusive practice and thus provides an attempt to redress a balance in those schools which have concentrated on raising student attainment at the expense of the development of a supportive school community for staff and students.” (CSIE, 2019) • Three objectives: (i) schools assess their practices, (ii) planning for improvement, (iii) ensures participation of all stakeholders. • However, the lack of impact of the Index for Inclusion in the last decade is evidence that it is perhaps not the most malleable of tools. Onerous, complex, and does not allow for easy comparisons across jurisdictions. It is more focused on the transformation of individual sites.
- 12. Need for large scale qualitative tools that go beyond existing tools (contd.) • Some challenges lie ahead in relation to the development of sustainable qualitative tools that might allow for comparative analysis of the various ways national education systems tackle the task of differentiating education. • Epistemological hurdle: quantitative stats cannot be sufficiently deep to examine and analyze the impact of the various ecological factors that impact the inclusion of diverse learners. It is urgent that large scale qualitative tools become available. • Persistent distrust with regards to qualitative tools in the field of education; represents an epistemological malaise (Cunningham, 2019) • The sector now has tools that are sufficiently refined and sophisticated to support large scale data collection and analysis, and the exploration of complex ecological indicators (Cope & Kalantzis, 2016). • There is not political will, however, currently to invest in the creation and application of these tools.
- 13. Key indicators that remain largely ignored in learning outcomes • Ableism: There is a need for subtlety and nuance here as much of the discourse on inclusion remains bio- medical, instead of embracing the Social Model of Disability (). • Racism: The Black Lives Matter movement has made most jurisdictions realize to what extent racism has been ignored in all aspects of society and public governance. Education is not an exception • Cultural ethno-centrism: There is increasing awareness that cultural diversity must be addressed effectively in school, in the midst of a disrupted geo-political landscape where mass migration represents specific challenges. • Sexism: The #MeToo movement has increased public awareness of sexual and gender power inequities. These exists in schools and the K-12 sector currently has no effective indicators to track their impact. • Socio-economic challenges: While economic or sociological reports will often highlight the crucial impact of poverty on education, standardized large scale data collection in the K-12 sector currently fails to acknowledge how these factors shape learning outcomes. • Decolonizing the curriculum: This is a key process in the K-12 sector (both with regards to ongoing internal colonization in the Global North – TRC in Canada for example - and to post-colonial processes in the Global South). There is a pressing need for indicators of success with regards to efforts to decolonize the K-12 curriculum. • Importance of examining EDI, in terms of tangible ecological indicators, in a decade where the rest of society finally grasps its impact on all sociological outcomes (employment, justice, law enforcement, media, etc.). This momentum cannot be successfully driven and assessed if there are no indicators available in the K-12 sector.
- 14. References Ainscow, M. (2015). Struggles for Equity in Education: The Selected Works of Mel Ainscow. Routledge. DOI:10.4324/978131156882213. Booth, T., & Ainscow, M. (2002). Index for Inclusion: Developing Learning and Participation in Schools. Centre for Studies on Inclusive Education. Breakspear, S. (2014) How does PISA shape education policy making? Why how we measure learning determines what counts in education. Centre for Strategic Education. https://allchildrenlearning.org/wp-content/uploads/2019/11/Breakspear-PISA-Paper.pdf Brooks, C. (2021) The quality conundrum in initial teacher education. :Teachers and Teaching, 27(1-4), 131-146, DOI: 10.1080/13540602.2021.1933414 Cope, B., & Kalantzis, M. (2016). Big Data Comes to School: Implications for Learning, Assessment, and Research. AERA Open. https://doi.org/10.1177/2332858416641907 CSIE (2019) Breaking down the barriers: the Index for Inclusion. Centre for Studies on Inclusive Education supporting inclusion, challenging exclusion. http://www.csie.org.uk/resources/breaking-barriers.shtml Cunningham, J. (2019). Missing the mark: Standardized testing as epistemological erasure in U.S. schooling. Power and Education, 11(1), 111–120. https://doi.org/10.1177/1757743818812093 Dickinson, K. (2019, February 15) Standardized tests: Finland’s education system vs. the U.S. Big Think. https://bigthink.com/politics-current-affairs/standardized-testing?rebelltitem=2#rebelltitem2 Interagency Working Group on Inclusion in STEM (2021) Best practices for diversity and inclusion in STEM Education and research: A guide by and for Federal Agencies. National Science and Technology Council. https://www.whitehouse.gov/wp-content/uploads/2021/09/091621-Best-Practices-for-Diversity- Inclusion-in-STEM.pdf Krammer, M., Gasteiger-Klicpera, B., Holzinger, A., & Wohlhart, D. (2021) Inclusion and standards achievement: the presence of pupils identified as having special needs as a moderating effect on the national mathematics standards achievements of their classmates. International Journal of Inclusive Education, 25(7), 795-811. DOI: 10.1080/13603116.2019.1573938 Krieg, J. (2019) International Student Assessments Performance and Spending. Fraser Institute. https://www.fraserinstitute.org/sites/default/files/international-student-assessment.pdf Lundahl, C., & Serder, M. (2020) Is PISA more important to school reforms than educational research? The selective use of authoritative references in media and in parliamentary debates. Nordic Journal of Studies in Educational Policy, 6(3), 193-206,. DOI: 10.1080/20020317.2020.1831306 Organisation for Economic Co-operation and Development [OECD] (2000–2015). Program for International Student Assessment (PISA), 2000, 2003, 2006, 2009, 2012, and 2015 Mathematics, Reading, and Science Assessments. OECD Organisation for Economic Co-operation and Development [OECD](2019) Strength through Diversity: Education for Inclusive Societies. Design and Implementation Plan. EDU/EDPC(2019)11/REV2. Organisation for Economic Co-operation and Development. https://www.oecd.org/education/strength-through- diversity/Design-and-Implementation-Plan.pdf Patrinos, H.A.., & Angrist, N. (2018) Global Dataset on Education Quality A Review and Update (2000–2017). World Bank Group. https://documents1.worldbank.org/curated/en/390321538076747773/pdf/WPS8592.pdf Penninckx, M. (2020). Quality education without central examinations: The case of Flanders. Management in Education, 34(1), 40–43. https://doi.org/10.1177/0892020619881045 Pons, X. (2017) Fifteen years of research on PISA effects on education governance: A critical review. Eur J Educ., 52: 131– 144. https://doi.org/10.1111/ejed.12213 Poulsen, J., & Hewson, K. (2014) Standardized Testing: Fair or Not? Teaching Centre, University of Lethbridge. https://www.ulethbridge.ca/teachingcentre/standardized-testing-fair-or-not Rozgonjuk, D., Täht, K. & Vassil, K. (2019) Internet use at and outside of school in relation to low- and high-stakes mathematics test scores across 3 years. International Journal of STEM Education, 8, 27. https://doi.org/10.1186/s40594-021-00287-y Shelton, S., & Brooks, T. (2019) “We Need to Get These Scores Up”: A Narrative Examination of the Challenges of Teaching Literature in the Age of Standardized Testing. Journal of Language and Literacy Education. 19(2). https://files.eric.ed.gov/fulltext/EJ1235207.pdf Simon, S. (2012, June 12) Parents Protest Surge in Standardized Testing. Reuters, U.S. edition . http://www.reuters.com/article/us-usa-education-testing-idUSBRE85B0EO20120612. Tveit, S., & Lundahl, C. (2018). New modes of policy legitimation in education: (Mis)using comparative data to effectuate assessment reform. European Educational Research Journal, 17(5), 631–655. https://doi.org/10.1177/1474904117728846 Wells, A.S., Fox, L., & Cordova-Cobo, D. (2016) How Racially Diverse Schools and Classrooms Can Benefit All Students. The Century Foundation. https://tcf.org/content/report/how-racially-diverse-schools-and-classrooms-can-benefit-all-students/?agreed=1 Wu, X., Wu, R., Chang, H. H., Kong, Q., & Zhang, Y. (2020). International Comparative Study on PISA Mathematics Achievement Test Based on Cognitive Diagnostic Models. Frontiers in psychology, 11, 2230. https://doi.org/10.3389/fpsyg.2020.02230
- 15. Contact details • Frederic Fovet (PhD.) • Associate Professor, School of Education and Technology, Royal Roads University • Frederic.fovet@royalroads.ca • @Ffovet • www.implementudl.com