Assurance of Learning - The ECU Story
Download as PPTX, PDF1 like457 views
The document outlines the assurance of learning (AOL) initiatives at Edith Cowan University, emphasizing their commitment to enhancing education quality and student outcomes through structured assessment and continuous improvement. It details the university's history, strategic intent, and the steps taken towards achieving AACSB accreditation, including curriculum mapping and stakeholder engagement. Additionally, it highlights the challenges faced in implementing AOL and the importance of fostering a culture of ongoing assessment and development within the institution.
Assurance of Learning - The ECU Story
- 1. Assurance of Learning The ECU Story PROFESSOR MARYAM OMARI EXECUTIVE DEAN SCHOOL OF BUSINESS AND LAW EDITH COWAN UNIVERSITY
- 3. Edith Cowan University • Large multi-campus university • 100+ year history, 27 years as a university • ~28,000 students • International students from 100+ countries • 2018 • QS, Top 150 under 50 • THE, Top 100 Asia-Pacific Universities • THE World University Rankings, in the 500- 600
- 4. QILT 2018 - Released May 16, 2018 • The top public university in Australia for student experience – 2nd year in a row • Also rated top public university in Australia for: • Teaching Quality • Skills Development • Student support • Business and Management Discipline – top public university in Australia • Overall Education Experience • Teaching Quality • Skills Development • Student Support
- 6. ECU Renewal and Change 2015 • April 1, appointment of the new Vice-Chancellor • Review of the academic units at ECU 2016 • Academic restructure: 8 new Schools • Appointment of 8 new Executive Deans • New leadership team in the School of Business and Law
- 7. Our Strategic Intent | 2017–2021 ECU’S PURPOSE To transform lives and enrich society through education and research. ECU’S VISION To be recognised for our world-ready graduates and leading edge research. SBL’S MISSION The School of Business and Law develops industry-ready graduates through high quality teaching and learning experiences and engaged, applied research.
- 8. Our Four Pillars • Enhancing Learning and Teaching • Advancing Research and Knowledge Translation • Growing Internationalisation • Ensuring Organisational Sustainability
- 9. The School of Business and Law • January 2016 - New name, Executive Dean and leadership team • Has grown by over 30% in academic staff numbers in just 2 years • Huge quantum leap in research and quality publications • Teaching outcomes going from strength to strength • February 2016 - AACSB accreditation journey began • May 2017 – Eligibility Application (EA) submitted • May 2018 – First mentor visit • September 2018 – iSER will be submitted
- 10. What is Assurance of Learning (AoL) about?
- 11. Meet the AOL team at ECU!
- 12. The essence of AoL | Herding Cats • Scoping • Review • Standardisation • Measurement • Improvement • Keep the cycle going to eventually close the loop! • Most importantly: Have good people driving it! Image: Photo still from EDS ad campaign ‘Cat Herders’, launched at the Superbowl in 2000. Agency: Fallon Worldwide
- 13. Who are the stakeholders in the AoL process? • Students • Industry/potential employers • Our Course Consultative Committees and Advisory Board • Professional associations • Alumni • Staff • The wider university • The higher education sector Image: Rodion Kutsaev accessed at Unsplash, Feb 21, 2018
- 14. Who is ultimately responsible?
- 15. The QA processes we already had in place • Annual unit reviews • Annual course reviews • Professional accreditations • Ad-hoc program reviews given new markets and industry developments • 3 yearly/Triennial reviews of our international offerings • 5 yearly comprehensive course reviews with benchmarking against other institutions and courses
- 16. What did we do to make it better? • Regular seminars and use of technologies to facilitate the process: 450+ hrs of TEL training • Teaching and learning grants to support and encourage: • 5 grants a year since 2016 totaling $15K AUD/annum, • 3-6 conference sponsorships between $15K - $18K AUD/annum, • $60K AUD for curriculum renewal in 2017 across 4 discipline areas • Built and strengthened a team to drive the initiative • Identified champions for the cause
- 17. What did we already do to assure learning? • 2012 onwards - The Australian Qualifications Framework (AQF) describes Course Learning Outcomes (CLOs) for all degree types • 2016 onwards – with the commencement of the AACSB journey, the CLOs were refocused around 3 School based core themes: 1. Applying knowledge 2. Industry-readiness 3. Developing an ethical, professional identity
- 18. Our Undergraduate CLOs THEME 1 APPLIED KNOWLEDGE THEME 2 INDUSTRY READINESS THEME 3 AN ETHICAL PROFESSIONAL IDENTITY UGCLO1 Applying Knowledge UGCLO2 Solving Problems & Thinking Critically UGCLO3 Communicating Effectively UGCLO4 Working in Teams UGCLO5 Creating & Innovating UGCLO6 Reflection & Feedback UGCLO7 Behaving Professionally & Ethically
- 19. Our Postgraduate CLOs THEME 1 APPLIED KNOWLEDGE THEME 2 INDUSTRY READINESS THEME 3 AN ETHICAL PROFESSIONAL IDENTITY PGCLO1 Mastering Knowledge PGCLO2 Solving Problems & Thinking Critically PGCLO3 Communicating & Working in Teams PGCLO4 Initiating & Completing a Project PGCLO5 Managing & Leading Ethically
- 20. AoL Implementation • CLO confirmation and curriculum mapping • Developing assessments • Developing marking criteria and rubrics • Measuring student achievement • Reporting on students’ achievement • Identifying improvements • Implementing improvements • Evaluating the impact of improvements
- 21. The Roll-out!2016 2017 2018 2019 Tranche Course Jul–Dec Jan–June Jul–Dec Jan–June Jul–Dec Jan–June Jul–Dec Pilot MPA M0 Refine CLOs M3 Criteria and Rubrics M1 Curriculum Map M4 Assess Outcomes M2 Assessment Strategy M5 Report and Analysis M6 Identify Improvements MHRM M0 Refine CLOs M3 Criteria and Rubrics M1 Curriculum Map M4 Assess Outcomes M2 Assessment Strategy M5 Report and Analysis M6 Identify Improvements M Marketing and Innovation M0 Refine CLOs M3 Criteria and Rubrics M1 Curriculum Map M4 Assess Outcomes M2 Assessment Strategy M5 Report and Analysis M6 Identify Improvements Tranche 1 MBA M1 Curriculum Map M4 Assess Outcomes M2 Assessment Strategy M5 Report and Analysis M3 Criteria and Rubrics M6 Identify Improvements MBAI M1 Curriculum Map M4 Assess Outcomes M2 Assessment Strategy M5 Report and Analysis M3 Criteria and Rubrics M6 Identify Improvements B Int. Resort Hotel Management M1 Curriculum Map B Hospitality and Tourism Management M1 Curriculum Map M3 Criteria and Rubrics M4 Assess Outcomes M5 Report and Analysis M6 Identify Improvements M7 Implement Improvements M3 Criteria and Rubrics M4 Assess Outcomes M5 Report and Analysis M6 Identify Improvements M7 Implement Improvements M2 Assessment Strategy M7 Implement Improvements M7 Implement Improvements M7 Implement Improvements M7 Implement Improvements M7 Implement Improvements M2 Assessment Strategy
- 22. Overview of Assessment Results for the Pilot PGCLO1 Mastering Knowledge PGCLO2 Solving Problems and Thinking Critically PGCLO3 Communicating and Working in Teams PGCLO4 Initiating and Completing a Project PGCLO5 Managing and Leading Ethically Master of Professional Accounting Group data Not enough data Missed Written Communication Group data CLOs overlapped in rubric Master of Human Resource Management Rubric too late Rubric late or not used Unit didn’t run Rubric late or not used Unit didn’t run Master of Marketing & Innovation Management Unit didn’t run HURRAH! Mostly group data Unit didn’t run Group data
- 23. Assessing - Bachelor of Commerce (Accounting) Course Learning Outcomes Subordinate Learning Outcome n (students) Does Not Meet Expectations (%) Meets Expectations (%) Exceeds expectations (%) UGCLO1: Applying knowledge UGCLO1.1 80 6 56 38 UGCLO1.2 59 10 10 80 UGCLO2: Solving problems & thinking critically Aggregated 59 14 5 81 UGCLO2.1 59 19 29 53 UGCLO2.1 59 12 5 83 UGCLO3: Communicating effectively Aggregated 54 2 63 35 UGLCO3.1 54 4 61 35 UGCLO3.2 54 0 33 67 UGCLO4: Working in teams N/A 54 0 31 69 UGCLO5: Creating & innovating N/A 59 12 5 83 UGCLO6: Reflection & feedback N/A No data collected yet. First iteration in 2019 UGCLO7: Behaving professionally and ethically N/A No data collected yet. First iteration in 2019
- 24. Assessing – Master of Business Administration Int. Course Learning Outcomes Subordinate Learning Outcome n (students) Does Not Meet Expectations (%) Meets Expectations (%) Exceed Expectations (%) PGCLO1: Mastering knowledge Aggregated 47 2 55 43 PGCLO1.1 & 1.2 47 2 53 45 PGCLO1.3 47 2 51 47 PGCLO1.4 47 2 55 43 PGCLO2: Solving problems & thinking critically Aggregated 71 51 21 28 PGCLO3: Communicating and working in teams Aggregated 72 4 54 42 PGLCO3.1-W 72 13 43 44 PGLCO3.1-O 72 4 49 47 PGCLO3.2 72 4 53 43 PGCLO4: Initiating and completing a project Aggregated 47 4 57 38 PGCLO4.1 47 0 57 43 PGCLO4.2 47 2 60 38 PGCLO5: Managing and leading ethically Aggregated 67 13 40 46
- 25. Designing Assessments Identification of assessment instruments and measures • Assessment instruments should be direct (assignment, case study, exam), with the use of indirect instruments to support (student evaluations, graduate survey, employer feedback). • Assessments must measure individual performance against the outcome/objective. • Establish benchmarks and descriptors to evaluate student performance (marking rubrics).
- 26. Reviewing Data Collection, analysis and dissemination of assessment information • Student performance against outcomes and objectives must be collected and analysed by a committee. • The collection process should be systematic across the review cycles. • The School will collect (direct measure) data every semester through our Learning Management System (LMS).
- 27. Use of data for improvement Using assessment information for continuous improvement • Must document the process for continuous improvement using the performance data gathered and analysis undertaken: ‘close the loop’. • Examples of student performance on assessment measures should be used to demonstrate the continuous improvement of curricula.
- 28. Never send a boy to do a man’s job. • AoL was more complex to implement than anticipated. • AoL rests on assessment artefacts which are not always integral to the development of units (afterthoughts). • Curriculum mapping revealed some gaps, misalignments and differing interpretations. • In order to meet timelines, quick fixes were applied with reviews to follow. Image: Accessed at Lakeview consulting: http://www.lakeviewconsulting.net/mickis-blog/
- 29. The way forward • Move away from compliance and move towards continuous improvement • Reframe the work of course assessment so that it enhances the learning experience. • Celebrate the AoL process, not the achievement of compliance. • Be more directive, and provide more support and encouragement for AoL processes. • Foster constructive conversations and actions about assessment throughout the development process to ‘close the loop’. • Recognise and reward the development processes around AoL through performance management to close the gap between espoused and enacted quality assurance. • Inject new blood! Image source: Hello I’m Nik from Unsplash, accessed on 7th August 2018
- 30. Learnings • AoL, we should look at it as a continuous improvement process, the work is never truly done! • Expect the unexpected all the time! • You need a firm but patient driver for the initiative • Circumstances beyond ones control – timetabling • Non-compliance – inadvertent and intentional • The ‘yes, yes’ people can be the ‘no, no’ people! • It’s a JOURNEY! Image: Filip Bunkens, accessed at Unsplash, 6th February 2018
- 31. 19/09/2018 Assurance of learning & accreditation: Re-aligning practices and attitudes 31 Thank you for your time and attention!
Editor's Notes
- #11: It’s about having an evidence base to underpin continuous improvement for student achievement. Hence the picture of a student using an iPad. I can find a nicer picture of a student but you get the drift.
- #22: This is a placeholder. Need to revise this.
- #24: The curriculum team decided not to assess the subordinate learning outcomes of CLO1 in a single unit, preferring to split it into two more appropriate units that develop theory and practice respectively. Consequently, it was not possible to aggregate the subordinate outcomes into one single figure. Discussions ensued about the appropriateness of this when the subordinate outcomes were intended to be linked. The results for UGCLO2 were unusual but deemed acceptable because the cohort struggled with conceptualising theory and applying technical skills to solve problems. The unit coordinator agreed to check the calibration of his marking rubric to see if the results were fair. The Accounting major is to be reviewed in 2019 as per the normal 5-year continuous improvement cycle, so it was decided to continue with the existing assessment instruments until the review was completed.
- #25: Most results for the MBAI were deemed normal in distribution except for PGCLO2. This CLO was being assessed by an examination question in a core finance unit. The question itself was deemed appropriate but the examination setting and the marking key did not produce results that were considered acceptable by the curriculum team and a better instrument was to be developed for the next iteration. The assessment task for PGCLO4 was considered unsuitable, being too reflective and not having a strong evidence base for the task. The task was to be revised for the next timetabled offering. The data for PGCLO3 and 5 are drawn from a unit with a mixed cohort as the unit is the core capstone experience for three courses. The MBAI is also offered in different delivery periods in Sri Lanka. The AoL reporting method requires further development to isolate student achievement by cohort and by location to ensure consistency of delivery and assessment. This cross-check of results will also extend to online versus on-campus students to provide solid evidence of consistency across modality. The current ECU continuous improvement activities already report results by location and mode but the AoL process has yet to integrate these into a single reporting method. Better sustainability of the AoL process is an important development aspect for the School.
- #26: We’re not actually using any indirect measures at present. You might want to pull out that bit above.