Digital Skills Gap Peer Learning Activity - Analysis of usage of LMS activities and resources as an aspect of e-course quality
Download as PPTX, PDF0 likes94 views
The document discusses the analysis of e-course quality at the Faculty of Organization and Informatics, University of Zagreb, focusing on the use of learning management systems (LMS) like Moodle from 1962 to 2016. It outlines the various levels of e-learning implementation, methodologies for evaluating e-course structures, and the construction of composite indicators to assess e-course richness. Additionally, it describes the data analysis techniques employed to improve course design and delivery.
Digital Skills Gap Peer Learning Activity - Analysis of usage of LMS activities and resources as an aspect of e-course quality
- 2. Faculty of Organization and Informatics Pavlinska 2 42 000 Varaždin Croatia Analysis of usage of LMS activities and resources as an aspect of e-course quality 1962. - 2016. Prof.dr.sc. Diana Šimić diana.simic@foi.hr Darko Grabar, mag.inf. darko.grabar@foi.hr
- 3. UNIVERSITY OF ZAGREB - UNIZG The oldest and the largest university in Croatia 1669. Founded 29 Faculties 3 Art academies 3 University centres • SRCE- IT Centre • University & National Library • Student Centre 80 000 Students (all study levels) 9 000 Teaching staff (all study levels) 1874. Became State university
- 4. FACULTY OF ORGANIZATION AND INFORMATICS - FOI • Medium sized faculty • Leading higher education institution in Croatia to provide education in applied information technology and information sciences 1974. Founded
- 5. FOI STUDENTS IN NUMBERS OVER 3000 STUDENTS MORE THAN 500 DEGREES PER YEAR FROM MORE THAN 100 SCHOOLS FROM REGION
- 6. FOI E-Learning in numbers OVER 300 STUDENTS MORE THAN 500 DEGREES PER YEAR FROM MORE THAN 100 SCHOOLS FROM REGION OVER 300 online courses EACH YEAR MORE THAN 3000 ACTIVE ONLINE USERS
- 7. Main LMS - Moodle 7https://elf.foi.hr/ 2006/2007
- 8. 8
- 9. Blended learning at FOI • Three levels of usage of e-learning at FOI • Level 1: To enhance availability of teaching materials and teacher to student communication • Level 2: To facilitate better acquisition of knowledge through advanced integration of LMS system with traditional classroom • Level 3: Improvement of teaching methods and techniques of teaching through hybrid course organization according to the instruction design principles 9
- 10. Blended learning at FOI • Three levels of usage of e-learning at FOI • Level 1: To enhance availability of teaching materials and teacher to student communication • General course information • Learning outcomes • Course plan and work program • Literature • Selected teaching materials • Student to teacher communication by e-mail • General discussion forum 10
- 11. Can quantitative analysis give us some useful information 11
- 12. Can quantitative analysis give us some useful information 12
- 13. What if we dig deeper into LMS 13
- 14. 14
- 15. 15 Typical FOI Moodle database has around 390 tables containing more than 5 GB of data for each academic year ~ 15 000 average books
- 16. What we can use for analysis • Student and teacher overall activity • Student usage of specific resources • Number od specific resources, modules used in course • Number of forum posts, student discussions,.. • Student grades, points, badges, activity completions, … • … 16
- 17. Faculty of Organization and Informatics Pavlinska 2 42 000 Varaždin Croatia Rich or sparse structure? 1962. - 2016.
- 18. Goal • To identify structurally / organizationally poor e-courses • To provide additional training and support to authors of structurally poor e-courses 18
- 19. Questions … • When is an e-course well structured (makes good use of LMS)? • … uses more types of resources and activities • … use LMS for two-way communication with students • … LMS is used for formative assessment and self- assessment • … various types of activities and resources are used … 19
- 20. How do we compare e-courses? • Use AHP to combine multiple criteria for deciding which e-courses have richer structure • Create a composite indicator and rank e- courses 20
- 21. Composite indicators • Used to compare countries, universities etc. with multiple criteria • Combine criteria into a single number • Examples: • Competitiveness index • e-Readiness index • ARWU … Academic Ranking of World Universities 21
- 22. Methodology • OECD and JRC (2008) Handbook on constructing composite indicators methodology and user guide https://composite-indicators.jrc.ec.europa.eu/ • Overview of methodology • Checklist for building a composite indicator (CI) 22
- 23. CI building workflow 1. Theoretical framework 2. Data selection 3. Imputation of missing data 4. Multivariate analyses 5. Normalization 6. Weighting and aggregation 7. Uncertainty and sensitivity analysis 8. Back to data 9. Links to other indicators 10. Visualization of results 23
- 24. 1. Theoretical framework • What constitutes richness of e-course structure? • Various forms of content and activities (educational content) • Various types of assignments and assessments (assessment) • Rich communication with students (communication) • Use of LMS elements to create structure in educational content / activities (structure) • …. what else? … we will try to figure it out together 24
- 25. 2.-4. Next steps • Selecting data • Information routinely available from an LMS system • Imputation of missing data • Not necessary in our case • Multivariate analyses • Descriptive statistics and visualization • Principal components analysis • Variable clustering 25
- 26. 5-7 Next steps • Normalization • Weighting and aggregation • Uncertainty and sensitivity analysis • we illustrate some of the steps … 26
- 27. Descriptive statistics 27 vars n mean sd skew kurtosis se Q0.25 Q0.75 assign 1 210 6.73 6.89 2.23 6.63 0.48 2.25 9.00 broj_aktivnosti_na_predmetu 2 210 7.38 2.59 0.59 0.88 0.18 6.00 9.00 broj_forum_diskusija_samo_vijesti 3 210 11.97 10.29 1.19 1.51 0.71 4.00 17.00 broj_grade_itema 4 210 13.68 13.89 2.65 9.15 0.96 5.00 18.00 broj_labela_na_predmetu 5 210 6.44 9.19 1.87 4.25 0.63 0.00 10.00 broj_nastavnika 6 210 3.07 1.53 1.07 1.64 0.11 2.00 4.00 Outliers Skewed distribution
- 28. Principal components analysis (PCA) 28 Variable RC1 RC2 RC3 RC4 RC5 l.broj_testova_na_predmetu 0,93 l.quiz 0,92 l.broj_pitanja_na_predmetu 0,88 l.page 0,62 broj_aktivnosti_na_predmetu 0,56 0,37 0,38 0,44 l.broj_pracenih_aktivnosti_na_predmetu 0,36 -0,32 l.assign 0,83 broj_tipa_zadaca_novi 0,71 l.broj_grade_itema 0,53 0,70 l.tjedni_broj_zadaca 0,68 l.razlika_gradeitem_ocijenjeno 0,41 0,57 -0,30 l.broj_skrivenih_nastavnih_cjelina 0,57 -0,52 l.Omjer.labela.i.tema 0,94 l.broj_labela_na_predmetu 0,90 l.broj_vidljivih_nastavnih_cjelina 0,59 l.broj_resursa_na_predmetu 0,57 l.broj_forum_diskusija_samo_vijesti 0,56 0,48 l.choice 0,54 l.url 0,30 0,46 l.broj_stvarno_ocjenjenih_stavaka 0,37 0,40 0,65 broj_forum_postova_bez_vijesti_po_diskusiji 0,41 broj_forum_diskusija_bez_vijesti_po_studentu 0,36 Identify: • direction of maximal variability • variables that contribute most to differentiational • variables that vary together Reliability – Cronbach 𝛼 Sampling adequacy KMO
- 30. Normalization • Need to bring all variables to the same scale before aggregation • Look-out for outliers, very skewed data • Some approaches: • Ranking • Standardization (z-score) • Min-max (resulting in range 0 to 1) • Distance to a reference course • Transform to categorical variable using quantiles 30
- 31. Weighting • Choosing weights • From statistical models: • Equal weights • PCA / FA • Data Envelopment Analysis (DEA) • Participatory methods • Budget Allocation Process (BPA) • Analytical Hierarchical Process (AHP) • Conjoint Analysis 31
- 32. At the end • Aggregation • Linear • Geometric • …. • Sensitivity analysis • Vary choices from each step to see how they influence the final results 32
- 33. Back to the Theoretical Framework • How do we choose which variables / data should enter into the CI, and what is the structure? • From literature • From own experience • From other experts … like you 33
- 34. Q-Sort • Developed by Stephenson (1953) for personality research • Often used in development of conceptual frameworks • Experts provided with a set of statements / items • Asked to provide measure of importance for each item and dimension. Stephenson, W (1953) The Study of Behavior. Chicago: University of Chicago Press. 34
- 35. Q-method • Analysis of consistency of responses • Analysis of typical profiles of attitudes • Selection of relevant features for the theoretical framework 35
- 36. Faculty of Organization and Informatics Pavlinska 2 42 000 Varaždin Croatia Practical activities 1962. - 2016.