Tenc Winterschool09 Davinia Slideshare

Analysis of Multiple Pilots for ICT-supported Lifelong Competence Development Davinia Hernández-Leo [email_address] Winter School 2009, 1-6 February 2009 Innsbruck, Austria http://www.tencompetence.org/node/167

Outline : Introduction Experimental models for validating technology TENCompetence pilots Individual activity Expert group activity Mixed group activity General discussion

Overview Experiences and motivation Research questions Reasons for doing research / possible outcomes Literature review Context of the research /conceptual framework Methods to answer research questions

It is not simple… “ The particular technique is effective” It conveys not real information Way to measure it (to say one technique is more or less effective than another) “ It works” The creator of the technology shows that it works. Very rarely does such experimentation involve any collection of data to show that it is effective… “ The results were positive” What does it means? “ I tried it, and I like it” Experimentation helps determine the effectiveness of proposed theories and methods

Overview, experimentation in ICT Experiences and motivation Research questions Reasons for doing research / possible outcomes Context of the research /conceptual framework Experimental validation methods Literature review Experimental Methods Observational Historical Controlled

Experimental validation methods (for ICT) Observational methods Collects data as a project develops Little control over the development process other than through using the new technology that is being studied Historical methods Collects data from projects that have already been completed The data already exist; it is only necessary to analyze what has already been collected Controlled methods Provides for multiple instances of an observation statistical validity of the results The classical method of experimental design in other scientific disciplines

Example (Zelkowitz, Wallace, 1998)

TENCompetence http://www.tencompetence.org/ Develop a technical and organizational infrastructure, which is both open source and standards based, to support lifelong learning in Europe, to serve the needs of individuals, groups, and organizations. 7 Core Requirements: 1. Support new pedagogical & organisational Models for Lifelong Competence Development 2. Support individuals to search the most suitable formal and informal learning activities 3. Stimulate pro-active sharing of resources 4. Support competence assessment 5. Provide various forms of user support services 6. Provide decentralized, self-organised management 7. Integrate isolated models & tools from four different areas (KM, Learning, PDP, CoP)‏

TENCompetence pilots Validation of the real world strength of TENCompetence solutions Cycle 1 : ‘Proof of concept’ Cycle 2 : Validate that the TENCompetence solutions are usable Cycle 3 : Business model demonstrators

Multi-pilot evaluation Core Use Cases Want to improve a specific competence Want to study for a new job Want to keep up-to-date Want to explore, select & connect to the learning resources, units of learning, people, etc. in a certain field Want to assess my competences Want to reflect on my competences Want support for some non-trivial learning problem 1, 4 1, 2, 3, 4, 5, 6 1, 2, 3, 4 1, 2, 3, 5 Comp. dev. for social inclusion Agora LLL in the film industry Digital Cinema Incr. capacity of w.m. professionals UNESCO-IHE Water Management Continuing learning of teachers ICT Teacher Learning

Multi-pilot evaluation (cycle 1) Comp. dev. for social inclusion Agora LLL in the film industry Digital Cinema Incr. capacity of w.m. professionals UNESCO-IHE Water Management Continuing learning of teachers ICT Teacher Learning ISSUE: See if the TENCompetence concept is feasible and capable of being exploited in a useful manner Cycle 1 LLL in the film industry 1)LD system 2)PCM1.0 Experiment, two groups. Questionnaires Continuing learning of teachers 1) PCM1.0 Experiment, two groups. Questionnaires Better job (performance), prof. setting, incl. workplace, certificate, CoP blended Better job (performance), informal setting, incl. workplace, distance

Cycle 1 ‘proof of concept’ and approach in cycle 2 Better results than alternative methods: higher number of people passed the competence assessment, feeling of being more in control of their own learning Busy professionals Cycle 2, focus on what participants learned and how this influences their competence level and future plans and how they use the tools

Multi-pilot evaluation (cycle 2) PCM2.0 PDP, LD system, TENTube Web container Better job (performance), share; informal setting, incl. workplace Continuing learning of teacher s PCM2.0 (PDP, LearnWeb2) Better job (performance), prof. setting, incl. workplace, certificate,CoP blended CoP for special education CD Special Education Bulgaria Comp. dev. for social inclusion Agora LLL in the film industry Digital Cinema Incr. capacity of w.m. professionals UNESCO-IHE Water Management LLL in the film industry Comp. dev. for social inclusion Incr. capacity of w.m. professionals Continuing education of teachers ICT Teacher Learning

Individual activity Scan the text, and complete the titles of the sections, 20 minutes Take a look again at the text and look for answers to these questions ( 15 minutes ): Which type of experimental method is applied? Which are the data collection instruments? Type of data collected? Why? How the data have been analyzed? Which are the research questions and the main conclusions? (You may also identify missing information in the text…)

Expert group activity Discuss (20 minutes): Which type of experimental method is applied? Which are the data collection instruments? Type of data collected? Why? How the data have been analyzed? Which are the research questions and the main conclusions? (You may also identify missing information in the text…)

Mixed group activity Experts explain each pilot to the other members of the mixed group according to the previous questions (30 minutes) Which type of experimental method is applied? Which are the data collection instruments? Type of data collected? Why? How the data have been analyzed? Which are the research questions and the main conclusions? (You may also identify missing information in the text…)

Discussion – whole group Which type of experimental method is applied? Which are the data collection instruments? Type of data collected? Why? How the data have been analyzed? (You may also identify missing information in the text…)

Overview, experimentation in ICT Experiences and motivation Research questions Reasons for doing research / possible outcomes Context of the research /conceptual framework Experimental validation methods Literature review Experimental Methods Data collection instruments Data analysis Observational Historical Controlled Questionnaires Interviews Log files … Qualitative Quantitative

Data generation or collection instruments The means by which you produce empirical (field) data or evidence (quantitative or qualitative) Examples Interviews: a kind of guided conversation (one-to-one and group) Observations: watching what people actually do (also hearing, smelling, touching…) Questionnaires: respondents are asked to answer a pre-defined set of questions Documents: reports previous to a research (e.g., minutes of meetings) and made for the purposes of the research (multimedia documents, visual data sources e.g. photos, animations, etc.) Log files: automatic data collected by software tools …

Data analysis Analyze the data looking for relationships or themes Quantitative data analysis uses mathematical approaches such as statistics to examine and interpret the data Note: the reason for all the statistical apparatus in quantitative research is to explain that relationships between variables are not due to chance (but cause and effect relationships). Remember though that statistics are useful for showing trends and indicating probabilities, but they don’t necessarily tell us what is going to happen in a particular individual case Qualitative data analysis looks for themes and categories within the words people use or the images they create Note: the conclusions of qualitative research are not as generalizable as those of quantitative research. Generalizations must be limited to the subjects and the settings studied.

Using different data collection instruments Look at the issue of interest in different ways More data, could improve the quality of the research Findings from one instrument to be corroborated or questioned by comparing with data from another method E.g., if you are carrying out a case study of how people interact with a web-based system: - you could observe people using it, - collect data electronically about their movements around the website, - interview them about their attitudes and perceptions concerning it, and - ask them to complete evaluation questionnaire The people in the interview might tell they found the website easy to use, but the log files show they accessed frequently the help system….

Enhance validity of findings Triangulation : use of more than one data collection instrument to corroborate findings and enhance their validity Different types: Triangulation : study uses two or more data collection instruments Time triangulation : study takes place at two or more different points in time Space triangulation : study takes place in two or more different countries or cultures Investigator triangulation : study carried out by two or more researchers that compare their results …

Mixed group, Cross-pilot analysis Which are the research questions and the main conclusions? Create a poster: Compare the two pilots Formulate findings than can be supported by the results of both piltos

References P. Goubil-Gambrell, “What do practitioners need to know about research methodology”, Proceedings of the IPCC 1991, pp. 243-248. M.V. Zelkowitz y D.R. Walace, “Experimental models for validating technology”, IEEE Computer, May 1998, pp. 23-31. Louys, A., Hernández-Leo, D., Schoonenboom, J., et al. “Self-Development of Competences for Social Inclusion Using the TENCompetence Infrastructure” accepted to be published in the Journal of Educational Technology & Society, Special Issue on ‘Technology Support for Self-Organized Learners’ Schoonenboom, J., Sligte, H., Moghnieh, A., Hernández-Leo, D., Stefanov, K., Glahn, C., Specht, M., Lemmers, R., (2008) “Supporting life-long competence development using the TENCompetence infrastructure: a first experiment”, International Journal of Emerging Technologies for Learning, vol. 3 53-59. TENCompetence, D4.4 - Report on the results of the evaluation of the Cycle 2 pilots, available at http://dspace.ou.nl/handle/1820/1719

Additional slides to be used during the session

Multi-pilot evaluation (cycle 2) ISSUE: See if the solutions developed to make the TENC concept a reality are usable LEARNERS How do learners use: Competences, competence development plans, actions and resources S upport for social interactions What functionalities do participants use and what for? How do they evaluate their usefulness ? How do the functionalities interlink ? To what extent match t he intention of the provider ? Specific challenges of each pilot PDP, LD system, TENTube Web container Better job (performance), share; informal setting, incl. workplace Intrinsic motivation to learn Non-formal setting, blended Better job (performance), prof. setting, incl. workplace, certificate,CoP blended PDP, LD system Questionnaires, log files, observations, discussion groups Questionnaires, log files PDP and external tool Cycle 2 Better job (performance), groups, distance setting, incl. workplace, certificate CoP for special education CD Special Education Bulgaria PCM2.0 Questionnaires Comp. dev. for social inclusion Agora LLL in the film industry Digital Cinema Incr. capacity of w.m. professionals UNESCO-IHE Water Management LLL in the film industry Questionnaires, log files Comp. dev. for social inclusion Incr. capacity of w.m. professionals Continuing education of teachers ICT Teacher Learning Continuing learning of teacher s PCM2.0 (PDP, LearnWeb2) Questionnaires

Specific challenges Agora pilot Learners involved have typically low educational profiles and have not always the necessary confidence to take the primary responsibility for the planning and performance of learning activities low computer skills , which are probably too low to make sensible use of the TENCompetence infrastructure non-formal learning context not possible to oblige participants to acquire these skills or to continue competence development (First contact with the approach, previous to a new longer Agora pilot in cycle 3 with new tools) Thus, we would expect that it would be much harder to achieve lifelong competence development in this pilot.

Tools: PDP tool, LD runtime, (PCM, ReCourse for the experts) Competence profiles: Basic English, Advanced English and ICT skills (participants free to create any PDP) more than 100 learners (>130 persons involved, experts, volunteers…) setting: computer room available in La Verneda (Agora), home Start: 15 September, Finish: 31 October 2008 Evaluation results in Louys, A., Hernández-Leo, D., Schoonenboom, J., et al. “Self-Development of Competences for Social Inclusion Using the TENCompetence Infrastructure” accepted to be published in the Journal of Educational Technology & Society, Special Issue on ‘Technology Support for Self-Organized Learners’ Further description of the Agora pilot

Data sources for the evaluation of the Agora pilot Records of opinions and observations of what was being perceived in Àgora once the pilot had finished (collected by Àgora staff) Observations post-pilot Qualitative descriptions of the context characteristics in which the pilot is framed Description of the Àgora context TENCompetence server logs, analysis of 512 sessions (a session is considered one usage period of a user from login to logout) Log files Qualitative: experts’ opinions two weeks before the end of the pilot Focus group with experts Qualitative: participants’ opinions two weeks before the end of the pilot Focus group with participants Record of observations (technical issues, about the activities, interactions with experts and other participants, behaviour, other incidents, etc.) The observations were done by 6 different experts (Àgora staff, UPF researchers) Observations during the pilot Quantitative participant characteristics, expectations and evaluation. Questionnaires before (pre-test) and after (post-test) the pilot experience Type of data Data source

Main results of the Agora pilot Despite the profiles of the participants (not higher education, low experience using computers), most of them completed the pilot experience a large majority of the participants including the less experienced in using computers used most of the PDP functionalities 1. TENCompetence can be useful and beneficial even for participants with low educational levels Partial results Results

Main results of the Agora pilot Prior to the pilot the participants had a preference for the traditional way of learning. The pilot meant a change in their learning habits they were very active and used quite often the main elements of the PDP tool they enjoyed the possibility to work at their own rhythm , to choose the activities in accordance to their interests and needs they asked to continue self-developing competences at home 2. Participants appreciated this new way of self-directed learning Partial results Results

Main results of the Agora pilot The self-assessment functionality encouraged the participants to reflect on new learning possibilities and on previous experiences The participants’ felt that they developed social and reflective skills The large majority of the participants let the system generate a plan based on their self-assessment and had high expectation with regard to this functionality, which should be improved The participants valued positively the potential of controlling their learning progress and being aware of their evolution 3. The experience fostered the participants’ reflection and self- confidence Partial results Results

Main results of the Agora pilot The participants found out that they could develop more competences thanks to the competence profile list provided by the PDP tool They did not only want to develop a concrete competence available in the system but others they did not think of before the pilot 4. The participants discovered further competence development options Partial results Results

Main results of the Agora pilot “ When can we register for the next pilot? ” Agora participant(s) The participants had a clear preference for the interactive activities and feedback provision (such as it can be done with LD) The users expressed the need of more communicative functionalities Some participants also requested some automatic assessment support It is important to provide different ways of organizing activities within personal development plans The users expect recommendations based on their personal needs (such as taking into account the proficiency level) 5. Recommendations for further developments Partial results Results

Specific challenges Water Management pilot Professionals with a need of developing competences to perform their job better, for whom receiving some kind of formal certificates is crucial in their career perspective, and for whom the choice of developing their competences on-line is a personal choice , working at a distance , need to communicate implications of the distributed audience, from many different countries including the Southern World

Further description of the WM pilot Tools: PDP tool, LD runtime, (PCM, ReCourse for the experts) (own server) Competence profiles: related to Flood Modeling Management 90 learners setting: home , workplace Start: 22 September, Finish: 23 November 2008

Data sources for the evaluation of the WM pilot Learners’ outcomes as described by the pilot implementers Final learners’ outcomes Qualitative descriptions of the context characteristics in which the pilot is framed Context TENCompetence server logs, analysis of 4095 sessions (a session is considered one usage period of a user from login to logout) Log files Record of observations by pilot implementers (technical incidents, about the activities, etc.) Observations during the pilot Quantitative participant characteristics, expectations and evaluation Questionnaires before (pre-test) and after (post-test) the pilot experience Type of data Data source

Main results of the WM pilot - Participants were homogeneous : highly educated , relatively young, and for most of them the motivation to join the pilot was job-related (study for a new function or job or improve their current job level, keep up to date within my existing function or job…) - Heterogeneous : from 47 different countries, different number of years of experience in the profession - The participants also recognized that they learnt with respect to the various competence types - Participants used shared blogs for communication purposes and rated it as (very) useful - The majority of the participants were very active in the pilot and more than a half of them were able to complete it 1. The TENC infrastructure can be used in LLL situation involving professionals in the WM area from all over the World with different job-related experience Partial results Findings

Main results of the WM pilot A large majority appreciated the way of learning and wished to continue to develop these competences further The possibilities to engage in self-assessment and choosing part of the order of the learning activities were largely appreciated by the participants The possibility of marking activities as completed and being able to see the history was found by many learners as a useful strategy to manage their learning progress 2. Encouraging potentials of TENCompetence for the support of this pilot scenario Partial results Findings

Tenc Winterschool09 Davinia Slideshare

More Related Content

What's hot (19)

Viewers also liked (8)

Similar to Tenc Winterschool09 Davinia Slideshare (20)

Recently uploaded (20)

Tenc Winterschool09 Davinia Slideshare