Representing Situations in Assessment - Getting better value from our investment
0 likes324 views
The document discusses situational effects in assessments. It finds that (1) simulations provide low incremental validity over context-free measures, (2) situational models provide a better fit than trait-only models for assessment center data, and (3) situational influences can be disentangled from traits using situational judgement tests. Cognitive models of situations show how encoding and judgments are affected by various situational dimensions. Improving assessments requires evaluating both situational and dispositional factors.
Representing Situations in Assessment - Getting better value from our investment
- 1. Representing Situations in Assessment – extracting better value from our investment Division of Occupational Psychology Annual Conference 2011 Michael Burnett & Almuth McDowall
- 2. Overview • The Problem with Situations • Disentangling Situational Influences: • Part 1: Predominance of situational effects in high stakes Assessment Centres • Part 2: Understanding trait v. situational effects in low stakes Situational Judgement Tests • Part 3: How cognitive Situational Models address sources of situational effects • Implications
- 3. The Problem with Situations •Techniques used today range from ‘context free’ cognitive ability and personality measures to simulations (ACs and SJTs) of work or organisational situations •The first problem is that simulations give significant (ACs .281, SJTs .18-.382) but in absolute terms low additional incremental validity over ‘context free’ measures • The second problem is explaining what simulations actually measure – e.g. the methods effect in ACs3, construct validity of SJTs4 • The implication is that applied psychology needs to provide a better account of individual differences in dealing with situations * 1AC Criterion Validity –Harmelin et. al. (2007) 2SJT Criterion Validity - Christian et. al. (2010) 3Sackett & Dreher (1982) 4McDaniel & Whetzel (2005)
- 4. Multimedia based simulations will happen • Multimedia-based ‘simulations’ in assessment will grow quickly • Video-based, Avatar-based simulations (Second LifeTM), 2-D Animations, Apps (i-phone/ipad/mobile), Games • Occ. Psych. Design Methodology needs to adapt • Maintain control over what is being measured!
- 5. Measures Exercises T Part 1: The predominance r of the situation a i t s Model x2 df GFI AGFI RMSEA NNFI SRMR Study 1a Traits Only 3034.6 67 0.69 0.51 0.22 0.53 0.17 (CT0E) Exercises Only 96.6 57 0.99 0.97 0.027 0.99 0.014 (0TCE) Correlated Traits 208.05 59 0.97 0.95 0.052 0.98 0.047 & Exercises (CTCE) Correlated 62.45 44 0.99 0.98 0.021 1.0 0.013 Exercises & Traits (CECT)*
- 6. Situational effects in high stakes simulations Structural Equation Models of assessment centre results based on situations (exercises) provide a good fit to the data, trait-only (competency dimensions) models do not fit The best fit combines situations moderated by trait (competency dimensions) factors Adding in Cognitive ability and Big 5 Personality dimensions showed best fit when added to the Exercise model, cognitive ability predominated Models that did not fit included Exercise Type model and Trait Activation model (Lievens et. al., 2006)
- 7. Part 2: Exploring situational effects • Use of low fidelity simulation based on SJT method to explore comparative effect of situational and ‘trait’ effects •Academic situations used with samples drawn from University and job applicants seeking SJT practise (n. 339) • Situational influence assessed by examining within-situation item relationships (zero correlation expected if competency/traits determine performance) • Competency and personality trait influence assessed by examining across-situation scale relationships between SJT items and independent measures of competencies & traits
- 8. Design of Situational Judgement Items Situation Situation 1 2 Item 1 Item 2 Item 3 Item 4 Item 1 Item 2 Item 3 Item 4 Plan & Relate & Persuade & Analysis Organise Network Influence Conscientiousness Openness Agreeableness Extraversion
- 9. Situational Judgement Item You have been selected to present a lecture to your course in front of the head of Department and senior tutors 1. You set yourself objectives for what you are going to say to include new and innovative ideas (Planning_Open) 2. You try and make sure that the content of your lecture will appeal to both students and staff (Relating_Agreeable) 3. You aim to keep the audience captivated by questioning established assumptions (Analysis_Extravert) 4. You check your material and rehearse the presentation to ensure you appear credible (Persuading_Conscientious)
- 10. Likert what I ‘would do’ responses Plan & Relate & Persuade & Analysis Organise Network Influence .36 .33 .24 .10 Plan & Relate & Persuade & Analysis Organise Network Influence Average .12 Situation (.06 to.20) Conscientiousness Openness Agreeableness Extraversion .31 .22 .05 .13 Conscientiousness Openness Agreeableness Extraversion
- 11. Part 3: Understanding situational effects • Situations are multidimensional - consider effects on encoding & retrieval: (Zwaan & Radvansky, 1998; Magliano et.al. 2008): - Space – context, layout and movement - Time – chronology of events, temporal shifts - Protagonists - Actors & Objects - Intention – Goals & Motivations - Causation – inferences of what has caused events or what could happen next
- 12. Cognitive Model of Situation Assumption about power 2. Time relationship5 Goals & motivations4 Assumed ‘agreement’ Manager3 becomes disagreement Goals & motivations4 Manager’s expectation 1. Space becomes “use of initiative” Goals Supplier3 Assumption Participant about power as relationship5 Protagonist3 ‘Non-performing’ supplier Goals & motivations4 becomes threat to supply chain The Situational Judgement 3. Actors 4. Motivations Causality5 determines these………….. 5. Causes
- 13. Personality & Competency effects on encoding & judgements Person Variables e.g. goals, domain knowledge competences, personality traits, state Situation Actual as construed behaviour & encoded Time enacted Goals & Motivations A a I i Tutor Goals & B b Motivations II ii C c Influence on future evaluation III iii D d Space Goals Student IV iv E e Protagonist V v F f Influence on future relationships VI vi Goals & Motivations G g VII vii Objective Potential characteristics behaviours of Situation afforded Adapted from Reis, 2008
- 14. Implications •Performance in Simulation-based assessments is determined by both situational and dispositional (trait & competency) factors •AC feedback should reflect both aspects •SJTs can be structured to return required facets •Situational Models hold promise in improving the yield from simulations (if we change the paradigm) •assessing encoding as well as judgement to widen the range of information gathered from assessments •providing a systematic way of describing situations to understand how these affect behaviour (IRT correlates) •enabling greater control over how situations are used in simulations including use of multimedia & gaming techniques (impact on Occ Psych methodology)