MANOVA/ANOVA (July 2014 updated)
3 likes1,036 views
This document summarizes and critiques a research paper that examined the effects of handicap source, frequency, and performance on observer impressions. It used a 2x2x2 factorial ANOVA/MANOVA design. The summary found partial support for the paper's 3 hypotheses but noted low variance explanation and some contrary findings. The critique raised concerns about not testing assumptions, excluding variables from MANOVA, and small effect sizes. It concluded the paper advanced the field but improvements could have been made by addressing the critique's methodological points.
MANOVA/ANOVA (July 2014 updated)
- 1. July 2014 updated Prepared by Michael Ling Page 1 QUANTITATIVE RESEARCH METHODS SAMPLE OF ANOVA/MANOVA ANALYSIS Prepared by Michael Ling Reference: McElroy, J. C, & Crant, J. M. (2008). “Handicapping: The effects of its source and frequency,” Journal of Applied Psychology, Vol. 93, 893-900.
- 2. July 2014 updated Prepared by Michael Ling Page 2 I. INTRODUCTION Prior to the event of performance, handicapping meant setting up to “deflect blame away” in case of failure and “accept credit for success by having overcome difficulty” in case of success. Handicapping was considered a “direct impression management tactic” in relation to the “credit or blame attributed to and level of affect” allocated towards the actor by an observer. Past research in handicapping was limited to single instance of self- handicapping. This paper contributed to handicapping research by extending a single instance self-handicapping event into a much broader framework where multiple instances of handicapping and indirect handicapping were evaluated. The research question was to which extent that frequency and sources of handicapping could influence reactions of an observer and his/her perceived credibility about the actor concerned. These effects were further examined in the context of success or failed performance. Three hypotheses were proposed and ANOVA/MANOVA procedures were used as the research method.
- 3. July 2014 updated Prepared by Michael Ling Page 3 II. SUMMARY A 2x2x2 factorial ANOVA/MANOVA design was used to examine the main and interaction effects of three factors - handicap source (self or other), frequency (once or multiple times) and performance (success or failure) – on credit/blame, interpersonal affect and perceived credibility. The sample size was 246 with an average cell size of 28 to 34, and participants were randomly assigned to one of eight cells. Manipulation checks were conducted on the measurement scales to ensure their consistency and reliability. Partial support was found in hypothesis 1, which stated that frequency moderated the source of handicapping and the observer impressions where (i) impressions (credit/blame and interpersonal affect) were more favourable in third- party handicaps than self-handicaps; and (ii) increased frequency reduced impressions more strongly in self-handicaps then third-party handicaps. The results showed that there was (i) a significant Source*Frequency effect (p < .05) on the set of dependent variables (MANOVA); (ii) a significant Source*Frequency effect (p<.05) on interpersonal affect (ANOVA); and (iii) a moderating effect of Frequency on Source and Interpersonal affect. However, no significant Source*Frequency interaction effect was found on credit/blame (ANOVA). Partial support was found for hypothesis 2, which stated that frequency moderated the source of handicapping and the perceived credibility of the handicapping information where (i) handicaps were more credible in third-party handicaps than self-handicaps; and (ii) increased frequency reduced credibility more strongly in self handicaps than third-party handicaps. The results showed that there
- 4. July 2014 updated Prepared by Michael Ling Page 4 was Source*Frequency interaction effect (p < 0.05) on Credibility (ANOVA). Despite the significant interaction effect, Eta-squared (η2) was 2 percent. Contrary to expectations, self-handicaps were found more credible than third-party handicaps in single handicaps. Partial support was found for hypothesis 3, which stated that performance moderated the source and frequency of handicapping and the observer impressions and perceived credibility where (i) impressions were more favourable following failed than successful performance; and (ii) perceived credibility was more favourable following failed than successful performance. It was found that multiple handicaps decreased credibility for all handicaps except in the case of third-party handicaps following failed performance.
- 5. July 2014 updated Prepared by Michael Ling Page 5 III. CRITIQUE The use of ANOVA/MANOVA procedures were appropriate because it examined the main and interaction effects of independent categorical variables (Source, Performance and Frequency) on multiple dependent interval variables (Credit/blame, Interpersonal affect and Credibility) by comparing group differences. The sample design suggested that a balanced design was adopted. In MANOVA, the cell sizes should be roughly equal because normality of the dependent variables was important. The coefficient alpha values for the measurement scales were high and the outcomes of the manipulation checks were satisfactory. A key weakness of the paper was that it did not specify the assumptions of the ANOVA/MANOVA tests. For example, i. No results were provided for univariate and multivariate normality that the dependent variables, and their combinations, were distributed normally. No Scatterplots were checked for linear relationships among the dependent variables. ii. No results of multicollinearity were provided to examine the correlations of the dependent variables. iii. No results of multivariate outliners, such as Maximum Mahalanobis Distance, were provided. iv. No results of homogeneity of the covariance matrices, such as Box’s M test, were provided. If Box’s M test showed the covariance matrices were significantly different across levels of the independent variables, it
- 6. July 2014 updated Prepared by Michael Ling Page 6 indicated an increased possibility of Type I error and hence there was a need to use a smaller error region than p < .05. v. No evidence was provided for the independence of observations. As the questionnaires were completed by the respondents in their workplaces, it was probable that the respondents might have discussed the questions amongst themselves. Data quality would be a potential issue. Hypothesis 1 Despite that the Source*Frequency interaction effects were reported significant in Hypothesis 1, there were a few areas of concerns:- i. Wilks’s λ was a measure of the percent of variance in the dependent variables that was not explained by differences in the level of the independent variable. Wilks’s λ for Source*Frequency was .97 in the MANOVA test, which meant that 97 percent of variance was still unexplained. ii. The Eta-squared for Source*Frequency was 3 percent, which meant the percent of total variance in the dependent variable explained by the variance between groups formed by the independent variable was only 3 percent, which not very impressive despite a significant result. iii. If the MANOVA omnibus test was significant, it was common practice to conduct separate ANOVAs. However, considerations such as Bonferroni adjustment should have been given to adjust the
- 7. July 2014 updated Prepared by Michael Ling Page 7 significance cut-off level of ANOVAs in order to minimize the probability of Type I error. Hypothesis 2 An area of concern was that credibility had not been included in the MANOVA omnibus test. The authors argued that they “felt that these reactions to handicapping were conceptually distinct enough to preclude us from including all of the dependent variables in just one… (MANOVA)”. This argument was not relied upon any substantive literature but was based on what the authors “felt” it should be. On the contrary, it was possible that credibility, credit/blame and affect were correlated. If this was the case, the independent ANOVA would have ignored their interrelations and substantial information would be lost. The resultant p values for the 1-way independent ANOVA would have been incorrect. The Bartlett’s Test of Sphericity could have been used to test the hypothesis that the population correlation matrix was an identity matrix. If the determinant was small, independence of the variables would be rejected and there was a need for MANOVA. Again, the Eta-squared for Source*Frequency was 2 percent, which was not impressive at all despite a significant result. No discussions were provided on the adjustments in relation to the treatment of experiment-wise and testwise errors in the univariate ANOVA tests. Hypothesis 3
- 8. July 2014 updated Prepared by Michael Ling Page 8 The MANOVA omnibus result failed to find significance in the three-way interactions effect of Performance*Source*Frequency on impressions. A three-way interactions effect was found on credibility in the ANOVA test. Despite the interaction effects were reported significant, a concern was that Eta-squared (η2) of the three- way interactions effect was very low (0.02) in the ANOVA result. Although all forms of handicaps were found more credible following failure than success, their mean differences were very small in the range of 0.13 to 0.71, as shown in the table below. A larger sample size could have been considered to test if reasonable effect sizes could be achieved. Proper measures of effect sizes such as Cohen’s d, as the difference in group mean divided by the pooled standard deviation, should have been provided to measure effect sizes. Success Performance Failure Performance Mean Difference Self-handicaps Single 4.11 4.82 0.71 Self-handicaps Multiple 3.65 3.92 0.27 Other handicaps Single 3.98 4.11 0.13 Other handicaps Multiple 3.61 3.92 0.31 Note: the mean values above were extracted from Figure 3. Overall Assessment Though the authors reported partial support of the three hypotheses, the low values of Wilks’s λ and Eta-squared created some concerns. A large portion of the variance was not accounted for in the model. Not all relationships, despite their importance, suggested in the hypotheses were found significant. The assumptions of
- 9. July 2014 updated Prepared by Michael Ling Page 9 ANOVA/MANOVA procedures were left untested. The exclusion of Credibility in the MANOVA test was not adequately supported. No consideration was given to the need for adjustment of significance criteria in ANOVA. Contrary evidence was found with respect to self-handicaps and multiple handicaps in hypotheses 2 and 3 respectively. As a result, the conclusions drawn about the hypotheses were not totally trustworthy.
- 10. July 2014 updated Prepared by Michael Ling Page 10 III. CONCLUSION The contribution of the paper rested on advancing a theoretical framework in handicapping research, from single instance self-handicapping into multiple instances of handicapping and indirect handicapping. The authors started off on the right track by using a full factorial design and ANOVA/MANOVA procedures as the research method to establish their hypotheses. Unfortunately, the results were not quite satisfactory as all hypotheses were only partially supported and the variances explained by the independent variables were very negligible. The research could have improved by addressing the concerns raised in this critique. In particular, assumptions of the ANOVA/MANOVA procedures needed to be tested and data quality needed to be reinforced, especially independence of samples; a redesign of the scenario-based experiment methodology, for example, by replacing questionnaire with testing respondents in a laboratory setting. Given that there was only partial support of the three hypotheses, an important step would be a review of the handicapping model from the theoretical perspective.