Diagnostic methods for Building the regression model
- 1. Building the Regression Model II: Diagnostics CHAPTER 10 APPLIED LINEAR STATISTICAL MODELS (NETTER) MEHDI SHAYEGANI M.SHGN@YAHOO.COM 1
- 2. Building the Regression Model II: Diagnostics We also described the plotting of residuals against predictor variables not yet in the regression model to determine whether it would be helpful to add one or more of these variables to the model. Added-variable plots provide graphic information about the marginal importance about predictor variable X. In addition, these plots can at times be useful for identifying the nature of the marginal relation for a predictor variable in the regression model. 2
- 3. Added variable plots: Added variable plots or partial regression plot: 1.shows the marginal importance of this variable in reducing the residual variability 2 may provide information about the nature of the marginal regression relation for the predictor variable Xk under consideration for possible inclusion in the regression model. 3 X1contains no additional information useful for predicting Y beyond that contained in X2 linear term in XI may be a helpful addition to the regression model Curvature effect in XI may be a helpful addition to the regression model
- 4. Multiple regression model with two predictor variables X1 and X2 In the previous plots: X2 is already in the regression model and X1 is under consideration to be added. Plot A) X1 contains no additional information useful for predicting Y beyond that contained in X2 Plot B,C) addition of X1 to the regression model may be helpful and suggesting the possible nature of the curvature effect by the pattern shown. 4
- 5. Example annual income of managers average annual income of managers during the past two years (X1) a score measuring each manager's risk aversion (X2) amount of life insurance carried (Y) Y = -205.72 + 6.2880 X1 +4.738 X2 5
- 6. Added-variable plot 𝑌𝑖 (𝑋2) = 𝑏0+𝑏2 𝑋𝑖2 >>>>> 𝑒𝑖(YI𝑋2) = 𝑌𝑖- 𝑌𝑖 (𝑋2) 𝑋𝑖1 (𝑋2) = 𝑏0*+𝑏0∗𝑋𝑖2 >>>>> 𝑒𝑖(𝑋1I𝑋2) = 𝑋𝑖1- 𝑋𝑖1 (𝑋2) What's the nature relationship in here? Not linear relation for X1 6
- 7. Example A fit of the first-order regression model yields: Y = -205.72 + 6.2880𝑋1+ 4.738𝑋2 in attention to plots : Residual plot shows >> a linear relation for X1 is not appropriate in the model already containing X2 But what is the nature of this relationship in here? For answer we have to use added-variable plot(b). Added-variable plot shows >> suggested the curvilinear relation between Y and X1 when X2 is already in the regression model is strongly positive scatter of the points around the least squares line through the origin with slope bl = 6.2880 is much smaller than is the scatter around the horizontal line e( YIX2) =0 indicating that adding XI to the regression model with a linear relation will substantial reduce the error sum of squares 7
- 8. Residuals- identifying cases These outlying cases may involve large residuals and often have dramatic effects on the fitted least squares regression function. A case may be outlying or extreme with respect to its Y value, its X value(s), or both. Case 1 and 2 in may not be too influential because a number of other cases have similar X or Y values that will keep the fitted regression function from being displaced too far by the outlying case Cases 3 and 4, on the other hand, are likely to be very influential in affecting the fit of the regression function. 8 outlying with respect to its Y value outlying with respect to their X values cases 3 and 4 are also outlying with respect to their Y values, given X.
- 9. Identifying Outlying Y Observations some cases that are outlying or extreme These outlying cases may involve large residuals and often have dramatic effects on the fitted least squares regression function When more than two predictor variables are included in the regression model, however, the identification of outlying cases by simple graphic means becomes difficult Some univariate outliers may not be extreme in a multivariate regression model, and, conversely We introduce now two refinements to make the analysis of residuals more effective for identifying outlying observations 9
- 10. Outlying Y-Use residuals & hat matrix Analysis the residuals: Hat matrix the fitted value and for residuals we have e = (I - H)Y variance-covariance matrix of the residuals These variances and covariances are estimated by using MSE as the estimator of the error variance Estimated: Variance of residuals ei >>>> Covariance between residuals ei ej >> 10 ith element on the main diagonal of the hat matrix
- 11. Example-residual with hat matrix N = 4 and two predictor variable Fitted model: s2{e} = 574.9( I - .3877) = 352.0 We see from last Table, column7 (s2{ ei }), that the residuals do not have constant variance and residuals for cases are outlying with Respect to the x variable have smaller variance. The estimated variance- covariance matrix of the residuals, s"{e} = MSE(I - H) 11 Fitted values residuals Diagonal element of hat
- 12. Deleted Residuals- identifying outlying Y The second refinement to make residuals more effective for detecting outlying Y observations is to measure the ith residual ei = Yi - Yi when the fitted regression is based on all of the cases except the ith one. The procedure then is to delete the ith case, fit the regression function to the remaining n - 1 cases, and obtain the point estimate of the expected value when the X levels are those of the ith case, to be denoted by Yi(i) Deleted residual for the ith case >> Thus, deleted residuals will at times identify outlying Y observations when ordinary residuals would not identify these. We identify as outlying Y observations those cases whose studentized deleted residuals are large in absolute value. In addition, we can conduct a formal test by means of the Bonferroni test procedure of whether the case with the largest absolute studentized deleted residual is an outlier. 12
- 13. Example-deleted residuals we wish to examine whether there are outlying Y observations for example:XII = 19.5 and XI2=43.1 studentized deleted residual Test for case 13 13
- 14. Example deleated residuals We would like to test whether case 13, which has the largest absolute studentized deleted residual, case 13 is an outlier resulting from a change in the model? use the Bonferroni simultaneous test procedure with a family significance level of a = .10 few other outlying cases are influential in determining the fitted regression function because the Bonferroni procedure provides a very conservative test for the presence of an outlier. 14
- 15. Identifying Outlying X Observations-hat matrix , Leverage Values The hat matrix also is helpful in directly identifying outlying X observations The diagonal elements hii of the hat matrix have some useful properties: hii is a measure of the distance between the X values for the i th case and the means of the X values for all cases. Thus, a large value hii indicates that the ith case is distant from the center of all X observations The diagonal element hii in this context is called the leverage of the ith case. If the ith case is outlying in terms of its X observations and therefore has a large leverage value hii. 15 greater than 2mean hii=2p/n exceeding .5 existence of a gap between the leverage values
- 16. Example -x outlying body fat example with two predictor variables-triceps skinfold thickness (X1) and thigh circumference (X2) Note that the two largest leverage values are h33 = .372 and h15.15 = .333.Both exceed the criterion of twice the mean leverage value,2p/n = 2(3)/20 = .30 both are separated by a substantial gap from the next largest leverage values, h55 = .248 and h11 = .201 Case 15 is outlying for X I Case 3 is outlying in terms of the pattern of multicollinearity 16
- 17. Identifying Influential Cases-DFFITS, Cook’s Distance- and DFBETAS Measures After identifying cases that are outlying with respect to their Y values and/or their X values, the next step is to ascertain whether or not these outlying cases are influential. We shall consider a case to be influential if its exclusion causes major changes in the fitted regression function. We take up three measures of influence that are widely used in practice, each based on the omission of a single case to measure its influence. I. Influence on Single Fitted Value-DFFlTS II. Influence on All Fitted Values-Cook's Distance III. Influence on the Regression Coefficients-DFBETAS 17
- 18. Influence on Single Fitted Value- DFFlTS measure of the influence that case i OR the value (DFFITS)for the ith case represents the fitted value Yi increases or decreases with the inclusion of the ith case in fitting the regression model. we suggest considering a case influential if the absolute value of DFFITS exceeds 1 for small to medium data sets and 𝟐/ 𝒑/𝒏 In for large data sets. 18
- 19. Example-DFFITS value Body fat example: consider the DFFITS value for case 3, which was identified as outlying with respect to its X values This value is somewhat larger than our guideline of 1. However, the value is close enough to 1 that the case may not be influential enough to require remedial action. 19
- 20. Influence on All Fitted Values- Cook's Distance Cook's distance measure considers the influence of the ith case on all n fitted values. Cook's distance measure, denoted by Di, is an aggregate influence measure In matrix term >> relate Di to the F(p, n - p) distribution and ascertain the corresponding percentile value If the percentile value is less than about 10 or 20 percent, the i th case has little apparent influence on the fitted values. If, on the other hand, the percentile value is near 50 percent or more, the fitted values obtained with and without the I th case should be considered to differ substantially, implying that the i th case has a major influence on the fit of the regression function. 20 fitted values when the i th case is deleted
- 21. Example Body fat example two predictor variable: we consider again case 3, which is outlying with regard to its X values p = 3 for the model with two predictor variables case 3 clearly has the largest Di value, with the next largest distance measure Dl3 = .212 being substantially smaller. To assess the magnitude of the influence of case 3 (D3 = .490), we refer to the corresponding F distribution, namely, F(p, 17 - p) = F(3, 17). 21
- 22. Example- Cook's distance Figures: clearly show that one case stands out as most influential (case 3) and that all the other cases are much less influential the size of the plotted points being proportional to Cook's distance measure Di identifies the most influential case as case 3 but does not provide any information about the magnitude of the residual for this case assess the magnitude of the influence of case 3 (D3 = .490) F(p, n - p) = F(3, 17) so We find that .490 is the 30.6th percentiles of this distribution. Hence, it appears that case 3 does influence the regression fit, but the extent of the influence may not be large enough to call for consideration of remedial measures. 22 residual for the most influential case is large negative
- 23. Influence on the Regression Coefficients- DFBETAS measure of the influence of the i th case on each regression coefficient bk is the difference between the estimated regression coefficient bk based on all n cases and the regression coefficient obtained when the ith case is omitted. variance of bk is: 2{bk}= 2ckk kth diagonal element of (X'X)-1 regression coefficient obtained when the ith case is omitted error mean square obtained when the ith case is deleted in fitting the regression model 23
- 24. DFBETAS The DFBETAS value by; Sign: indicates whether inclusion of a case leads to an increase or a decrease in the estimated regression coefficient absolute magnitude : shows the size of the difference relative to the estimated standard deviation of the regression coefficient A large absolute value of (DFBETAS)k(i) is indicative of a large impact on the ith case on the kth regression coefficient And we recommend considering a case influential if the absolute value of DFBETAS exceeds 1 for small to medium data sets and 𝟐/ 𝒏 for large data sets We explain this with next example 24
- 25. Example –DFBETAS value Body fat example two predictor variable only case that exceeds our guideline of 1 for medium-size data sets for both b1 and b2 Thus, case 3 is again tagged as potentially influentiaL Again, however, the DFBETAS values do not exceed 1 by very much so that case 3 may not be so influential as to require remedial action. 25
- 26. Multicollinearity Diagnostics- informal diagnostics Indications of the presence of serious multicollinearity are given by the following informal diagnostics: 1. Large changes in the estimated regression coefficients when a predictor variable is added or deleted, or when an observation is altered or deleted 2. Non significant results in individual tests on the regression coefficients for important predictor variables. 3. Estimated regression coefficients with an algebraic sign that is the opposite of that expected from theoretical considerations or prior experience. 4. Large coefficients of simple correlation between pairs of predictor variables in the correlation matrix rxx. 5. Wide confidence intervals for the regression on coefficients representing important predictor variables. 26
- 27. Example- Multicollinearity informal diagnosis three predictor variables: skinfold thickness (X1), thigh circumference (X2), and midarm circumference (X3) 1. predictor variables triceps skinfold thickness and thigh circumference are highly correlated with each other. 2. We also noted large changes in the estimated regression coefficients and their estimated standard deviations when a variable was added 3. Non significant results in individual tests on anticipated important variables 4. estimated negative coefficient when a positive coefficient was expected. These are suggest serious multicollinearity among the predictor variables 27
- 28. Multicollinearity Diagnostics- Variance inflation Factor A formal method of detecting the presence of multicollinearity that is widely accepted is use of variance inflation factors. These factors measure how much the variances of the estimated regression coefficients are inflated as compared to when the predictor variables are not linearly related. variance-covariance matrix of the estimated regression coefficients is: 28
- 29. Multicollinearity Diagnostics- Variance inflation Factor standardized regression model: (𝜎∗)2 is the error term variance for the transformed model variance inflation factor for bk (VIF)k denote the kth diagonal element of the matrix 𝑟𝑥𝑥 −1 transforming the variables by means of the correlation transformation (VIF)k =1 then 𝑅 𝑘 2 =0 𝑋 𝑘is not linearly related to the other X variables 𝑅 𝑘 2 is the coefficient of multiple determination when Xk is regressed on the p - 2 other X variables in the model 29
- 30. Multicollinearity Diagnostics- Variance inflation Factor The largest VIF value among all X variables is often used as an indicator of the severity of multicollinearity The mean of the VIF values also provides information about the severity of the multicollinearity in terms of how far the estimated standardized regression coefficients bk are from the true values Bk. sum of the squared errors: & effect of multicollinearity on the sum of the squared errors:(mean of the VIF values) 30 no X variable is linearly related to the others in the regression model If greater then 1 >>>> serious multicollinearity
- 31. Example -Variance inflation Factor body fat example with three predictor variables Mean VIF values considerably larger than 1 are indicative of serious multicollinearity problems. all three VIF values greatly exceed 10, which again indicates that serious multicollinearity problems exist. Thus, the expected sum of the squared errors in the least squares standardized regression coefficients is nearly 460 times as large as it would be if the X variables were uncorrelated 31
- 32. Summary in model building Building the regression model: Model selection 𝑹 𝒂,𝑷 𝟐 𝑹 𝑷 𝟐 𝑨𝑰𝑪 𝒑 𝑪 𝒑 𝑷𝑹𝑬𝑺𝑺 𝒑 𝑺𝑩𝑪 𝒑 𝑺𝑺𝑬 𝒑 Stepwise Methods Model validation Collection of new data & Comparison with earlier empirical results & Data Splitting diagnostics Outliers influential case multicollinearity interaction effect Remedial measures 32
- 33. Example – surgical unit Model selected : lny = 𝑋1 + 𝑋2 + 𝑋3 + 𝑋8 Examine the interaction effect with added variable plots for 6 two factor interaction: To examine interaction effects further, a regression model containing first- order terms in XI, X2, X3, and X8 was fitted and added-variable plots for the six two-factor interaction terms. these plots did not suggest that any strong two-variable interactions are present and need to be included in the model. The residual plots shows no evidence of serious departures from the model. use a residual plot and an added-variable plot to study graphically the strength of the marginal relationship between X5 and the response when X1, X2, X3, and X8 are already in the model. 33
- 34. Example- pg413 Multicollinearity was studied by calculating the variance inflation factors . Multicolinarity among 4 predictor not problem(all >1) plots of four key regression diagnostics: 1.deleted studentized residuals 2.the leverage values 3.Cook's Distances 4. DFFlTS values Case 17 was identified as outlying with regard to its Yvalue according to its studentized deleted residuals. outlying by more than three standard deviations. Bonferroni test >> not an outlier identifying outlying X observations, cases 23, 28, 32, 38, 42, and 52 were identified as outlying according to their leverage values with guide 2p/n=0.185 case 17 is the most influential, with Cook's distance D17=.3306 Referring to the F distribution with 5 and 49 degrees of freedom, we note that the Cook's value corresponds to the 11th percentile(bitween 10 to 30).>>>> It thus appears that the influence of case 17 is not large enough to warrant remedial measures, 34
- 35. 35