data Sreening.doc
- 1. 1 Data Screening (Missing Values, Outliers, Normality etc.) The purpose of data screening is to: (a) check if data have been entered correctly, such as out-of-range values. (b) check for missing values, and deciding how to deal with the missing values. (c) check for outliers, and deciding how to deal with outliers. (d) check for normality, and deciding how to deal with non-normality. 1. Finding incorrectly entered data Your first step with “Data Screening” is using “Frequencies” 1. Select Analyze --> Descriptive Statistics --> Frequencies 2. Move all variables into the “Variable(s)” window. 3. Click OK. Output below is for only the four “system” variables in our dataset because copy/pasting the output for all variables in our dataset would take up too much space in this document. The “Statistics” box tells you the number of missing values for each variable. We will use this information later when we are discussing missing values. Each variable is then presented as a frequency table. For example, below we see the output for “system1”. By looking at the coding manual for the “Legal beliefs” survey, you can see that the available responses for “system1” are 1 through 11. By looking at the output below, you can see that there is a number out-of-range: “13”. (NOTE – in your dataset there will not be a “13” because I gave you the screened dataset, so I have included the “13” into this example to show you what it looks like when a number is out of range.) Since 13 is an invalid number, you then need to identify why “13” was entered. For example, did the person entering data make a mistake? Or, did the subject respond with a “13” even though the question indicated that only numbers 1 through 11 are valid? You can identify the source of the error by looking at the hard copies of the data. For example, first identify which subject indicated the “13” by clicking on the variable name to highlight it (system1), and then using the “find” function by: Edit --> Find, and then scrolling to the left to identify the subject number. Then, hunt down the hard copy of the data for that subject number.
- 2. 2 2. Missing Values Why do missing values occur? Missing values are either random or non-random. Random missing values may occur because the subject inadvertently did not answer some questions. For example, the study may be overly complex and/or long, or the subject may be tired and/or not paying attention, and miss the question. Random missing values may also occur through data entry mistakes. Non-random missing values may occur because the subject purposefully did not answer some questions. For example, the question may be confusing, so many subjects do not answer the question. Also, the question may not provide appropriate answer choices, such as “no opinion” or “not applicable”, so the subject chooses not to answer the question. Also, subjects may be reluctant to answer some questions because of social desirability concerns about the content of the question, such as questions about sensitive topics like past crimes, sexual history, prejudice or bias toward certain groups, and etc. Why is missing data a problem? Missing values means reduced sample size and loss of data. You conduct research to measure empirical reality so missing values thwart the purpose of research. Missing values may also indicate bias in the data. If the missing values are non-random, then the study is not accurately measuring the intended constructs. The results of your study may have been different if the missing data was not missing. How do I identify missing values? 1. Select Analyze --> Descriptive Statistics --> Frequencies 2. Move all variables into the “Variable(s)” window. 3. Click OK. Output below is for only the four “system” variables in our dataset because copy/pasting the output for all variables in our dataset would take up too much space in this document. The “Statistics” box tells you the number of missing values for each variable. How do I deal with missing values? Irrespective of whether the missing values are random or non-random, you have three options when dealing with missing values. Option 1 is to do nothing. Leave the data as is, with the missing values in place. This is the most frequent approach, for a few reasons. First, missing values are typically small. Second, missing values are typically non-random. Third, even if there are a few missing values on individual items, you typically create composites of the items by averaging them together into one new variable, and this composite variable will not have missing values because it is an average of the existing data. However, if you chose this option, you must keep in mind how SPSS will treat the missing values. SPSS will either use “listwise deletion” or “pairwise deletion” of the missing values. You can elect either one when conducting each test in SPSS. a. Listwise deletion – SPSS will not include cases (subjects) that have missing values on the variable(s) under analysis. If you are only analyzing one variable, then listwise deletion is simply analyzing the existing data. If you are analyzing multiple variables, then listwise deletion removes cases (subjects) if there is a missing value on any of the variables. The disadvantage is a loss of data because you are removing all data from subjects who may have answered some of the questions, but not others (e.g., the missing data). b. Pairwise deletion – SPSS will include all available data. Unlike listwise deletion which removes cases (subjects) that have missing values on any of the variables under analysis, pairwise deletion only removes the specific missing values from the analysis (not the entire case). In other words, all available data is included. For example, if you are conducting a correlation on multiple variables, then SPSS will conduct the bivariate correlation between all available data point, and ignore only those missing values if they exist on some variables. In this case, pairwise deletion will result in different sample sizes for each correlation. Pairwise deletion is useful when sample size is small or missing values are large because there are not many values to begin with, so why omit even more with listwise deletion.
- 3. 3 c. In other to better understand how listwise deletion versus pairwise deletion influences your results, try conducting the same test using both deletion methods. Does the outcome change? . Option 2 is to delete cases with missing values. For example, for every missing value in the dataset, you can delete the subjects with the missing values. Thus, you are left with complete data for all subjects. The disadvantage to this approach is you reduce the sample size of your data. If you have a large dataset, then it may not be a big disadvantage because you have enough subjects even after you delete the cases with missing values. Another disadvantage to this approach is that the subjects with missing values may be different than the subjects without missing values (e.g., missing values that are non-random), so you have a non- representative sample after removing the cases with missing values. Once situation in which I use Option 2 is when particular subjects have not answered an entire scale or page of the study. Option 3 is to replace the missing values, called imputation. There is little agreement about whether or not to conduct imputation. There is some agreement, however, in which type of imputation to conduct. For example, you typically do NOT conduct Mean substitution or Regression substitution. Mean substitution is replacing the missing value with the mean of the variable. Regression substitution uses regression analysis to replace the missing value. Regression analysis is designed to predict one variable based upon another variable, so it can be used to predict the missing value based upon the subject’s answer to another variable. Both Mean substitution and Regression substitution can be found using: Transform --> Replace Missing Cases. The favored type of imputation is replacing the missing values using different estimation methods. The “Missing Values Analysis” add-on contains the estimation methods, but versions of SPSS without the add-on module do not. The estimation methods be found by using: Transform --> Replace Missing Cases. 3. Outliers – What are outliers? Outliers are extreme values as compared to the rest of the data. The determination of values as “outliers” is subjective. While there are a few benchmarks for determining whether a value is an “outlier”, those benchmarks are arbitrarily chosen, similar to how “p<.05” is also arbitrarily chosen. Should I check for outliers? Outliers can render your data non-normal. Since normality is one of the assumptions for many of the statistical tests you will conduct, finding and eliminating the influence of outliers may render your data normal, and thus render your data appropriate for analysis using those statistical tests. However, I know no one who checks for outliers. For example, just because a value is extreme compared to the rest of the data does not necessarily mean it is somehow an anomaly, or invalid, or should be removed. The subject chose to respond with that value, so removing that value is arbitrarily throwing away data simply because it does not fit this “assumption” that data should be “normal”. Conducting research is about discovering empirical reality. If the subject chose to respond with that value, then that data is a reflection of reality, so removing the “outlier” is the antithesis of why you conduct research. There is one more (less theoretical, and more practical) reason why I know no one who conducts outlier analysis. It is common practice to use multiple questions to measure constructs because it increases the power of your statistical analysis. You typically create a “composite” score (average of all the questions) when analyzing your data. For example, in a study about happiness, you may use an established happiness scale, or create your own happiness questions that measure all the facets of the happiness construct. When analyzing your data, you average together all the happiness questions into 1 happiness composite measure. While there may be some outliers in each individual question, averaged the items together reduces the probability of outliers due to the increased amount of data composited into the variable. Checking outliers: 1. Select Analyze --> Descriptive Statistics --> Explore 2. Move all variables into the “Variable(s)” window. 3. Click “Statistics”, and click “Outliers” 4. Click “Plots”, and unclick “Stem-and-leaf” 5. Click OK. Output on next page is for “system1”
- 4. 4 “Descriptives” box tells you descriptive statistics about the variable, including the value of Skewness and Kurtosis, with accompanying standard error for each. This information will be useful later when we talk about “normality”. The “5% Trimmed Mean” indicates the mean value after removing the top and bottom 5% of scores. By comparing this “5% Trimmed Mean” to the “mean”, you can identify if extreme scores (such as outliers that would be removed when trimming the top and bottom 5%) are having an influence on the variable. “Extreme Values” and the Boxplot relate to each other. The boxplot is a graphical display of the data that shows: (1) median, which is the middle black line, (2) middle 50% of scores, which is the shaded region, (3) top and bottom 25% of scores, which are the lines extending out of the shaded region, (4) the smallest and largest (non-outlier) scores, which are the horizontal lines at the top/bottom of the boxplot, and (5) outliers. The boxplot shows both “mild” outliers and “extreme” outliers. Mild outliers are any score more than 1.5*IQR from the rest of the scores, and are indicated by open dots. IQR stands for “Interquartile range”, and is the middle 50% of the scores. Extreme outliers are any score more than 3*IQR from the rest of the scores, and are indicated by stars. However, keep in mind that these benchmarks are arbitrarily chosen, similar to how p<.05 is arbitrarily chosen. For “system1”, there is an open dot. Notice that the dot says “42”, but, by looking at “Extreme Values box”, there are actually FOUR lowest scores of “1”, one of which is case 42. Since all four scores of “1” overlap each other, the boxplot can only display one case. In summary, this output tells us there are four outliers, each with a value of “1”.
- 5. 5 4. Outliers Another way to look for univariate outliers is to do outlier analysis within different groups in your study. For example, imagine a study that manipulated the presence or absence of a weapon during a crime, and the Dependent Variable was measuring the level of emotional reaction to the crime. In addition to looking for univariate outliers for your DV, you may want to also look for univariate outliers within each condition. In our dataset about “Legal Beliefs”, let’s treat gender as the grouping variable. 1. Select Analyze --> Descriptive Statistics --> Explore 2. Move all variables into the “Variable(s)” window. Move “sex” into the “Factor List” 3. Click “Statistics”, and click “Outliers” 4. Click “Plots”, and unclick “Stem-and-leaf” 5. Click OK. Output below is for “system1” “Descriptives” box tells you descriptive statistics about the variable. Notice that information for “males” and “females” is displayed separately. “Extreme Values” and the Boxplot relate to each other. Notice the difference between males and females.
- 6. 6 5. Outliers – dealing with outliers First, we need to identify why the outlier(s) exist. It is possible the outlier is due to a data entry mistake, so you should first conduct the test described above as “1. Finding incorrectly entered data” to ensure that any outlier you find is not due to data entry errors. It is also possible that the subjects responded with the “outlier” value for a reason. For example, maybe the question is poorly worded or constructed. Or, maybe the question is adequately constructed but the subjects who responded with the outlier values are different than the subjects who did not respond with the extreme scores. You can create a new variable that categorizes all the subjects as either “outlier subjects” or “non-outlier subjects”, and then re-examine the data to see if there is a difference between these two types of subjects. Also, you may find the same subjects are responsible for outliers in many questions in the survey by looking at the subject numbers for the outliers displayed in all the boxplots. Remember, however, that just because a value is extreme compared to the rest of the data does not necessarily mean it is somehow an anomaly, or invalid, or should be removed. Second, if you want to reduce the influence of the outliers, you have four options. Option 1 is to delete the value. If you have only a few outliers, you may simply delete those values, so they become blank or missing values. Option 2 is to delete the variable. If you feel the question was poorly constructed, or if there are too many outliers in that variable, or if you do not need that variable, you can simply delete the variable. Also, if transforming the value or variable (e.g., Options #3 and #4) does not eliminate the problem, you may want to simply delete the variable. Option 3 is to transform the value. You have a few options for transforming the value. You can change the value to the next highest/lowest (non-outlier) number. For example, if you have a 100 point scale, and you have two outliers (95 and 96), and the next highest (non-outlier) number is 89, then you could simply change the 95 and 96 to 89s. Alternatively, if the two outliers were 5 and 6, and the next lowest (non-outlier) number was 11, then the 5 and 6 would change to 11s. Another option is to change the value to the next highest/lowest (non-outlier) number PLUS one unit increment higher/lower. For example, the 95 and 96 numbers would change to 90s (e.g., 89 plus 1 unit higher). The 5 and 6 numbers change to 10s (e.g., 11 minus 1 unit lower). Option 4 is to transform the variable. Instead of changing the individual outliers (as in Option #3), we are now talking about transforming the entire variable. Transformation creates normal distributions, as described in the next section below about “Normality”. Since outliers are one cause of non-normality, see the next section to learn how to transform variables, and thus reduce the influence of outliers. Third, after dealing with the outlier, you re-run the outlier analysis to determine if any new outliers emerge or if the data are outlier free. If new outliers emerge, and you want to reduce the influence of the outliers, you choose one the four options again. Then, re-run the outlier analysis to determine if any new outliers emerge or if the data are outlier free, and repeat again. 7. Normality Below, I describe five steps for determining and dealing with normality. However, the bottom line is that almost no one checks their data for normality; instead they assume normality, and use the statistical tests that are based upon assumptions of normality that have more power (ability to find significant results in the data). First, what is normality? A normal distribution is a symmetric bell-shaped curve defined by two things: the mean (average) and variance (variability). Second, why is normality important? The central idea behind statistical inference is that as sample size increases, distributions will approximate normal. Most statistical tests rely upon the assumption that your data is “normal”. Tests that rely upon the assumption or normality are called parametric tests. If your data is not normal, then you would use statistical tests that do not rely upon the assumption of normality, call non- parametric tests. Non-parametric tests are less powerful than parametric tests, which means the non-parametric tests have less ability to detect real differences or variability in your data. In other words, you want to conduct parametric tests because you want to increase your chances of finding significant results.
- 7. 7 Third, how do you determine whether data are “normal”? There are three interrelated approaches to determine normality, and all three should be conducted. First, look at a histogram with the normal curve superimposed. A histogram provides useful graphical representation of the data. SPSS can also superimpose the theoretical “normal” distribution onto the histogram of your data so that you can compare your data to the normal curve. To obtain a histogram with the superimposed normal curve: 1. Select Analyze --> Descriptive Statistics --> Frequencies. 2. Move all variables into the “Variable(s)” window. 3. Click “Charts”, and click “Histogram, with normal curve”. 4. Click OK. Output below is for “system1”. Notice the bell-shaped black line superimposed on the distribution. All samples deviate somewhat from normal, so the question is how much deviation from the black line indicates “non-normality”? Unfortunately, graphical representations like histogram provide no hard-and-fast rules. After you have viewed many (many!) histograms, over time you will get a sense for the normality of data. In my view, the histogram for “system1” shows a fairly normal distribution. Second, look at the values of Skewness and Kurtosis. Skewness involves the symmetry of the distribution. Skewness that is normal involves a perfectly symmetric distribution. A positively skewed distribution has scores clustered to the left, with the tail extending to the right. A negatively skewed distribution has scores clustered to the right, with the tail extending to the left. Kurtosis involves the peakedness of the distribution. Kurtosis that is normal involves a distribution that is bell-shaped and not too peaked or flat. Positive kurtosis is indicated by a peak. Negative kurtosis is indicated by a flat distribution. Descriptive statistics about skewness and kurtosis can be found by using either the Frequencies, Descriptives, or Explore commands. I like to use the “Explore” command because it provides other useful information about normality, so
- 8. 8 1. Select Analyze --> Descriptive Statistics --> Explore. 2. Move all variables into the “Variable(s)” window. 3. Click “Plots”, and unclick “Stem-and-leaf” 4. Click OK. Descriptives box tells you descriptive statistics about the variable, including the value of Skewness and Kurtosis, with accompanying standard error for each. Both Skewness and Kurtosis are 0 in a normal distribution, so the farther away from 0, the more non-normal the distribution. The question is “how much” skew or kurtosis render the data non-normal? This is an arbitrary determination, and sometimes difficult to interpret using the values of Skewness and Kurtosis. Luckily, there are more objective tests of normality, described next. Third, the descriptive statistics for Skewness and Kurtosis are not as informative as established tests for normality that take into account both Skewness and Kurtosis simultaneously. The Kolmogorov-Smirnov test (K-S) and Shapiro-Wilk (S-W) test are designed to test normality by comparing your data to a normal distribution with the same mean and standard deviation of your sample: 1. Select Analyze --> Descriptive Statistics --> Explore. 2. Move all variables into the “Variable(s)” window. 3. Click “Plots”, and unclick “Stem-and-leaf”, and click “Normality plots with tests”. 4. Click OK. “Test of Normality” box gives the K-S and S-W test results. If the test is NOT significant, then the data are normal, so any value above .05 indicates normality. If the test is significant (less than .05), then the data are non-normal. In this case, both tests indicate the data are non-normal. However, one limitation of the normality tests is that the larger the sample size, the more likely to get significant results. Thus, you may get significant results with only slight deviations from normality. In this case, our sample size is large (n=327) so the significance of the K-S and S-W tests may only indicate slight deviations from normality. You need to eyeball your data (using histograms) to determine for yourself if the data rise to the level of non-normal. “Normal Q-Q Plot” provides a graphical way to determine the level of normality. The black line indicates the values your sample should adhere to if the distribution was normal. The dots are your actual data. If the dots fall exactly on the black line, then your data are normal. If they deviate from the black line, your data are non- normal. In this case, you can see substantial deviation from the straight black line.
- 9. 9 Fourth, if your data are non-normal, what are your options to deal with non-normality? You have four basic options. a. Option 1 is to leave your data non-normal, and conduct the parametric tests that rely upon the assumptions of normality. Just because your data are non-normal, does not instantly invalidate the parametric tests. Normality (versus non-normality) is a matter of degrees, not a strict cut-off point. Slight deviations from normality may render the parametric tests only slightly inaccurate. The issue is the degree to which the data are non-normal. b. Option 2 is to leave your data non-normal, and conduct the non-parametric tests designed for non- normal data. c. Option 4 is to transform the data. Transforming your data involving using mathematical formulas to modify the data into normality.