Chapter 13
ANCOVA
As Judd and McClelland note, the material covered in this chapter represents an integration of the materials covered in Chapters 8 and 12. In Chapter 8 we finished what would have been traditionally called regression, where the predictors are continuous. In Chapter 12 we covered what would have been traditionally called ANOVA, where the predictors were nominal. In this Chapter we discuss models that have both continuous and nominal predictors.
History
This material originally developed out of the experimental tradition in research. In the experimental tradition researchers were interested in detecting differences between groups. However, sometimes there were continuous variables that the researcher wished to control before investigating whether differences in the dependent variable could still be accounted for by the categorical variable.
An example might clarify this idea. Suppose you are a researcher interested in determining if there are significant differences in reading scores produced by three different reading programs. You randomly assign 150 children to the three reading groups. Before you start the reading treatments for the three groups, you also gave each child an individual test of intelligence. You know that there is a strong relationship between intelligence and reading achievement. After the reading treatments are concluded, you measure the dependent variable (reading achievement). Your question is: Is there a significant difference in reading achievement across the three groups, controlling for intelligence?
In the experimental tradition the nominal variable (reading group) would be called the independent variable, the continuous variable (intelligence) would be called the covariate, and reading scores would be the dependent variable.
Using the Model Comparison procedure both the nominal and continuous variables can simply be thought of as predictors. Their ability to predict reading scores can be evaluated individually, or after other variables are included. Covariates can be either nominal or continuous variables. By multiplying variables together we can also investigate interactions.
Using this example, we could just as easily ask if the relationship between intelligence and reading was still significant after controlling for the reading treatment.
One of the major advantages of using ANCOVA techniques is to increase the power of the statistical analysis. This is the first use we will investigate.
Increasing Power
In Judd and McClelland they use an example (Exhibit 13.1) where students are randomly assigned to either a new or old curriculum, and are taught by two different teachers. These are the nominal categorical variables. The dependent variable in this study is the student's score on an achievement test given at the end of the curriculum. In addition the covariate (continuous variable) is a pretest that measures the students knowledge prior to exposure to either the curriculum or the teachers.
In the data set the variable Y is the dependent variable, variables X1 and X2 are are the effect codes for the two categorical variables, x3 is the interaction code, and Z is the score on the pretest.
The pretest scores have been created so that each each of the four groups' means are equal to exactly 50. This is done for a didactic reason. If each of the groups means are equal to 50 then the pretest is not correlated with the group conditions. However, the pretest and the dependent variable are highly correlated with one another.
Typically, we would not have a covariate and grouping variables that are uncorrelated. However, if we give the pretest (covariate) and then assign subjects randomly to groups, we would only expect a correlation different from zero by chance fluctuations due to the random assignment. If the covariate is related to the dependent variable, under these conditions ANCOVA increases the power of the analysis. The rest of this section illustrates how this happens.
Summary Statistics
------------------
Variable Count Mean Median Std. deviation
---------------------------------------------------------------------------
Y1 40 58.75 58.0 5.5873
Z 40 50.0 50.0 2.40725
---------------------------------------------------------------------------
Variable Minimum Maximum Range Std. skewness
---------------------------------------------------------------------------
Y1 47.0 71.0 24.0 0.2604
Z 46.0 54.0 8.0 -0.26977
---------------------------------------------------------------------------
Variable Std. kurtosis
---------------------------------------------------------------------------
Y1 -0.607877
Z -1.23932
---------------------------------------------------------------------------
Notice in the Summary using Statlets above, that the Mean of Z is equal to 50.
If we conduct an ANOVA using the two effect codes we get the following:
Dependent variable: Y1
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT 58.75 0.702179 83.67 1.0E-4
X1 3.25 0.702179 4.63 1.0E-4
X2 0.75 0.702179 1.07 0.2926
x3 1.25 0.702179 1.78 0.0835
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 507.5 3.0 169.167 8.58 1.0E-4
Residual 710.0 36.0 19.7222
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 41.6838 percent
R-squared (adjusted for d.f.) = 36.8241 percent
Standard error of est. = 4.44097
Coeff. of variation = 7.5591 percent
Mean absolute error = 3.55
Durbin-Watson statistic = 2.82676
Notice the regression equation is the same as that reported on page 366. Also notice that the SSE or SSResidual is the same as reported on 366.
Now looking at the t-statistics, explain how you would interpret this output to your reader -- or your dissertation committee.
We could use the TtoF conversion program to change these t values to appropriate F statistics, and create the table on page 367 in Judd and McClelland.
Quoting directly from Judd and McClelland on page 366-367:
In this analysis, we have made no use of the pretest Z. We might decide to add it as a predictor to the model for a number of different reasons. For instance, we might be interested in asking how curriculum and teacher affect posttest performance when we control for the pretest or when we look within levels of the pretest. In other words, we might be interested in these effects over and above the differences in performance that existed at the time of the pretest. A seemingly different reason for including it as a predictor in the model is that we might expect it to be highly correlated with the posttest, since presumably it is only an earlier version of the posttest, measuring the same domain of achievement. If this is so, then, as we shall see, it might make our tests of curriculum and teacher effects considerably more powerful.
Remember that the pretest score is unrelated to condition. Check this out if you don't believe it is so.
Now let's regress Y on X1, X2, X3, and Z. Here are the results from Statlets.
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT -4.52434 10.5004 -0.43 0.6692
X1 3.25 0.49862 6.52 1.0E-4
X2 0.75 0.49862 1.50 0.1415
x3 1.25 0.49862 2.51 0.0170
Z 1.26549 0.209771 6.03 1.0E-4
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 869.429 4.0 217.357 21.86 1.0E-4
Residual 348.071 35.0 9.94488
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 71.411 percent
R-squared (adjusted for d.f.) = 68.1437 percent
Standard error of est. = 3.15355
Coeff. of variation = 5.36775 percent
Mean absolute error = 2.46327
Durbin-Watson statistic = 2.09058
If I wanted to compare the augmented model above with a model that only contained the covariate, I would either need to estimate the compact model, or since the predictors are uncorrelated, I could do some simple math with the TtoF conversion program. However, below I have estimated the compact model with Statlets.
Multiple Regression Analysis
---------------------------------------------------------------------------
Dependent variable: Y1
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT -4.52434 15.7995 -0.29 0.7762
Z 1.26549 0.315633 4.01 3.0E-4
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 361.929 1.0 361.929 16.08 1.0E-4
Residual 855.571 38.0 22.515
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 29.7272 percent
R-squared (adjusted for d.f.) = 27.878 percent
Standard error of est. = 4.745
Coeff. of variation = 8.0766 percent
Mean absolute error = 3.75077
Durbin-Watson statistic = 1.24274
Notice that the SSResidual is 855.571 in the compact model and 348.071 in the augmented model. Take one min. and calculate the PRE and F values associated with this comparison. Where did the 507.5 SSE go? Have you seen it in any other output?
If you look at the SSResidual for the regular ANOVA (our first output on this page) you notice that it is 710. If you look at the SSResidual for the full model it is 348.071. The difference here is 361.929. Have you seen that value in any of the outputs?
Write a simple statement explaining how ANCOVA increases power under these circumstances.
Summary Statement
Judd and McClelland page 371
Even with random assignment of subjects to condition after measuring the covariate, it will almost never be the case that the covariate will be completely orthogonal to the condition contrast codes. In other words, it will be a very rare event for all of the pretest or covariate means in the various experimental conditions to be identical. Our example, then, is obviously a constructed one, designed simply to illustrate what happens in the pure case, when the covariate is completely independent of condition. In any given study, there will in all probability be some nonsignificant relationships between the covariate and the contrast codes that represent condition. Nevertheless, the inclusion of a covariate will increase the statistical power of tests of condition differences, given a covariate that is reliably related to the dependent variable within levels of the categorical variable.
Turning our first interpretation around
We could also concentrate on the test of the pretest-posttest relationship while controlling for differences in condition.
If we simply regress the posttest score on the pretest we get the following:
Multiple Regression Analysis
---------------------------------------------------------------------------
Dependent variable: Y1
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT -4.52434 15.7995 -0.29 0.7762
Z 1.26549 0.315633 4.01 3.0E-4
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 361.929 1.0 361.929 16.08 1.0E-4
Residual 855.571 38.0 22.515
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 29.7272 percent
R-squared (adjusted for d.f.) = 27.878 percent
Standard error of est. = 4.745
Coeff. of variation = 8.0766 percent
Mean absolute error = 3.75077
Durbin-Watson statistic = 1.24274
Notice that the regression coefficient is approximately 1.27 and the F statistic is approximately 16 with a PRE equal to .297.
If we examined the pretest-posttest relationship after controlling for the categorical conditions, we find that the regression coefficient is still the same (approximately 1.27), but the F and PRE have changed dramatically. Use the output below and the TtoF conversion program to determine these values. Check your answers with those given on page 371.
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT -4.52434 10.5004 -0.43 0.6692
X1 3.25 0.49862 6.52 1.0E-4
X2 0.75 0.49862 1.50 0.1415
x3 1.25 0.49862 2.51 0.0170
Z 1.26549 0.209771 6.03 1.0E-4
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 869.429 4.0 217.357 21.86 1.0E-4
Residual 348.071 35.0 9.94488
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 71.411 percent
R-squared (adjusted for d.f.) = 68.1437 percent
Standard error of est. = 3.15355
Coeff. of variation = 5.36775 percent
Mean absolute error = 2.46327
Durbin-Watson statistic = 2.09058
Pretest-Posttest Difference Scores
On pages 372-375 Judd and McClelland demonstrate the similarities and differences between ANCOVA and pretest-posttest difference score analysis. The major conclusions are that when one is doing a pretest-posttest difference analysis, you have restricted the regression coefficient to 1.0 for the pretest. Also, the Difference score analysis can never be a more powerful analysis than the ANCOVA, and may be substantially less powerful than the simple ANOVA model.
Nonorthogonal Continuous Variable
Sometimes researchers like to control a covariate precisely because there are differences on the covariate between groups. The researcher wishes to statistically control for those differences as they investigate differences in the dependent variable across the grouping condition.
According to Judd and McClelland this is called the adjustment function. There are a variety of reasons why statistically controlling for preexisting differences between conditions will seldom be an adequate strategy for estimating condition effects on the dependent variable.....Attempts to adjust for preexisting differences between treatment conditions will typically adjust insufficiently, and resulting conclusions about the condition effects are likely to be erroneous. p 377
Exhibit 13.7 has been coded for your convenience. It is a bit different than that shown on page 377 in Judd and McClelland. The variables I've coded are Y1, X1, X2, and Z these are the same as explained in the text. I have also coded X3 the interaction code for X1 and X2, and a final variable GROUP which simply codes each of the four groups in case you want to run this data through a typical ANOVA or ANCOVA program.
First, look to see if there is a relationship between the pretest (Z) and the effect codes (X1, X2, X3). If we just simply click the correlations tab under the multiple regression routines in Statlets you would see the following:
Correlation Coefficients
Z X1 X2 X3
---------------------------------------------------------------------------
Z 1.0000 0.5727 0.0818 0.2454
(1.0E-4) (0.6158) (0.1269)
X1 0.5727 1.0000 0.0000 0.0000
(1.0E-4) (1.0000) (1.0000)
X2 0.0818 0.0000 1.0000 0.0000
(0.6158) (1.0000) (1.0000)
X3 0.2454 0.0000 0.0000 1.0000
(0.1269) (1.0000) (1.0000)
---------------------------------------------------------------------------
Notice that the effect codes are unrelated to one another, but there is a definite relationship with Z.
Another way we could test to see if the covariate (pretest) and the effect codes are related would be to regress the covariate on the effect codes. Here is that output.
Dependent variable: Z
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT 50.25 0.396162 126.84 1.0E-4
X1 1.75 0.396162 4.42 1.0E-4
X2 0.25 0.396162 0.63 0.5320
X3 0.75 0.396162 1.89 0.0664
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 147.5 3.0 49.1667 7.83 1.0E-4
Residual 226.0 36.0 6.27778
---------------------------------------------------------------------------
Total (Corr.) 373.5 39.0
R-squared = 39.4913 percent
R-squared (adjusted for d.f.) = 34.4489 percent
Standard error of est. = 2.50555
Coeff. of variation = 4.98617 percent
Mean absolute error = 2.0
Durbin-Watson statistic = 3.0177
Finally, we have not changed any of the data except for the covariate, so if we did the regular ANOVA we would obtain the same results as before. These are repeated here just for convenience.
Dependent variable: Y1
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT 58.75 0.702179 83.67 1.0E-4
X1 3.25 0.702179 4.63 1.0E-4
X2 0.75 0.702179 1.07 0.2926
X3 1.25 0.702179 1.78 0.0835
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 507.5 3.0 169.167 8.58 1.0E-4
Residual 710.0 36.0 19.7222
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 41.6838 percent
R-squared (adjusted for d.f.) = 36.8241 percent
Standard error of est. = 4.44097
Coeff. of variation = 7.5591 percent
Mean absolute error = 3.55
Durbin-Watson statistic = 2.82676
We see that the first contrast is significant, the second and third are not.
This time, however, we might be concerned that the significance and magnitude of the difference between the posttest means between the new and old curriculum groups might be due to previously existing differences in the pretest. Let's estimate the full augmented model with the covariate included.
---------------------------------------------------------------------------
Standard T
Parameter Estimate Error Statistic P-Value
---------------------------------------------------------------------------
CONSTANT -4.84071 10.5528 -0.46 0.6493
X1 1.0354 0.61918 1.67 0.1034
X2 0.433628 0.50137 0.86 0.3930
X3 0.300885 0.522852 0.58 0.5687
Z 1.26549 0.209771 6.03 1.0E-4
---------------------------------------------------------------------------
Analysis of Variance
---------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
---------------------------------------------------------------------------
Model 869.429 4.0 217.357 21.86 1.0E-4
Residual 348.071 35.0 9.94488
---------------------------------------------------------------------------
Total (Corr.) 1217.5 39.0
R-squared = 71.411 percent
R-squared (adjusted for d.f.) = 68.1437 percent
Standard error of est. = 3.15355
Coeff. of variation = 5.36775 percent
Mean absolute error = 2.46327
Durbin-Watson statistic = 2.09058
Look at the changes in coefficients for the effect codes. Because the effect codes and the covariate were related, large changes in the coefficients were created. Look at the p-values for each coefficient.
Since the covariate and the are not orthogonal the sums of squares are not additive as they were when everything was orthogonal. To calculate the omnibus test, you will need a compact model with only the covariate used to estimate the posttest.
Judd and McClelland comment page 383 -- As a final comment before proceeding to consider interactions between covariates and contrast codes, we should mention that models incorporating two or more covariates are simple extensions of models with single covariates that have been discussed at some length.
Go to part 2