production date 2/5/00

One-Way Analysis of Variance (ANOVA)

Table of Contents	Objectives
Why Not t tests	Why can't we just do a bunch of t tests?
Terminology	A dictionary for ANOVA
Derivation of ANOVA	Deriving the ANOVA equations
Mean Squares	Mean Squares are variance estimates.
F statistic	The F statistic formula and what it means.
ANOVA Calculation Formulas	Formulas you would use in a calculator
Critical Values	Determining the critical value for the F statistic
A Hand Calculation Problem	Calculating the F statistic with a calculator
Two Statlets Problems	Two ways to conduct a one-way ANOVA
A Hand Calculation Problem	Conducting a hand calculation for the dependent t
Assumptions	Assumptions for ANOVA
ANOVA using Color	A Simple Applet to play with and learn about ANOVA
Multiple Comparisons	Learn about the different follow-up tests
Computer Problem 25	Calculating F statistic
Computer Problem 26	Calculating multiple comparison tests
Computer Problem 27	Using the Analyze/Multiple Samples/Completely Randomized Design procedure
Additional Information	Discover John Tukey
Questions/Test	Take the End of Chapter Test
Report	Send a Chapter Report to your Instructor

Analysis of variance is the procedure used when we want to investigate situations where there are more than two levels of the independent variable. Stated another way, when there are more than two groups, and we would like to compare their performance across a dependent variable use ANOVA. Using ANOVA, we determine if the groups differ on some continuous variable of interest.

An example of a situation where ANOVA would be used is a teacher who wants to find out if three different reading texts (Holt, Ginn, Lippincott) produces different reading levels in children. Another example requiring the ANOVA technique would be, "Are the survival rates different for five different groups of AIDS patients when the five groups are defined by different dosage levels of a new experimental drug?" An as another example, we look at the Obedience to Authority data to determine if the voltage level of the shocks given to subjects was significantly different depending on the experimental situation in this Chapter's Computer Projects section.

Why Not t tests

When comparing two groups, we have used t tests. Why not just continue? Couldn't a researcher compare group 1 against group 2, then group 1 against group 3, then group 2 against group 3 etc.? The answer is no; at least not without grave consequences. What happens is that we increase the family-wise error rate. Suppose that a friend shows you a machine that when you stick your hand inside can give you a severe shock. It is programmed to give this severe shock only 5% of the time. Each time you stick your hand into the machine, you have only a 5% chance of receiving this very painful shock. Your friend tells you that they will give you $50 if you give it a try. Oh, by the way, if you get shocked, you are required to give them back all the money given to you up to that point. Remember on each trial, the probability that you will get shocked is only 5%. However, if you repeatedly try this foolish attempt to make money, you will need to figure the probability of getting shocked over the repeated trials (family of trials). If you try it twice, the probability of your getting shocked is greater than the 5% considering both individual trial. It shouldn't seem illogical to you that if you keep trying, eventually you will get shocked even though the probability of getting shocked on any particular trial was still 5%.

The same thing happens with repeated statistical tests. If you set the alpha level of all your t tests at 5% and do repeated ones with the same subjects, you will eventually reject the null hypothesis when you shouldn't, the probability of rejecting the null hypothesis with repeated statistical tests increases just like the probability of your getting shocked increases with repeated trials. The following table illustrates the degree of this phenomena by showing what the actual alpha level would be with repeated t tests if each of the tests was set at .05 or .01.

[Image]

Analysis of variance procedures were developed to eliminate this increase in error rates. When using ANOVA, you are able to set one alpha level and test if any of the groups differ from one another. If you reject the null hypothesis, you know there is a difference. Unfortunately, you don't know where the difference is. Is it between group 1 and group 2 or between group 4 and group 12? There are follow-up tests discussed later in this chapter that allow detection of where these differences are, without inflating the alpha level over the entire experiment.

Before proceeding, there are some difficulties with Analysis of Variance techniques which are simply caused by the terminology used. When I was a student first studying ANOVA, I was dumbfounded, that I never encountered the word variance. If we were analyzing variance then where were these variance units we were analyzing? In order to make learning ANOVA as painless as possible, we start with a short ANOVA dictionary.

Terminology

In ANOVA, statisticians refer frequently to sums of squares. The term sums of squares really refers to the sums of squared deviation scores. You take a score and subtract some reference point (a mean) from the score. Then you square that difference and add these together. A typical sum of squares formula is shown on the left.

[Image]

As I noted above, the term variance is never mentioned in Analysis of Variance -- Strange isn't it. Variance has been calculated before when a sum of squares is divided by it's degrees of freedom -- we have called that variance. The equation on the right shows the formula used for calculating sample variance.

In ANOVA what we have previously called variance is perversely now called a mean square. The term makes sense, but did we need another? You will need to decide later. So when studying Analysis of Variance, the term mean square is a variance.

There are three very important sums of squares that we need to know about in ANOVA. These sums of squares are: total sum of squares, the between-groups sum of squares, and the within-group sum of squares.

Total Sum of Squares

The total sums of squares measures the total scatter of scores around the grand mean. The grand mean is the mean of all the subject's scores regardless of the group to which they belong. The symbol for the grand mean has two bars above the x (The grand mean symbol

)

Between-Groups Sum of Squares

The between-groups sum of squares measures the total scatter of the group means with respect to the grand mean. Since there are several groups in an ANOVA design, there needs to be a symbol for the mean of the groups which is unique for each group. We do this by placing the subscript j next to the symbol for a group mean. Thus, group means are symbolized by

. The mean for group 1 would be

and the mean for group 2 would be symbolized by

etc..

Within-group Sum of Squares

The within-group sum of squares measures the scatter of scores within each group with respect to the mean of that particular group. We have calculated these sum of squares before for samples. These within-group sum of squares are nothing new - they just have a new name.

Derivation of ANOVA

Suppose you are involved in an experiment with four separate cancer study groups. Group 1 gets a none of an experimental drug, Group 2 gets a light dose of the drug, Group 3 gets a medium dose, of the drug, and Group 4 gets a large amount of the drug. The score you are interested in is the number of cancer fighting white blood cells that remain in your subject's blood. The symbol for a person's score needs two subscripts. The first subscript (i) indicates which person you are referring to in a group. The second subscript (j) indicates which group you are referring to. Individual's scores are thus symbolized by x_ij. Thus, if Mike were the third person in the second group, Mike's white blood count would be symbolized with an x_3,2. If Alex were the 10th person in the fourth group, Alex's score would be symbolized as x_10,4.

Each subject's dependent variable score (the number of white blood cells) is composed of three parts. First, there is the grand mean which is the average white cell count for all the groups involved in this experiment. Second, there is a treatment effect that is symbolized as T_j. This is the effect of getting one of the four treatments. T₂ is the treatment effect of the light drug dose. Finally, there are many uncontrolled variables in this design these uncontrolled variables are called error. For example, Mike may have had more white blood cells than many of the other subjects because of his genetic makeup, or his diet etc.. Error effects are symbolized by E_ij. A particular subject's score is thus determined by the grand mean, the treatment effect and error. The equation below illustrates this.
[Image]

Both treatment and error effects can be estimated. Treatment effects are captured by the difference between a group's mean and the grand mean. If the treatment isn't having any effect, everyone's score will be about the same. Each group will have about the same mean and the grand mean will be the same as the individual group means.

If, on the other hand, the different treatments are changing the white cell count, then individuals within a treatment group will have distinctly different white cell counts than persons in another group. Their group mean will be different than the grand mean. The equation below shows how treatment effects are estimated.
[Image]

Error effects are estimated by comparing a subject's score with the mean of their group. Since everyone in their group receives the same treatment, any differences between their score, and the mean of their group can't be caused by the treatment. Everybody in their group gets the same exact treatment. Any differences in scores are termed error and so the difference between their individual score and their group's mean is a very good estimate of error. The equation below illustrates this calculation.
[Image]

Now take the equation above, and modify it slightly. We just subtract the grand mean from both sides of the equation producing the equation below.
[Image]

Now, stick with me; this isn't some trick. We simply substitute our estimates for the treatment and error effect into equation the equation directly above to produce the next equation.
[Image]

Now take the last equation and square each of the three differences and add them up for everyone. Remember these are called sums of squares. Some new symbolism is necessary to show that you are adding the scores for everyone. Here we introduce what is called double summation notation. Since people in a group are represented by the subscript i and the groups are represented by the subscript j, we need to sum everybody (all the i's) in group 1 (our first j) then all the i's in group 2 etc. We do this for every individual in every group. Our double summation symbol looks like the following:

. The subscripts tell you where you start (i = 1 and j = 1) and the superscripts tell you where you end. You stop when i = n and when j = k. You always do these double summations from the inside out, beginning with the i=1 summation. J is also 1 so add until i = n (the number of persons in that group). Next j goes to 2 and i returns to 1 for the second group. The following equation is formed when we square all our components and add them using double summation notation.
[Image]

Look carefully at the second component in the equation above. Here the grand mean is taken away from the group's mean, squaring that value, and adding the result together for every person in the group. The same value would be more simply calculated if the grand mean was subtracted from the group mean once for each group and that result then multiplied by the number of people in the group. Using this change, produces the equation directly following.
[Image]

Equation 12.9 illustrates the fundamental equation in ANOVA. There are three terms (sums of squares) in this equation. The first term is on the left of the equals sign, and the second and third terms are separated by the plus sign. The first term is the total sums of squares, the second term is the between-groups sum of squares and the final term is the within-group sum of squares. Equation 12.9 can be abbreviated using equation 12.10. Equation 12.9 and 12.10 simply show that the sum of squares total is equal to the sum of squares between-groups plus the sum of squares within-groups.
[Image]

Mean Squares

As noted much earlier, Mean Squares (MS) are another variance estimate. Mean Squares are the sum of squares divided by their respective degrees of freedom. We can calculate two important mean squares [between-groups (MS_BG) and within-groups (MS_WG)].

MS_BG is influenced both by treatment effects and by error effects. MS_WG is only influenced by error effects. After all, in this term, you are always comparing your score to your group's mean score. Equations 12.11 and 12.12 give the formulas for both of these mean squares.
[Image]

F Statistic

Remember that error is defined in ANOVA as all the effects that are not caused by treatment. Here we have two mean squares. One measures both treatment and error effects and the other measures only error effects. Sir Ronald Fisher saw that if you divided the mean square between groups by the mean square within groups that if there were no treatment effect present, both mean squares would only measure error effects so the result should be 1.0. If treatment was having an effect the scatter represented in the numerator of the fraction would be greater than the scatter in the denominator yielding a value greater than one. He named this statistic the F ratio. Equation 12.13 is the equation for the F statistic.
[Image]

ANOVA Calculation Formulas

Unfortunately, all the formulas just presented in this chapter are conceptual in nature. There are much better formulas to use in calculators. Note that there are essentially three components to calculate. The summation signs can be simplified because we indicate everyone with a subscript T. Individual groups are indicated with numerical subscripts. Here they are:

Sums of Squares

Degrees of Freedom

Mean Squares

F Statistic

Critical Values

There are several ways to find the critical values for ANOVA problems and the F statistic. If the degrees of freedom between groups (df_bg is 5 or fewer, you can use this simple Java applet. For problems with larger values for df_bg, you can use the equivalent of table typically found in the appendix of a textbook, or Statlets.

To use Statlets to find a critical value for the F statistic, simply choose the Plot/Probability Distributions procedure using menu selections. Click on the F distribution choice. Click the PDF tab, and the Options button to enter the correct degrees of freedom. The degrees of freedom for the numerator are the degrees of freedom between groups, and the degrees of freedom for the denominator are the degrees of freedom within groups (aka degrees of freedom error). The figure directly below shows the PDF graph and Options for an F distribution with 4 and 70 degrees of freedom.

Next click the Critical Values tab, and enter the tail areas using the Options button. The Critical values output and Options inputs are shown in the following figure. Since the F distribution values can never be below zero, and the distribution tests are only one-tailed, when the tail area is set at .95, the alpha level for the experiment is set at 0.05. With 4 and 70 degrees of freedom, the calculated critical F equals 2.50267 as shown below.

A Hand Calculation Problem

Suppose a teacher is interested in determining whether there is a difference in reading achievement results using three different reading approaches. She randomly assigns 5 children to each of the three reading programs.

We use our six-step approach.

Step 1

Step 2

alpha = .05

df_BG = (k-1) = 2

df_WG = (nT - k) = 12

F_cv = 3.88

Step 3

Group 1 48 50 53 52 50

Group 2 51 50 52 50 51

Group 3 61 62 60 63 60

Step 4

Try to calculate the F value following these steps using your calculator. After you have produced your answer, this Java applet will allow you to check your answer. You may also use Statlets to perform the calculations for the check. A problem requiring the use of Statlets is provided below. It is important that you are able to do these calculations by hand as problems similar to this are posed in the end of chapter examination.
a. Calculate SXT andSXT2

b. Calculate SSTOT

c. Calculate SSBG

d. Calculate SSWG

e. Calculate both mean squares

f. Calculate the F statistic

Step 5

Reject H_o

Step 6

The means of the three groups are significantly different. There are differences in reading depending upon which reading treatment a child was assigned (F (2,12) = 88.935, p < .05).

TWO Statlets Problems

To conduct a simple ANOVA using Statlets use the procedure Model/Analysis of Variance/ One Way ANOVA. Before beginning to use Statlets to perform this analysis read the user manual for this procedure.

The data below was collected by a research group investigating the whitening power of four new toothpaste formulas. The dependent variable value (Dep_Var) is a whiteness measure where the lower the number the whiter the teeth. The independent variable (Group) codes the four different toothpaste formulas using letters.
Dep_Var Group
26 A
25 A
29 A
21 A
20 A
18 A
24 B
17 B
16 B
13 B
21 B
19 B
32 C
34 C
29 C
19 C
27 C
28 C
23 D
29 D
26 D
20 D
24 D
25 D

After entering the data into Statlets using Copy and Paste, use the Model/Analysis of Variance/ One Way ANOVA procedure. Using the Input tab, enter Dev_Var as the Data (dependent variable) and Group as the Level Code (independent variable) as illustrated in the figure below.

The tab that actually calculates and constructs the ANOVA summary table is the ANOVA tab. Clicking it produces the following output.

Notice that the F ratio is 5.79 with a p value of .005, indicating that there is a difference in whiteness between these groups.

Next the Range tab was clicked and the Options shown first below was set to conduct the Scheffé follow-up test as discussed later.

While it is beyond the scope of this text, there are several follow-up tests available besides the Scheffe test, these include the LSD, Tukey HSD, Bonferroni, Newman-Keuls, Duncan and Dunnett.

The results of the Scheffé test shown below indicate that groups B and C were significantly different.

Of course, the significant difference between these two groups is quite nicely visually demonstrated by clicking the Boxplot tab. This output is shown directly below.

Data Arranged in Columns

If your grouping variable constitutes a column, and you want to do an ANOVA, you do not need to recode the data to do the statistical test. The Analyze/Multiple Samples/Completely Randomized Design procedure in Statlets will conduct a similar analysis to that above, with the data arranged in columns. Before proceeding, read the user manual for this procedure.

Some professional associations are reluctant to hold meetings in New York City because of high hotel prices and taxes. Are hotels in New York more expensive than in other major cities?

To answer that question look at the data below. For each of the four major cities, a random sample of eight hotels was taken from the 1992 Mobile Travel Guide to Major Cities. The cities are Washington DC (DC), Los Angeles (LA), New York (NY) and San Francisco (SF). Their daily price follows below.

DC LA NY SF
60 70 130 80
110 110 160 80
115 111 161 100
118 130 170 105
125 140 215 160
175 150 225 162
180 160 250 180
180 170 250 260

Enter the data into Statlets using the Copy and Paste procedure. Then use the Analyze/Multiple Samples/Completely Randomized Design procedure by selecting those Statlets' menu choices. Complete the Input tab like that shown below.

Clicking the ANOVA tab produces the ANOVA summary table and the interpretation as shown directly below.

Clicking the Range tests tab, and leaving the default LSD test selected produces the following results.

New York looks significantly different than the others. However, LSD tests, as discussed below have some limitations. Change the follow-up test conducted using the Options button, and see what happens. The reasons why sometimes a comparison is significant using one follow-up test, and may not be significant with a different test is discussed below in the Multiple Comparisons section.

Assumptions

To conduct an analysis of variance, certain assumptions must be made. It is assumed that subjects are assigned to the treatment conditions using a random and independent procedure. The population of scores for the dependent variable should be normally distributed. The variances of the populations from which the groups are drawn are assumed to be equal. Finally, the scores for the dependent variable must be continuous (interval or ratio).

Again, it is very hard to know about the first three assumptions. However, if you have equal sample sizes, the ANOVA procedure is robust to violations of the first three assumptions. The final assumption must always be true for the ANOVA results to be meaningful.

ANOVA using Color

The Java Applet found at this link allows you to conduct an ANOVA problem using both numbers and color as the key. The directions for using the applet and suggestions for playing with the applet are found directly on the page. It is strongly suggested that you follow the directions, and learn what this applet can teach. Be especially mindful of how outliers effect the F statistic and the MS_wg.

Multiple Comparisons

As we noted earlier, when we reject the null hypothesis using ANOVA, we know there is a difference between our groups. Unfortunately, we don't know where that difference is. A significant F statistic tells us that at least two of the groups were different, but not which ones. Multiple comparison tests are used to discover which groups are different, usually without inflating the type 1 error. There are lots of these multiple comparison tests. Some major statistics packages allow users to chose about a dozen different tests. Statlets as noted above, allows you to choose between seven different tests. Most of these tests are based on changing the critical value for a t test that would compare the differences in means between groups.

t test approach

Suppose we are using the Obedience to Authority Data as shown below in Computer Problem 25. Further suppose that we have conducted an ANOVA and rejected H_o. If the null hypothesis for the ANOVA had not been rejected then follow-up multiple comparison tests would not be attempted. The t test for the comparison of the first group (Remote) with the second group (Voice) can be written as shown below.

This t test is almost exactly like the independent t test performed in Chapter 10 with the exception that the pooled standard deviation is based on all the groups (four in this case) instead of the typical two for the t test. This increased information given by the four groups increases the power of this test. The degrees of freedom are also changed. Instead of the typical n₁ + n - 2, the degrees of freedom for this t test are those associated with the pooled standard deviation for all groups (for this example df = 76).

To perform most of the multiple comparison procedures, the t statistics for all the mean comparisons are computed, and then the calculated t is compared to a critical t value to determine whether the means of any particular comparison is large enough to indicate a significant difference between the means. It is how the t_cv value is determined that distinguishes the different follow-up tests displayed in Statlets Range tests tab. Why different t_cv values are chosen is beyond the scope of material presented in this text. However several of the tests most important characteristics are discussed below.

In Statlets, the Range tests tab provides the means, the differences between means and what in several textbooks is called the minimum significant difference (MSD). The MSD is calculated by the following formula, and is the value shown under the +/- Limits column.

Of course, if the difference in means is greater than the MSD, then the difference is marked as significant. While the tcv is not displayed, it could be calculated by substituting in the pooled s value from the Means tab, and the +/- Limits value from the Range tests tab into the formula for the MSD.

LSD

The least-significant differences method (LSD) simply uses the typical tcv value for comparison to the calculated t. This leads to one very unfortunate characteristic. While each test will have a set alpha level, the overall experimental error rate will typically be unacceptably large. For example, using our Obedience data, comparing group 1 with two and setting the alpha level at 0.05 is acceptable, but doing all six tests (1 vs 2, 1 Vs 3, 1 Vs 4, 2 Vs 3, 2 Vs 4, and 3 Vs 4) would lead to an overall error rate of 0.3 (0.05 * 6).

Tukey HSD

The Tukey HSD procedure is based on John Tukey's multiple comparison procedure. The HSD part stands for Honest Significant Difference. If all six follow-up tests were conducted with the Obedience data using this procedure, the error rate would be 0.05 for all six tests together. In statistical language, the Tukey procedure is testing at the 5% level experimentwise, while the LSD procedure is testing at the 5% level pairwise.

Scheffé

The Scheffé test was named after the mathematician Scheffé To conduct the test, simply subtract the means of any two groups you are interested in detecting a difference between. If the difference between the means is larger than the critical difference (CD) then those two groups are different from one another. The Scheffé test is quite conservative. It is possible to find a significant ANOVA and not be able to find which two groups are different using the Scheffé test. To calculate the MSD or critical difference use the equation below.

Computer Problem 25

Data from the Milgram Obedience to Authority study appears directly below. Use the Model/Analysis of Variance/Oneway ANOVA procedure to evaluate whether there is a significant difference in Volts across the different situations. If your instructor requests, submit the project 25 report.
Situation Volts Designation
Remote 300.000 Moderate
Remote 300.000 Moderate
Remote 315.000 Severe
Remote 315.000 Severe
Remote 330.000 Severe
Remote 345.000 Severe
Remote 375.000 Severe
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Remote 450.000 Danger
Voice 135.000 Slight
Voice 150.000 Slight
Voice 150.000 Slight
Voice 165.000 Slight
Voice 285.000 Moderate
Voice 315.000 Severe
Voice 315.000 Severe
Voice 360.000 Severe
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
Voice 450.000 Danger
In_Room 105.000 Slight
In_Room 150.000 Slight
In_Room 150.000 Slight
In_Room 150.000 Slight
In_Room 150.000 Slight
In_Room 150.000 Slight
In_Room 180.000 Slight
In_Room 270.000 Moderate
In_Room 300.000 Moderate
In_Room 300.000 Moderate
In_Room 315.000 Severe
In_Room 315.000 Severe
In_Room 450.000 Danger
In_Room 450.000 Danger
In_Room 450.000 Danger
In_Room 450.000 Danger
In_Room 450.000 Danger
In_Room 450.000 Danger
In_Room 450.000 Danger
In_Room 450.000 Danger
Must_Touch 135.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 150.000 Slight
Must_Touch 180.000 Slight
Must_Touch 210.000 Moderate
Must_Touch 255.000 Moderate
Must_Touch 300.000 Moderate
Must_Touch 315.000 Severe
Must_Touch 450.000 Danger
Must_Touch 450.000 Danger
Must_Touch 450.000 Danger
Must_Touch 450.000 Danger
Must_Touch 450.000 Danger
Must_Touch 450.000 Danger

Computer Problem 26

Using the Obedience to Authority data above, and continuing to use the Model/Analysis of Variance/Oneway ANOVA procedure, conduct a Scheffé´´ follow-up test to determine which groups are significantly different. If your instructor requests, submit the project 26 report.

Computer Problem 27

You are responsible for determining how different newspaper coupons for your company's product affect sales. You set up four groups For group 1, a newspaper advertisement without a redemption coupon will be printed. For group 2, the same advertisement will be used with a coupon worth 10 cents. For group 3, the advertisement and a 30-cents coupon will be used. For group 4, the advertisement and a 50-cents coupon will be used. You select twenty communities with daily newspapers within your sales region and randomly assign these communities to your four groups. For each group the local newspaper carries an advertisement for your product in the Wednesday edition. Sales figures from the twenty communities are totaled on the following Saturday, and that data is reproduced below. Is there a significant difference between the advertisement schemes? You will need to use the Analyze/Multiple Comparisons/Completely Randomized Design procedure. If there is a significant difference, use the Tukey HSD test and determine which schemes are significantly different. If your instructor requests, submit the project 27 report.
Group1 Group2 Group3 Group4
16 37 21 45
11 32 12 59
20 15 14 48
21 25 17 46
14 39 13 38

Additional Information

As noted above, John Tukey developed the Tukey HSD procedure. To learn more about him, visit http://www-history.mcs.st-and.ac.uk/~history/Mathematicians/Tukey.html.

Questions/Test

This link allows you to take a computer scored end-of-chapter test. If your instructor requests to see the results of this examination, you can either copy and e-mail or print the feedback you will receive immediately after taking the test.

Report

Please send a report indicating your understanding of this chapter to your instructor. You will need to know both your and your instructor's e-mail addresses.