1. Miss America
Instrumentation
While the authors state that one might think that the extraordinary rise in name recognition might be due to a change in the way the interviews were conducted, because the same researchers conducted both the pre- and post-tests, and because it is likely that the researchers were fairly skilled at interviewing, and finally, because the stated interview question was quite simple, it is not likely that there was any difference between the way the pre- and posttests were conducted. Therefore, instrumentation, while a possible threat, is not a plausible threat.
TO RETURN--Click on the "Back" Button
1. Miss America
Regression to the Mean
Because there is no reason to assume that the subjects who were interviewed were at either extreme for recognizing the name of Miss America, there is no reason to believe that regression to the mean posed a threat in this study. Furthermore, it is likely that the people interviewed in the pretest were not the same as the people interviewed in the posttest. Therefore, regression is even less of a threat, because there is not consistent degree above or below any population mean by which to measure the respondents.
TO RETURN--Click on the "Back" Button
1. Miss America
History
Correct! History poses the greatest threat in this study. The authors give two reasons for this. First, the study seems to assume that during the time between their pre- and posttests, no other media existed. Of course, this is not the case. At any time during the two months the billboards were up other forms of media could have reported on Miss America, and the posttest respondents could have learned her name through any one of those media. Second, the authors note that the unique nature of the billboard, in fact, caused other forms of media to cover the story of the research study! In other words, Miss America's name was being discussed more than usual in newspapers, on the radio and on television. The study itself had become a historical event which probably caused more people to hear and remember Miss America's name. It is very plausible that it is because of this event, rather than simply that people saw the billboards, that the results were so extraordinary.
TO RETURN--Click on the "Back" Button
1. Miss America
Testing
It does not say explicitly in the text whether or not the same people who were interviewed in the pretest were interviewed in the posttest. As the authors state, "If they were the same, then clearly the pre-post percentage change could be attributable to the stimulus of the first interview rather than to the intervening outdoor advertisements. However, we...feel that this is an improbable competing explanation. We strongly suspect that the people interviewed during the pretest were not included in the post-test sample." If this is the case, then this study would be an example of the Separate Pre-Post Samples Design.
TO RETURN--Click on the "Back" Button
2. Air Force Officer School and Dogmatism
Maturation
Because the officers were all adults, and the course was only 14 weeks long, there is no real reason to believe that, without the treatment, in a 14 week period of the goup members' normal lives they would have matured in any way that would have a particular effect on their levels of dogmatism. The subjects were randomly selected from an overall group. This selection was an attempt by the researchers to improvise a kind of randomization of assignment for their study, and this kind of randomization is meant to control for threats such as maturation. Finally, the results of the study are not charateristic of a maturation threat, since some of the subjects have higher scores on the posttest and some have lower scores on the posttest. Maturation would generally be indicated by a consistent, though probably differential, increase or decrease across all the paticipants' scores.
TO RETURN--Click on the "Back" Button
2. Air Force Officer School and Dogmatism
Regression to the Mean
Correct! The results of this study fall into the classic pattern of regression to the mean, with the people at the high and low ends of the scale regressing more toward the mean than those people only slightly above or below, while those who scored at the mean barely change at all. The authors state, "the observed correlation between the Rokeach pretest and post-test dogmatism scores for the 250 participants in this particular study was +.71. Based on this correlation, the researcher made a prediction about how high each of the five subgroups would have scored on the post-test, assuming that the SOS training program had absolutely no impact on the participants' dogmatism levels and that any pre-post changes were caused entirely by the phenomenon of regression. These 'estimated' post-test means for the five subgroups turned out to be almost identical to that actual post-test means." In other words, even the researchers suspected that regression caused the pre-post differences.
TO RETURN--Click on the "Back" Button
2. Air Force Officer School and Dogmatism
Mortality
The researchers chose to study only the scores of participants who had completed both a pretest and a posttest. In other words, of the people from whom the study draws its results, none dropped out. Therefore, mortality is not a threat.
TO RETURN--Click on the "Back" Button
2. Air Force Officer School and Dogmatism
Instrumentation
First, the instrument used in this study was an accepted and previously-used test. It also did not rely on human observers who might grow more experienced over time or who might not be the same people for both tests. And the researchers do not say that the test was changed in any way from pre- to postest, so we can assume it was the same test. This leads us to conclude that instrumtation did not pose a treat in this study.
TO RETURN--Click on the "Back" Button
3. Camping Out
History
Because the group was out in an isolated camp site, it seems unlikely that there would have been some historical event, unrelated to the program -- i.e. the camping expedition -- under study, that the group would have been exposed to. The short amount of time between the pre- and posttest makes history an unlikely threat as well. Of course, it could have been something unplanned in the camping trip -- an accident, a chance encounter -- that actually caused the observed effect, but it would be hard to say whether such an event could be considered a history effect or just a part of the treatment of taking the patients away from the hospital routine.
TO RETURN--Click on the "Back" Button
3. Camping Out
Testing
The researchers stated that the data were collected by unobtrusively recording on audiotape five minutes of group interactions using a random time selection scheme. Of course, the it is difficult to know just how unobtrusive the recording was, but assuming the researchers were fairly careful, they could have turned the audiotape recorder on without the group members realizing it. If the group members were aware that they were being recorded, it would seem that they would feel self-conscious during both recording sessions. In other words, the data collection during the first session would still not have alerted or primed the group members to behave differently at the next session. If, however, the group members realized after the first recording what the purpose of the recording was, and therefore tried to behave differently the next time, testing could pose a threat. So, while testing is not a highly plausible threat, it is a possible one. There is, however, a more plausible threat.
TO RETURN--Click on the "Back" Button
3. Camping Out
Maturation
Because the group members were all adults and the treatment took place over only 5 days, it is doubtful that the group members would have otherwise, with no treatment, matured in any measurable way. In addition, the group members were chosen at random, which is a selection method that is used to balance out possible differential rates of maturation.
TO RETURN--Click on the "Back" Button
3. Camping Out
Instrumentation
Correct! The authors feel that instrumentation is the most plausible threat in this study. First of all, the researchers employed the Bales Interaction Matrix for examining group interactions, and they had the hospital staff and five patients do the ratings, using this instrument, of the group interactions. The Bales instrument is a very complex measurement tool, and it is not likely that many or any of the staff or patients had experience using it. Therefore, the learning curve for using this scale would have been large, and it is likely that the raters' ability improved with their second round of rating the group interactions.
Another alternative hypothesis, while not exactly instrumentation, relates to the aspect of the instrumentation threat that deals with the characteristics of the human observers, in this case, the staff and patients who served as the raters or judges of the group interaction tapes. This threat is called the Rosenthal effect. The authors say, "The staff and patients probably expected the camp-out to facilitate social interaction. And this expectancy could very well have distorted the judges' perceptions when they listened to the tapes and looked at the second set of pictures. It is not at all unlikely that they selectively heard and saw things that confirmed their expectations, while not noticing occurrences that ran contrary to their hopes." In other words, because the staff and patients had a personal investment in this program, they could have consciously or unconsciously tried to prove that the program was effective. The authors believe the data analysis would have been much stronger if outside raters had been used to evaluate the tapes and photographs.
TO RETURN--Click on the "Back" Button
4. Groups for Parents
Correct! Actually, the reason there are six choices for this example is that all six are threats to the internal validity of this study. The authors say, "First, we must consider the rival hypothesis of experimental mortality....This raises the possibility that those who were not finding the approach helpful were more likely to drop out than those for whom the approach was succeeding.This would also account for the high rate of client satisfaction reported in the study.
"Second,instrumentation could account for the increases in both reinforcement rates and compliance rates; that is, since in both cases the parents were the measuring instruments, it is likely that they were better able to identify both types of behaviors even when the rate was unchanging. We must also consider the rival hypothesis of history; other events outside the study may have been taking place during the eight-week experimental period which affected the dependent variables. It is also possible for the problems to have eased up on their own during this time, thus giving rise to the rival hypothesis of maturation.
"Other rival hypotheses to be considered include both testing (perhaps the pretest influenced the parents' responses to the post-test) and statistical regression....in this study parents were referred as a consequence of having extreme problems -- we would not expect their problems to be as extreme on a second measurement.
"Needless to say, although the program may well be effective (no data to the contrary are reported), we would not enroll on the basis of the data presented in this study."
TO RETURN--Click on the "Back" Button