(All of the examples provided on this page come from the book Rival Hypotheses: Alternative Interpretations of Data Conclusions by Shuyler W. Huck and Howard M. Sandler.)
1. Miss America
In an attempt to prove that billboard advertising is the best medium for advertising a product in order to increase brand name recognition, which would hopefully lead to increased sales of the product, the Institute of Outdoor Advertising (IOA) conducted a research study. Huck and Sandler state, "The goals and results of this empirical investigation were reported in two IOA promotional brochures. " The text of the brochures goes as follows:We've long believed that Outdoor can outperform other media in getting across a message to the public. What we needed, here at the Institute of Outdoor Advertising, was a way to prove it. Last year we thought of a way. We would see if our medium, by itself, could increase public awareness of the name of Miss America, 1975.Huck and Sandler say that, "the promotional material put out by the IOA gives the distinct impression that outdoor advertising was the causal agent that brought about the tenfold increase in public awareness of her name." Which of the following threats do you believe could provide an alternative hypothesis for the results found in this study?
We approached the Outdoor companies with our plan. We asked them to donate space not already sold or earmarked for public service announcements. They gave us 10,000 panels -- or about $1.5 million worth of Outdoor at the going rate.
Our poster [which showed a large picture of Miss America with her crown and the message "Shirley Cothran, Miss America, 1975"] was to go on display for two months beginning January 1, 1975. But before it did, the Outdoor companies sponsored a series of studies to determine public awareness of Miss America's name prior to posting. Random sample surveys were conducted during November and December, 1974 in 44 metropolitan markets by 25 colleges and universities and 12 independent research organizations. Over 15,000 adults were questioned.
Despite all the exposure Miss America had received (previously) on TV and radio and in print, only 1.6% of the respondents (italics theirs) gave the correct answer when asked, "What is the name of Miss America 1975?"
Then our posters went up. And in February and March 1975, a second wave of over 15,000 interviews was conducted by the same research teams. This time, 16.3% of the respondents (italics theirs)-- about one of every six -- knew who Miss America was. That's a 10-fold increase in awareness. Projected nationally it would mean that Outdoor had communicated a new and difficult name to more than 20 million adults. Through a two-month posting, Outdoor made Shirley Cothran the best-known Miss America in history.Instrumentation Regression to the Mean History Testing
For further information:
History | Instrumentation | Maturation | Mortality | Regression | Testing
2. Air Force Officer School and Dogmatism
Huck and Sandler write, "Many people believe that the structure of the armed services and the inherent chain-of-command basis of communication attract highly dogmatic volunteers. Military commanders, of course, disagree; they claim that they have a need for officers who are open-minded, tolerant, and able to win the respect and loyal cooperation of the personnel they direct. Aside from this question as to the type of person attracted to a military career, a researcher recently wondered whether a 14-week stint at officer training school would cause the junior officers to become more or less dogmatic, or would it have no effect on dogmatism? And would the influence of the 14-week program be the same for those participants who began with high levels of dogmatism as it was for those who began with low levels?
"The subjects in this investigation came from a pool of 764 officers who completed the three-and-a-half-month Squadron Officer School (SOS) at Maxwell Air Force Base in Alabama. As the researcher saw it, there were several facets of the SOS program that might have made for a change in dogmatism. For example, each student was given extensive feedback from peers, the opportunity to discuss the personality characteristics of other trainees, a chance to deal with unstructured situations, and experience in planning military strategy in areas divorced from his field of expertise. These and other similar activities might, according to the researcher's hypothesis, cause the students' dogmatism levels to decrease over the 14-week time interval.
"During the first and last weeks of training, all students in the SOS program were given a copy of the Rokeach Dogmatism Scale, Form E. (In this study, it was titled the Rokeach Opinion Scale.) This measuring instrument is made up of 40 statements, each of which is rated on a -3 to +3 scale so as to indicate the extent of one's agreement or disagreement. Two of the statements go as follows: 'Most people just don't know what's good for them,' and 'A group which tolerates too much difference of opinion among its members cannot exist for long.' From among the SOS students who completed and returned the Rokeach Scale at both the pretest and post-test periods, 250 were randomly selected. Then, based upon an examination of the pretest scores, the 250 subjects were subdivided into five groups of 50 subjects each. In terms of the dogmatism continuum, these subgroups were described as high, above average, average, below average, and low.
"The data were statistically analyzed in two ways. First, the pretest mean for all 250 subjects was compared to the overall post-test mean. Results indicated no significant difference. Next, a two-way analysis of variance was used to see whether the five subgroups were changing in a similar fashion between the beginning and end of the SOS program, or possibly not changing at all. The pre- and post-test means for the subgroups turned out as follows:
Subgroup pretest post-test High pretest scorers 170.04 161.14 Above average pretest scorers 151.02 145.42 Average pretest scorers 138.12 137.26 Below average pretest scorers 126.12 128.78 Low pretest scorers 108.24 117.32
"The statistical analysis indicated a significant interaction between subgroups and pre-post trials. (Such an interaction simply means that the change from pretest to post-test is not the same for all subgroups.)
"Based on the subgroup means presented in the preceding table and the significant statistical finding, the researcher stated that 'Subjects high in dogmatism on the pretest tended to become less dogmatic by the last week of training while those scoring below the mean tended to become more dogmatic'.... In a way, it looks as if the SOS training program causes the participants to become more homogenous in terms of dogmatism." Can you think of any alternative explanations to explain this result?Maturation Regression to the Mean Mortality Instrumentation
For further information:
History | Instrumentation | Maturation | Mortality | Regression | Testing
3. Camping Out
Huck and Sandler write, "...you are undoubtedly aware of the fact that the daily routine in mental institutions is extremely stifling for the patients....Fortunately, there are some staff members who are sensitive to the ill effects brought about by the same routine day in and day out -- and who care enough to try to do something about it. Usually, these attempts at breaking the monotony are not evaluated by means of any sort of formal research...But on occasion, data are collected in an attempt to verify scientifically the worth of the innovative program. One such research study...was conducted in Utah, and the new activity -- camping out -- was about as different from the daily institutional routine as you could imagine.
"The subjects in this investigation were 25 male and female adults aged 19 to 62, who were randomly selected from a state mental hospital located in an urban area of Utah. These individuals were taken to an isolated camp site in the mountains near Flaming Gorge. The patients and staff were on this camping trip for five days...and while on this excursion the staff maintained a very low profile. The patients had the responsibility of forming teams for cooking and clean-up, of arranging sleeping accomodations, and of structuring their own free-time activities. Other than busing the group to and from the campsite, the staff took charge on only two occasions -- when the group went on a raft ride down the river and when they visited a nearby trading post for Cokes and candy.
"The researchers expected this week-long camping retreat to serve as a therapeutic tool, and in particular they hypothesized that the activities would bring about increased social interaction among the patients. To test this hypothesis, two types of data were collected on both the first and final days of the camp-out. According to a prearranged random time sampling scheme, five-minute sessions of group interaction were taped, unobtrusively, on an audio recorder. In addition, photographs were taken of the patients.
"One week after returning, all staff members and five of the patients who had gone on the camp-out used the audiotapes and pictures to rate the 25 patient campers in terms of social interaction. These ratings were obtained by using a modified version of the Bales Interaction Matrix. For each of the 25 patients, average ratings from the staff judges were computed. Then, the ratings within each group of judges were averaged across the 25 patients to obtain overall Monday and Friday ratings for the entire group. Since the Bales Interaction Matrix yields 12 subscale scores, there were two sets of 12 pretest and post-test composite ratings on the 25 campers, one set from the staff judges and the other set from the five patient judges.
"When the prestest and post-test data were tested statistically, the researchers found that there was significantly more social interaction at the end of the five-day camping excursion than there had been at the beginning. The ratings from the patient group of judges showed increases on 11 of the 12 subscales of the Bales instrument, while the ratings from the staff members indicated significant improvement on all 12 subscales. One possible interpretation of these results is that the camping activities and unique environment brought about increased social interaction. Might there be other plausible explanations for the observed differences between the Monday and Friday ratings?"History Testing Maturation Instrumentation
For further information:
History | Instrumentation | Maturation | Mortality | Regression | Testing
4. Groups for Parents
Huck and Sandler write, "Groups for Parents is a packaged method that offers parents both a support group of other parents and didactic information on an integrated humanistic behavior modification approach. The authors of 'Groups for Parents'...published a study evaluating the effectiveness of their approach in 'improving both general child behaviors [and] individually targeted ones.' They also reported success in increasing the parents' rates of positive reinforcement along with the rates of compliance in their children.
"The method of evaluation was quite simple. Thirteen groups of parents (a total of 277) met once a week for two and one-half hours over an eight week period. About one-half of the parents were referred by various community agencies; the rest had heard about the program from friends or other informed sources. The pre- and post-test measures used included a problem behavior checklist, positive reinforcement rates (measured by the parents), compliance rates (also measured by the parents), and client satisfaction (self-report). Approximately two-thirds (180) of those enrolled completed the entire eight-week course.
"The data analyses were equally straightforward, consisting of analyses of the differences between pre- and post-test means. Significant results that concern us were reported on the problem behavior checklist, reinforcement rates, and compliance rates. In addition, a very high rate of client satisfaction at the end of the study was reported." Can you think of any plausible explanations, other than that the program was successful, for the results the researchers found?Mortality Instrumentation Regression to the Mean History Testing Maturation
For further information:
History | Instrumentation | Maturation | Mortality | Regression | Testing
