The study by Johnson and colleagues (2005) used an elementary school-based population to examine the effects of the Success in Stages (SIS): Build Respect, Stop Bullying ® program. They used a 3x3 experimental design which crossed three experimental groups (one control x two treatments) with three assessments. The control group received a pretest and three posttests. The two treatment groups received a pretest, up to three internet-based sessions, guides for the administrators, staff, and family, and three posttests. The sample of 1,807 fourth and fifth grade students was from 12 elementary schools located across the US and in various types of communities (including urban, rural, and suburban areas). Schools with higher percentages of students receiving free lunches were given preference for inclusion. Schools were matched on several key variables including type of community, number of students receiving free lunches, and region of the country. Males comprised 51.1 percent of the study. A majority of students were White (59.4 percent), followed by African-American (17.5 percent), and Hispanic (13.1 percent). Just over 40 percent of students were eligible for free lunches. An equal number of fourth and fifth grade student participated.
Initial testing found that there was a significant difference between the control and treatment groups at baseline for the bullying role. Specifically, there were fewer bullies in the control group. The study used statistical procedures to control for these baseline differences. There were no other significant differences between the treatment and control groups.
Treatment and control schools were given the pretest as soon as they were ready to participate. The treatment groups were asked to complete the final two internet sessions a minimum of 30 days apart. The first posttests for the treatment groups were to be given a minimum of 7 days after the last internet session. The second posttests were to be given 4 months after the first posttest for all groups, and the final posttest was to be completed 8 months after the second posttest for all groups. All posttests were to be completed approximately 1 year from each group’s start date.
The treatment sessions were completed in a 2 month window, ranging from 1 month to 3 months. The overall program length was 12 months when factoring in posttests.
The study measured outcomes on the percentages of students who were no longer categorized in the three roles related to bullying (bully, victim, or passive bystander) at posttest, as well as the percentage who were no longer participants in all three roles combined. Students completed all assessments and measures of self-reports on computers.
Data was measured using two sets of analyses: (1) posttest only comparisons of the number of participants who no longer reported involvement in any or all roles, and (2) a repeated measures comparison of the number of participants who reported they no longer participated in any or all roles. Posttest only comparisons used an arcsine transformation difference in proportion of movement into Action/Maintenance. An intent-to-treat analysis was used to examine all records, rather than only those that could be matched. The last observation carried forward (LOCF) method of replacement was used when a posttest value was missing. This method replaced the missing value with the last known observation for that record. The study also used repeated measures analyses for all students across all time points, as well as random effects logistic models to examine any changes across the course of the study and to control for baseline covariates.
The study by Evers and colleagues (2007) focused on middle and high school populations. The study used a 3x2 factorial experimental design that had three experimental groups crossed with two assessments. The experimental groups were given pretests and posttests as well as the intervention (SIS program). The control group only completed pre- and posttest measures.
The study recruited a total of 12 middle schools (grades 6-8) and 13 high schools (grades 9-12) from a variety of communities (rural, urban, suburban), and like the previous study, preference was given to schools with higher percentages of students eligible for free lunches (roughly 45 percent of all students combined). Schools were then matched based on region of country, type of community, and number of students eligible for free lunches. There were significant differences found for grade level and race in both population samples. The high school sample also had significant gender and baseline stage of change differences. Statistical analysis procedures were used to control for effects of all baseline differences.
There were 1,237 student participants with the largest percentage of students (45.1 percent) in the 7th grade and the rest spread throughout the 4th through 6th grades. Just over 50 percent were female and 48 percent were eligible for free lunches. Students who could be matched at baseline were more Hispanic (27.4 percent) than generally found in the US (14 percent). Non-Hispanic whites were fewer than found in the US as a whole, and there were comparable percentages of African-Americans and Asian-Americans.
There were 1,203 high school participants who could be matched at post-test and the student distribution more clearly paralleled the ethnic distribution throughout the US. Fifty-five percent of the sample was female with 42 percent of the total sample eligible for free lunches. Ninth graders made up the largest percentage of the sample at 41.6 percent.
The study experienced several problems for matching and retention. Matching the follow-up assessments to the first session assessments was difficult, with two-thirds of the students not retained in the study. The authors were able to calculate the rate of matched records based on the number of students whose login codes matched at their first session and posttest assessment session. Unmatched records included students who had used new login codes and students who were not retained in the study. The authors took these matched and unmatched records into account when planning for analyses. The middle school had significantly different rates of matching, with the control group having 59.2 percent of records matched. Treatment group 1 had 48.8 percent matched and treatment group 2 had 25.8 percent matched. The high school study was able to match 50.3 percent of the control group, 50.5 percent of treatment group 1, and a significantly lower match of 34.6 percent, for treatment group 2.
This study followed the same methodology as the elementary school study with regard to timeline and implementation of the program. This study also used the same statistical procedures from the Johnson and colleagues (2005) research study when analyzing the data.