Research & Action Report Fall/Winter 2004

Research projects at the Wellesley Centers for Women can take a variety of forms. The mix of approaches ranges from the most “traditional” in which researchers develop an hypothesis, design a study to test it, draw a sample to use in gathering information, and go into the field to collect data, to secondary analyses of existing data sets, and to reviews of published research, such as WCW’s 1992 report for the AAUW, How Schools Shortchange Girls.

Although often less visible within the mix, evaluation research plays an important role. Evaluations of program effectiveness are critical in helping us understand the kinds of interventions that can make positive differences in the lives of women and children. These evaluations are most reliable if they are conducted by scholars who have not been involved in developing or implementing the original program but who have a good understanding of the complex nature of the problems the interventions are designed to address. Here at WCW we evaluate programs developed outside the Centers, and we encourage evaluations by other groups of our own programmatic work.

One example of an outside evaluation of WCW work is the recently completed study of the effectiveness of Lisa Sjostrom’s and Nan Stein’s 1996 curriculum, Bullyproof. The evaluation was funded by the Centers for Disease Control and Prevention with a three-year grant of $630,000 to a consortium composed of the Austin, TX, public schools; SafePlace, a large domestic-violence and sexual-assault agency in Austin; and the University of Texas School of Social Work. The results clearly showed that students benefited from classroom lessons on bullying and harassment, and they underscored the importance of the gender socialization component in dating-violence prevention programming. Students in the intervention schools showed greater accuracy over time in identifying behaviors that constituted sexual harassment, as compared to students in the control group of schools. The study concluded that the Bullyproof curriculum lessons were successful in increasing student and staff knowledge about sexual harassment. These findings will help guide educators to materials that effectively address the bullying and harassment so widespread in our nation’s schools.

Our evaluations of projects from outside the Centers are smaller in scale than some of our other research and action efforts, but they are particularly important in answering both outcome and process questions. Program developers need to know as much as possible, as soon as possible, about whether their efforts are on track, and what, if any, adjustments they may need to make for their programs to be most effective. Gathering such information is referred to as process evaluation. Program developers and implementers also need to have outcome data. Do the programs really produce the desired results? Are they worth the investment in time and money required to implement them? WCW researchers are currently engaged in evaluating a number of innovative programs in a variety of fields. Five of them targeting girls and young women are described in the paragraphs that follow.

Evaluating Nontraditional Career Programs
Access to nontraditional careers has been an important part of increasing opportunities for girls for more than a quarter of a century. There is still much to be done to remove the persistent barriers to the full participation of girls and women in areas requiring mathematical and technical skills. WCW researchers are currently evaluating three programs specifically designed to encourage girls in science, technology, engineering, and mathematics.

4 Schools for Women in Engineering
The National Science Foundation (NSF) has funded the 4 Schools for Women in Engineering project, a consortium of four Massachusetts engineering colleges committed to achieving gender equity in science, technology, engineering, and mathematics (STEM). The consortium includes Northeastern, Tufts, Boston University, and the Worcester Polytechnic Institute. By joining forces, the partners hope to develop a model to demonstrate how engineering concepts can become part of the middle-school curriculum in ways that encourage girls to continue along the engineering pathway. Each institution is in the process of implementing a unit in eighth grade classrooms in their local public schools. It is an opportune moment for this intervention since Massachusetts is the first state in the nation to introduce engineering as part of the mandated preK–12 education frameworks.

The evaluation, headed by Sumru Erkut and Fern Marx, is designed to assess changes in attitudes towards STEM among participating students, to examine changes in Massachusetts Comprehensive Assessment System (MCAS) scores, and to track the number of math, science, and technology courses taken by students, particularly girls. WCW researchers gathered baseline data on program participants and then collected follow-up information at the end of the school year. Data analyses are currently underway.

Hear Our Voices
Erkut and Marx are also evaluating a second NSF project, Hear Our Voices (HOV), designed to develop and nurture girls’ skills and interest in technology. The program is administered through the Boston Museum of Science computer clubhouse, which funded 10 computer clubhouse sites across the country in year one; an additional 10 sites will be funded in year three, and first-year sites will receive continuation funding. The key element of the HOV program is that some portion of time be set aside at each clubhouse site for girls-only programming. Each site has staff specially trained by HOV to work with girls and young women. The evaluation is designed to measure change in participants’ attitudes toward computers and possible careers in computing. Baseline information on participating girls has been gathered at all 10 sites, and interim surveys are in the process of being administered. In addition to participant surveys, Erkut and Marx are monitoring program implementation among the 10 programs through quarterly reports and, in two instances, through program observation.

This evaluation effort will provide outcome data that can help to determine whether HOV should be slated for widespread dissemination and replication across the country. In addition, it will provide process information that can help fine tune the programs as they are being implemented.

Rosie’s Girls
Rosie’s Girls was developed in 2000 through a partnership between the Strong Foundations, Inc., and Northern New England Tradeswomen, Inc. This unique project consists of a three-week summer program for young adolescent girls. Designed to build self-esteem, perseverance, and leadership through hands-on instruction in the basic skills of such trades as construction, plumbing, welding, and auto repair, the goal is to increase awareness of career possibilities in nontraditional fields and to expand girls’ math and science skills through concrete applications. Fern Marx’s evaluation will measure changes in both knowledge and attitude among participants in the summer program.

Evaluating Self-Defense and Self-Esteem Programs
WCW is engaged in evaluating two approaches to increasing girls’ sense of their ability to take care of themselves.

Project BOLD
Marx designed and carried out a three-and-a-half year evaluation of Project BOLD, a selfdefense and violence-prevention program developed by Girls, Inc. The program has three components, each targeted to a different age span: Kid-Ability Jr. for 6- to 8-year-olds; Action for Safety for 9- to 11-year-olds; and Living Safe and Strong for 12- to 14-year-olds. The program was evaluated at a variety of New York City sites, including two settlement houses and several schools.

Fighting, teasing, and bullying were the most frequently mentioned problems among girls across all three age groups. After participating in the program, 60% of the girls in the two younger groups said they felt safer and better able to protect themselves both physically and mentally. Almost three-quarters of the girls felt that the program had provided them with a safe setting for discussing topics that they couldn’t talk about in other places.

The program was particularly effective in helping the oldest group of girls understand the causes of violence against women and increasing their awareness of this problem in their community. Living Safe and Strong appears to have positively affected the overall self-esteem of participants and to have particularly increased their sense of having strong bodies. These evaluation data support the continuation and wider implementation of Project BOLD.

Girls’ Life Empowerment Awareness Program
2004 marks the fourth year of Marx’s evaluation of Girls’ Life Empowerment Awareness Program (LEAP). LEAP, which serves girls ages< 8 to 14, provides them with opportunities to reflect on the nature of violence against women and girls in their communities and to learn techniques of self-protection and selfdefense. One unique aspect of the program is its use of “teaching women”—Wellesley College undergraduates and other women in the community who have been trained both in martial arts and in working with young adolescents. The evaluation has shown that, overall, the program has a positive effect on participants’ self-esteem. A five-item selfesteem scale was administered at the beginning and at the end of the six- to nine-weeksession program, and a significant difference was found. Participants were more aware, and prouder, of their physical strength.

Is Evaluation Worth the Effort?
People often assume that carefully developed programs based on research findings and designed in collaboration with the women and girls they are intended to serve, will, of course, be effective. Observers see young women and girls enjoying an activity or program and see this as evidence of effectiveness. The reality is often more complex. Good intentions are not enough, and research data can be incomplete or misinterpreted. Program implementers can make mistakes in the delivery of the activities. Only a carefully designed and executed evaluation can truly answer the critical question—is the program achieving the desired results for the majority of the participants and is it doing so efficiently? Such information can play a major role in future programmatic directions and crucial decisions on funding priorities.

Evaluation efforts are often called for and almost as often, are under funded. Designing an evaluation is not a simple “add-on” to a program, but as complex a process as any other research undertaking. Evaluation research provides critical information that can make a difference over the long run between money spent on less effective, albeit initially interesting, approaches and the replication and dissemination of programs that can truly influence the future of the participants.