Franklin County STEM Research Academies for Young Scientists (STEM RAYS)

University of Massachusetts Amherst – Morton Sternheim P.I.

Year 1 Evaluation Report

Year 1 Evaluation Activities

            The evaluation efforts in support of this project, as led by Peterfreund Associates, have as their primary focus the collection of formative information regarding the quality of the teachers’ and students’ experience and documentation of attitudinal responses regarding outcomes.  More detailed analysis of student-related outcomes is being performed by the NSFAYS Research & Evaluation Center and extensive research on the teacher and student effects is being conducted by the research team led by Allan Feldman.  This report refers to the Peterfreund Associates piece.

Year 1 activities centered on the pilot of the program with twelve teachers and two themes.  Background data on the participating teachers was collected in November 2006, and responses to the preliminary training were collected in December.  In June, a third survey was administered to teachers regarding the overall experience and impact of participation for the year.   A follow-up to the summer institute is planned for the Fall of 2007.

The second cohort of teachers has also begun their training as of the writing of this report.  Background data surveys have been collected with this cohort, and we plan to follow up the training with a survey to be given in October.

State of the Program

            For the first year of a program, the speed of start-up and quality of the first-year offering have been exceptional when compared to other projects we have evaluated, with recruitment of teachers and commencement of activities starting almost immediately upon receipt of funds.  This efficiency is due to the experience and dedication of the leadership, who have shown themselves very attentive to details of the program and to have strong and constant avenues of communication with all participants and members of the team.  It is doubtful that they could have accomplished what they did without the years of relationships built up prior to the awarding of this project.

            Once the project was underway, recruitment of student participants was very strong, despite beginning in the later half of the school year, with each school able to muster up substantial participation.  It remains to be seen to what extent students will continue in the program for multiple years.

            As was seen on the surveys, teacher responses to participation in the program were very positive.  They enjoyed being able to bring authentic science to their schools and found the project team supportive and easy to work with.  The greatest problems with participation have been because of the demands on their time.  Although they are being paid, the teachers are expected to do a lot as part of the educational research aspects of the program, and the burden of data collection was substantially increased with the addition of requirements from the NSFAYS Research & Evaluation Center.  Finding a way to lessen the demands on the teachers’ time without endangering the research agendas is a problem that has not yet been completely solved.

            A particular problem with the survey for students developed by the NSFAYS group was that it took a very long time to complete and it is likely that the participating children were not able to give it the attention that would lead to terribly useful results.  There was also considerable resistance from the teachers to this measure as it took up a large part of the class time. 

Part of the response to data collection burden was to reduce the extent of data collected by the evaluation team in order to offset the new data collection demands.

The Teachers Participating in the Program

            Twelve teachers participated in the pilot program for STEM RAYS, most of whom taught grades 4, 5, and 6 but two of whom taught in high school.  These teachers’ level of teaching experience varied considerably, though most had more than 10 years of experience, and about half had majored in the sciences before becoming teachers.  Two-thirds had at least some prior experience doing research.  Only three had any experience with after-school activities prior to their participation, although most had considerable experience using research in the classroom.  Most participated so that they could improve their own teaching ability and to be able to enrich the science experiences of their students.

            The second cohort of eleven teachers taught a wider range of grades than those in the pilot group, representing early elementary through high school with greater representation in grades 7 and 8.  More than half had over ten years of experience teaching and about half had been science majors in college.  They differed from the first cohort in having fewer with prior research experience (27%) but more (64%) having prior experience with after-school programs.

Reactions to the Workshop

            Eight of the twelve teachers responded to the workshop survey.  The initial workshop was divided into two groups, one for individuals who wanted to participate in the watershed activity and the other for those who were concerned with arsenic.  In both cases, responses to the workshop were overwhelmingly positive, with very few major concerns.  The preparation they received for setting up the afterschool program and the community-building aspects were seen as particularly important.  They would, however, have liked more time and materials to work with.  The current workshop has benefited from the feedback associated with this first one.

Reactions to the Year 1 Pilot Program

            Seven of the twelve participating teachers completed the end-of-year survey.  Overall, responses to the afterschool programs showed them to be very successful.  Positive aspects that were frequently cited included developing an understanding of the science, networking with other teachers, being able to introduce authentic scientific research to the school, and the training sessions provided by the project.  There were some negative responses, but they tended to be singular.

            The table below demonstrates the various impacts of the program, as cited by the teachers:

            The teachers identified the best features of the program as the ability to get children involved in real-world science.

            Suggestions for improvement included involving more than one teacher per site, allowing for topic changes, and having higher pay.  The main problems encountered centered around the field trips, which were too time-consuming for some schools.  A number of suggestions were also given for how STEM RAYS could better enhance the notion of teachers and students participating in authentic science.