The science content dimension of year three of Project PASS focused on teachers using a big question to explore the content of global climate change. A multiple-choice instrument was used to measure the development of teacher science content knowledge related to the curriculum big question, “What does it mean to me that Las Vegas was once under the sea?” The task was completed as a repeated measure at the beginning and end of the year three summer institute of Project PASS.
Task Development
Following the re-design of the CCSD 9th grade course, “Principles of Science,” a need was identified for increasing teacher’s content knowledge relating to gases and the atmosphere. The various activities of the PASS summer institute were then developed to address this need.
A multiple-choice instrument was developed, using questions from two different concept inventories and a commercially-available test question software program. The Greenhouse Effect Concept Inventory (Keller, 2006) and the Geoscience Concept Inventory (Anderson & Libarkin, 2006; Libarkin & Anderson, 2005) are validated concept inventories for use with introductory science courses for nonscience majors. ExamGen (ExamGen, 2007) is a software program that contains databases of externally-reviewed test questions; it is used by teachers within Clark County School District. Items were taken from the two concept inventories that most closely aligned with the content and activities of the summer institute, with additional topics covered through use of questions from the ExamGen software. A total of 25 items were used on the final instrument.
Method
Year three PASS participants completed the multiple-choice instrument described above as a repeated measure. The PASS summer institute, lasting for six days during the period August 7-15, 2007, constituted the intervention for this measure. The pretest was completed on the first day of the institute (August 7, 2007). The posttest was completed on the final day of the institute (August 15, 2007).
Results
The multiple-choice instruments were scored by comparing the participants’ responses to a key of correct answers. The responses were entered into Excel and then scored as a ‘1’ (correct) or ‘0’ (incorrect). Each participant’s total score (out of 25) and percent score were calculated. A total of 43 participants completed both the pretest and the posttest.
A paired-samples t-test was conducted on the participants’ scores on the instrument. A statistically significant increase was found for the score from pretest (M = 15.16, SD = 5.10) to post (M = 17.30, SD = 3.85), t(42) = -5.442, p < 0.000. A second measure of the increase of the scores that was calculated is the normalized gain, which is defined as the ratio of the change in score to the maximum possible change in score (Hake, 1998). The normalized gain for all participants was 0.18, considered a low gain.
References
Anderson, S. W., & Libarkin, J. C. (2006). The Geoscience Concept Inventory. Retrieved April 27, 2007
ExamGen. (2007). ExamGen: Test item databases for today’s teachers. Retrieved June 14, 2007
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74.
Keller, J. (2006). Eliciting and addressing student misconceptions regarding the atmospheric greenhouse effect and radiative equilibrium. Unpublished doctoral dissertation, University of Arizona, Tucson, AZ.
Libarkin, J. C., & Anderson, S. W. (2005). Assessment of learning in entry-level geoscience courses: Results from the Geoscience Concept Inventory. Journal of Geoscience Education, 53(4), 394-401.
A slightly modified version of the Science Teaching Efficacy and Beliefs Instrument (STEBI) (Riggs & Enochs, 1990) is administered annually to teachers participating in Project PASS. Project PASS focuses on using inquiry, conceptual change, and self-regulated as guiding principles for secondary science instruction. Results from the STEBI are intended to monitor the development of participants’ efficacy beliefs related to teaching science using these reform-minded ideals.
The STEBI measures teacher beliefs on two subscales:
Personal Science Teaching Efficacy Belief (PSTEB) – Self-efficacy (confidence) for teaching science.
Science Teaching Outcome Expectancy (STOE) – Belief that teaching science will cause students to learn science.
Method
Annually, during the first week of the summer institute, PASS participants complete the STEBI on-site in a traditional pencil and paper form. An identical version is given every year so that it functions as a repeated measure. In addition to the subscale items, participants report their primary teaching discipline, years of teaching experience, and whether they are new employees with CCSD.
Results
Thirteen teachers were identified as having participated in all of the PASS professional development activities during years 1 and 2 and their STEBI responses were used for pre/post significance testing. A paired-samples t-test was conducted for the PSTEB and STOE subscales for summer 2005 (time 1) and summer 2007 (time 3). Mean and standard deviations for each scale are presented in tables 1&2.
Table 1
Descriptive Statistics for Personal Science Teaching Efficacy Belief (PSTEB) for teachers participating in professional development activities throughout years 1 and 2 of Project PASS.
Time Period |
N |
Mean |
Standard |
Summer 2005 |
13 |
40.62 |
3.10 |
Summer 2007 |
13 |
42.00 |
4.02 |
Table 2
Descriptive Statistics for Science Teaching Outcome Expectancy (STOE) for teachers participating in professional development activities throughout years 1 and 2 of Project PASS.
Time Period |
N |
Mean |
Standard |
Summer 2005 |
13 |
32.23 |
2.09 |
Summer 2007 |
13 |
32.92 |
3.59 |
A paired samples t-test did not show the increase in the mean scores for the PSTEB from pre (M=40.62, SD=3.10) to post (M=42.00, SD=4.02) to be statistically significant. A paired samples t-test did not show the increase in the mean scores for the STOE from pre (M=32.23, SD=2.09) to post (M=32.92, SD=3.59) to be statistically significant.
Eight of the thirteen teachers whose STEBI responses were used for pre/post significance testing were used for a subscale vs. time correlation analysis. This group was created because this sample had uniquely completed all of the professional development activities during years 1 and 2 of the project (tables 3&4). Correlation analysis identified a positive linear change for outcome expectancy over time (r=0.996) and no relationship for self-efficacy (figure 1).
Table 3
Descriptive Statistics for Personal Science Teaching Efficacy Belief (PSTEB) for participants who completed all professional development activities during years 1 and 2 of Project PASS.
Time Period |
N |
Mean |
Standard |
Summer 2005 |
8 |
40.13 |
3.00 |
Summer 2006 |
8 |
41.13 |
3.18 |
Summer 2007 |
8 |
40.38 |
2.13 |
Table 4
Descriptive Statistics for Science Teaching Outcome Expectancy (STOE) for participants who completed all professional development activities during years 1 and 2 of Project PASS.
Time Period |
N |
Mean |
Standard |
Summer 2005 |
8 |
31.50 |
2.67 |
Summer 2006 |
8 |
32.88 |
2.03 |
Summer 2007 |
8 |
33.88 |
4.22 |
Figure 1
Correlation analysis for participants who completed all professional development activities during years 1 and 2 of Project PASS.
References
Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74(6), 625-637.
Teacher Classroom Observartions
As part of the graduate course requirements for Project PASS, participating teachers are observed in their classrooms. For year one of Project PASS, teachers were required to complete the observation requirement by submitting a videotaped lesson. During subsequent years, once per semester a graduate student from UNLV visits each participant’s classroom to complete a formal observation (table 1).
Table 1
Total classroom observations completed.
Time Period |
N |
Source |
Fall 2005 |
12 |
Submitted Videotape |
Fall 2006 |
22 |
Human Observation |
Spring 2007 |
21 |
Human Observation |
Data Collection Method
As the tapes were viewed or classroom observed, a standard observation protocol was completed that coded the lesson in five-minute increments using a predetermined set of categories on the following scales: type of instruction, classroom organization, student attention, and cognitive level. Once the lesson was coded, it was also scored using the Oregon Teacher Observation Protocol (O-TOP) (Morrell et al., 2004; Wainwright, C., Flick, F., & Morrell, P.D. 2003; Wainwright et al., 2004). The O-TOP is an instrument that asks researchers to rate subjects from 0 – 4 on 10 different dimensions of teaching approaches. These dimensions include: habits of mind, metacognition, student discourse and collaboration, rigorously challenged ideas, student pre- and misconceptions, conceptual thinking, divergent thinking, interdisciplinary connections, pedagogical content knowledge, and multiple representations.
Findings
Teachers participating in PASS initiate various teaching techniques, but struggle with full implementation. Generally, an attempt to use reformed teaching methods is observed. Discussion, inquiry, group work, and student-directed learning were clearly attempted, but quickly replaced by more teacher-directed classroom activities. Whole group activities are the predominate form of classroom organization. Though teacher-centered, student engagement is often high. This group of teachers attempts to use inquiry, discussion, high-level questioning, and hands-on activities, but it is not clear if they know how to effectively deliver these reformed teaching methods. The ineffective delivery of these techniques leads to a perpetually low cognitive demand for students. In summary, the observed teachers seem competent in their subject knowledge and are cognizant of basic components of reformed pedagogy, but struggle in effective delivery.
Changes are evident for those who chose to remain with Project PASS over time and participate in the full spectrum of professional development (figure 1). With few exceptions, the third semester of observations is showing marked improvement in nearly all categories measured on the O-TOP. This is especially encouraging as categories such as metacognition, conceptual thinking, student discourse & collaboration, student pre and mis-conceptions, and interdisciplinary connections are all supporting constructs to the macrothemes addressed during the first two years of PASS.
Figure 1
O-TOP classroom observation scores for teachers who have been continuous participants in the full spectrum of professional development offered through Project PASS.
References
Morrell, P. D., Flick, L., & Wainwright, C. (2004). Reform teaching strategies used by student teachers. School Science and Mathematics Journal, 104(5), 199-213.
Wainwright, C., Flick, L., & Morrell, P.D. (2003). The development of instruments for assessment of instructional practices in standards-based teaching. The Journal of Mathematics and Science: Collaborative Explorations, 6, 1-9.
Wainwright, C., Flick, L., Morrell, P. D., & Schepige, A. (2004). Observation of reform teaching in undergraduate level mathematics and science courses. School Science and Mathematics Journal, 104(7), 322-335.