Home Articles Reader Opinion Editorial Book Reviews Discussion Writers Guide About TCRecord
transparent 13

The Instructional Challenge in Improving Teaching Quality: Lessons From a Classroom Observation Protocol

by Drew Gitomer, Courtney Bell, Yi Qi, Daniel McCaffrey, Bridget K. Hamre & Robert C. Pianta - 2014

Background/Context: Teacher evaluation is a major policy initiative intended to improve the quality of classroom instruction. This study documents a fundamental challenge to using teacher evaluation to improve teaching and learning.

Purpose: Using an observation instrument (CLASS-S), we evaluate evidence on different aspects of instructional practice in algebra classrooms to consider how much scores vary, how well observers are able to judge practice, and how well teachers are able to evaluate their own practice.

Participants: The study includes 82 Algebra I teachers in middle and high schools. Five observers completed almost all observations.

Research Design: Each classroom was observed 4–5 times over the school year. Each observation was coded and scored live and by video. All videos were coded by two independent observers, as were 36% of the live observations. Observers assigned scores to each of 10 dimensions. Observer scores were also compared with master coders for a subset of videos. Participating teachers also completed a self-report instrument (CLASS-T) to assess their own skills on dimensions of CLASS-S.

Data Collection and Analysis: For each lesson, data were aggregated into three domain scores, Emotional Support, Classroom Organization, and Instructional Support, and then averaged across lessons to create scores for each classroom.

Findings/Results: Classroom Observation scores fell in the high range of the protocol. Scores for Emotional Support were in the midlevel range, and the lowest scores were for Instructional Support. Scores for each domain were clustered in narrow ranges. Observers were more consistent over time and agreed more when judging Classroom Organization than the other two domains. Teacher ratings of their own strengths and weaknesses were positively related to observation scores for Classroom Organization and unrelated to observation scores for Instructional Support.

Conclusions/Recommendations: This study identifies a critical challenge for teacher evaluation policy if it is to improve teaching and learning. Aspects of teaching and learning in the observation protocol that appear most in need of improvement are those that are the hardest for observers to agree on, and teachers and external observers view most differently. Reliability is a marker of common understanding about important constructs and observation protocols are intended to provide a common language and structure to inform teaching practice. This study suggests the need to focus our efforts on the instructional and interactional aspects of classrooms through shared conversations and clear images of what teaching quality looks like.

To view the full-text for this article you must be signed-in with the appropriate membership. Please review your options below:

Store a cookie on my computer that will allow me to skip this sign-in in the future.
Send me my password -- I can't remember it
Purchase this Article
Purchase The Instructional Challenge in Improving Teaching Quality: Lessons From a Classroom Observation Protocol
Individual-Resource passes allow you to purchase access to resources one resource at a time. There are no recurring fees.
Become a Member
Online Access
With this membership you receive online access to all of TCRecord's content. The introductory rate of $25 is available for a limited time.
Print and Online Access
With this membership you receive the print journal and free online access to all of TCRecord's content.

Cite This Article as: Teachers College Record Volume 116 Number 6, 2014, p. 1-32
https://www.tcrecord.org ID Number: 17460, Date Accessed: 1/21/2021 6:54:12 PM

Purchase Reprint Rights for this article or review
Article Tools
Related Articles

Related Discussion
Post a Comment | Read All

About the Author
  • Drew Gitomer
    Rutgers University
    E-mail Author
    DREW GITOMER is the Rose and Nicholas DeMarzo Chair in Education at the Rutgers University Graduate School of Education. His research centers on the assessment and evaluation of teaching and related policy issues in teaching and teacher education. His current work focuses on a range of constructs that are related to teaching quality, including the quality of classroom interactions, teacher knowledge, teacher beliefs, and student achievement. He is currently carrying out validity studies of a variety of measures, including classroom observation protocols, classroom assignment protocols, and new measures of teacher knowledge. Through this work, he strives to make progress on understanding the contextual factors that influence the quality of teaching that is observed. Recent publications include Gitomer, D. H., & Bell, C. A. (2013). Evaluating teaching and teachers. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology (Vol. 3, pp. 415–444). Washington, DC: American Psychological Association.
  • Courtney Bell
    Educational Testing Service
    E-mail Author
    COURTNEY BELL is a senior research scientist in the Understanding Teaching Quality Center at ETS. She completed her undergraduate degree in chemistry at Dartmouth College and her PhD at Michigan State University. A former high school science teacher and teacher educator, Courtney’s work looks across actors in the education system to better understand the intersections of policy and practice. Her prior work looked at school choice policies from the perspective of the parents. Her current studies use mixed methods to analyze how principals learn to use observation protocols and the validity of measures of teaching quality. Courtney’s recent publications include Bell, C. A., Gitomer, D. H., McCaffrey, D., Hamre, B., Pianta, R., & Qi, Y. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 1–26.
  • Yi Qi
    Educational Testing Service
    E-mail Author
    YI QI is a research associate at Educational Testing Service. Her current work centers around policy and evaluation issues related to teaching quality and how best to measure teaching practice, particularly using classroom observations. Her research interests also include professional development and program evaluation. Recent publications include (with Courtney Bell, Drew Gitomer, Daniel McCaffrey, Bridget Hamre, and Robert Pianta) “An Argument Approach to Observation Protocol Validity,” published in Educational Assessment.
  • Daniel McCaffrey
    Educational Testing Service
    E-mail Author
    DANIEL MCCAFFREY is a Principal Research Scientist at ETS. His research interests include teacher performance measurement using value -added and other measures and statistical methods related to causal modeling. Recent publications include (with Jodi Casabianca and others) “Effect of Observation Mode on Measures of Secondary Mathematics Teaching,” to be published in Educational and Psychological Measurement, and (with Kun Yuan and others) “Incentive Pay Programs Do Not Affect Teacher Motivation or Reported Practices: Results From Three Randomized Studies,” published in Educational Evaluation and Policy Analysis.
  • Bridget Hamre
    University of Virginia
    BRIDGET K. HAMRE is a Research Associate Professor at the Center for Advanced Study of Teaching and Learning at the University of Virginia. Her areas of expertise include student-teacher relationships and classroom processes that promote positive academic and social development for children. Recent publications include (a) Hamre, B. K., Pianta, R. C., Downer, J. T., Decoster, J., Jones, S., Brown, J., . . . Hamagami, A. (2013). Teaching through interactions: Testing a developmental framework of effective teaching in over 4,000 classrooms. The Elementary School Journal, 113(4), 461-487; and (b) Hamre, B. K., Pianta, R. C., Burchinal, M. Field, S., Locasale-Crouch, J. L., Downer, J. T., . . . Scott-Little, C. (2012). A course on effective teacher-child interactions: Effects on teacher beliefs, knowledge, and observed practice. American Education Research Journal, 49(1), 88–123.
  • Robert Pianta
    University of Virginia
    E-mail Author
    ROBERT C. PIANTA is the dean of the Curry School of Education, the Novartis U.S. Foundation Professor of Education, and a Professor of Psychology at the University of Virginia, where he is also founding director of the University of Virginia Center for Advanced Study of Teaching and Learning. Pianta's research and policy interests focus on the measurement and production of effective teaching in classrooms from preschool to high school. Pianta has published more than 200 scholarly papers and is lead author on several influential books related to early childhood and elementary education. He is the senior author and developer of the Classroom Assessment Scoring System, a method for assessing teacher/classroom quality used in many district-, state-, and national-level applications. Pianta’s assessments of teacher effectiveness are the national standard for Head Start classrooms and are included in the Gates Foundation’s Measures of Effective Teaching study. His recent publications include (a) Hamre, B. K., Pianta, R. C., Burchinal, M., Field, S., LoCasale-Crouch, J. L., Downer, J. T., . . . Scott-Little, C. (2012). A course on effective teacher–child interactions: Effects on teacher beliefs, knowledge, and observed practice. American Educational Research Journal, 49(1), 88–123. doi:10.3102/0002831211434596; and (b) Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333(6045), 1034–1037. doi:10.1126/science.1207998
Member Center
In Print
This Month's Issue