Michelle Rhee ~ Photo courtesy of Neighborhood Centers

Michelle Rhee. Credit: Neighborhood Centers

Score one for Michelle Rhee and performance pay.

A study released Wednesday of the controversial teacher evaluation system that Rhee initiated when she was chancellor of the District of Columbia Public Schools has found that both its threats of dismissal and big pay incentives worked as intended. Within its first three years, the system led to increases in the retention and the performance of effective teachers while encouraging ineffective teachers either to quit or improve.

The research, co-authored by Professors Thomas Dee of the Stanford Graduate School of Education and James Wyckoff of the University of Virginia, is one of the first studies to show a positive impact of offering more money to teachers who perform better. As they acknowledge, most of the research “raises considerable doubt about the promise of teachers’ compensation-based incentives as a lever for driving improvements in teacher performance.” Especially when the pay incentives were linked to increasing test scores alone, “it may be that teachers generally lack the willingness (or, possibly, the capacity) to respond to incentives that are linked narrowly and exclusively to test scores,” Dee and Wyckoff wrote.

Washington, D.C.’s IMPACT system also gave heavy weight – 50 percent of a teacher’s evaluation score – to valued-added test scores of students who took the district’s standardized tests. But only 17 percent of the district’s teachers taught subjects that were tested. The evaluations of the other 83 percent of teachers were based on multiple factors, including five observations by principals and master teachers of classroom management and instruction, a teacher’s impact on the school community and other measures of student achievement not involving standardized tests. Three other elements distinguished IMPACT from other pay-for-performance programs, the researchers said:

  • Strong incentives of dismissals after two straight “minimally effective” reviews and substantial monetary rewards, including first-year bonuses of up to $27,000 and permanent raises of as much as $25,000 for two consecutive “highly effective” ratings;
  • Instructional coaches to assist teachers in improving performance;
  • Recognition that the system would be neither small-scale nor temporary.

“IMPACT was not cash for test scores,” Dee said Wednesday. “It was based on multiple measurements and powerful incentives – a jump of five years on the salary schedule for those twice rated highly effective – that dwarf other programs.”

The research concentrated on those teachers who would be most likely to be motivated by IMPACT: those teachers facing the prospect of dismissal for a second consecutive “minimally effective” rating and borderline “effective” teachers motivated to become “highly effective.” The study attributed the higher rates of attrition of minimally effective teachers and higher rates of retention of highly effective teachers to IMPACT. A significant percentage of minimally effective teachers who didn’t quit saw improved evaluation scores.

“We think there should be new and different ways of assessing teachers,” Dee said. “IMPACT may not necessarily be the best way and the final word, but it does provide early important evidence.”

Dee acknowledged that the study didn’t focus on the vast majority of teachers rated effective whose jobs were not in jeopardy and may not have been motivated by IMPACT’s incentives. “For teachers square in the middle, not close to minimally or highly effective, reform might have washed over them,” he said.

Rhee resigned as chancellor of D.C. Public Schools in the fall of 2009, the first year that IMPACT went into effect, to head StudentsFirst, a Sacramento-based education nonprofit. Since 2010, 500 teachers with “ineffective” ratings on evaluations have been dismissed. There were investigations into allegations that teachers in some schools, motivated by the possibility of big bonuses or fear of dismissal, changed students’ test scores.

Dee said that most of the cheating charges surfaced a year before IMPACT appeared and that the small number of teachers identified with the potential violations were excluded from the study.

Dee and Wyckoff published “Incentives, Selection and Teacher Peformance: Evidence from IMPACT” as a working paper for the National Bureau of Economic Research.

 


Filed under: Evaluations, Featured, Reporting & Analysis, Teacher Pay, Teachers and Admin · Tags: , , ,

Comment Policy

EdSource encourages a robust debate on education issues and welcomes comments from our readers. The level of thoughtfulness of our community of readers is rare among online news sites. To preserve a civil dialogue, writers should avoid personal, gratuitous attacks and invective. Comments should be relevant to the subject of the article responded to. EdSource retains the right not to publish inappropriate and non-germaine comments.


EdSource encourages commenters to use their real names. Commenters who do decide to use a pseudonym should use it consistently.


Leave a Reply

Your email address will not be published. Required fields are marked *

 characters available

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

  1. Paul says:

    Phenomena ascribed to IMPACT almost certainly existed before. We have no way of knowing, because the study coincided with the launch of IMPACT and the researchers made no attempt to study prior conditions and draw comparisons.

    For example, the report mentions again and again that teachers resigned (i.e., left voluntarily) after receiving a “minimally effective” rating. Isn’t it natural to resign when you haven’t been successful in your job? There was a baseline level of teacher attrition before IMPACT, and presumably, some of those “leavers” were low performers.

    The report points out that the system average teacher effectiveness score increased during the three-year study. This result is only meaningful if we assume that teacher effectiveness was static or declining before IMPACT. Isn’t it natural for individuals, groups of individuals, and systems to strive for improvement? Were DC teachers stupid, lazy and ineffective until Michelle Rhee changed the evaluation system?

    My last criticism is that the study presumes that IMPACT teacher effectiveness scores are a valid measure of teacher effectiveness. I’ll address only one part of that big debate.

    As is fashionable, the authors point out that old teacher evaluation systems were binary, and that most teachers received “satisfactory” ratings. The authors give the percentages of “minimally effective”, “effective” and “highly effective” teachers under IMPACT. They also point out that observations play an important part of a teacher’s score.

    As it was politically unacceptable to continue rating almost all teachers “satisfactory”, what alternative guidance was handed down to principals and peer evaluators under IMPACT? IMPACT was designed to classify more teachers as “minimally effective”. Presumably, Michelle Rhee punished evaluators who granted too many “effective” ratings. A motive to classify more teachers as ineffective does not make more teachers ineffective in fact.

    The authors don’t seem to wonder whether “satisfactory” might have meant just that. Is there any profession in which a sizable percentage of the practitioners are prima facie “unsatisfactory”?

    Regarding the comments about medicine, doctors comprise a self-regulated professional group. Doctors elect their own leaders to licensing and discipline bodies. Teachers have never had control of their licensing or discipline bodies. Moreover, with exception of doctors in the Veterans Administration, in the former state mental health hospitals, and in municipal public hospitals and public health departments, the phenomenon of working for a corporation was unheard of before World War II and remained uncommon until the rise of HMOs in the 1980s. Direct self-supervision, i.e., standalone private practice, is still possible for doctors, where it has never been possible for public school teachers.

  2. Christopher says:

    Kaiser. Doctors have admin evaluating them. The state doesnt do it (although there are mechanisms there for really bad actors) because the hospitals arent public institutions. Schools are public and therefore governed by state/local district board oversight. Does this really need to be outlined?

    Point is, name me a profession — especially one as important as teachers — where employees are not evaluated and to some degree held accountable to results.

    If teachers do not have this kind of oversight in CA. The law and local unions preclude it. Why do you think we are having this debate?

    1. Gary Ravani says:

      “If teachers do not have this kind of oversight in CA. The law and local unions preclude it. Why do you think we are having this debate?”

      Really? So what is the Stull Bill about? Every LEA in CA has a contract with evaluation likely as a part of it. In many, if not most, cases evaluation is based on the CA Standards for the Teaching Profession. In what way is evaluation “precluded” by anyone?

  3. Christopher says:

    My doctor gets evaluated on performance/results. Not sure where you get healthcare. :D

    1. el says:

      Um, the United States! Who in your universe does performance evaluations of doctors in private practice?

      Teachers always have administrative supervisors who have oversight. Doctors frequently are their own oversight. The state does not come in and insist that a set of numbers be met for doctors.

    2. Ken Curtis says:

      Remember the old line?: Mrs. Cohen, your check came back.
      Mrs. Cohen: So did my arthritis.

  4. Christopher says:

    Hannah K.: What if your doctor told you “We are the medical experts, so we need no performance evaluations nor accountability.” Would that make you feel comfortable next time you showed up for a check-up?

    And get the “meddling parents” out of the classroom? Seriously? Teachers work for the parents. They have a right to know what their child is learning and how well it is being taught.

    All professions have evaluations — teachers are humans too and should not be immune from accountability.

    1. el says:

      Um… last time I looked, that is how American medicine works. :-)

    2. Sonia says:

      Agree with Cristopher. I cannot believe that Hannah K is an educator. People that thinks that way about “their” job, should be dismissed.

      Hanna K: Teachers are paid by the taxpayer money, which are 9 out of 10 times the PARENTS! Children do not need ignorant educators. Also, teaching is a profession,such as Medicine and Law. If the person chooses to be an educator, that person made a conscience choice about their profession, pay rate and expectations.

  5. Gary Ravani says:

    Allow me to begin with a quote, from this site, from a Louis Freedberg on John Merrow:

    ‘What matters much more is what she (Rhee) failed to accomplish in Washington. She espoused a certain approach to reforming failing schools, a path that she and her successor have followed for six years, and that approach has not worked. That’s the central point: Rhee’s ‘scorched earth’ approach of fear, intimidation and reliance on standardized tests scores to judge (and fire) teachers and principals does not lead to improved schools, educational opportunities, graduation rates or any of the other goals that she presumably embraces.s inquiries into the cheating scandal in the DC schools.”

    The authors of the study dismiss these critical cheating issues in a pretty cavalier way. I think the Merrow quote puts the study into an appropriate context. The study is still based on tests we know to be narrow and superficial, and uses VAM that every legitimate educational researcher has thoroughly debunked. The concept of “merit pay,” for despite the assertions of the studies authors that is what this is, has repeatedly been tried and abandoned and, again, debunked.

    1. el says:

      To some extent the assertion is a tautology, and obvious. “We identify these teachers as effective. We pay them enormous bonuses. We find that the people we pay these bonuses to are more likely to stay than those passed over for bonuses.”

      There is one rather enormous flaw though… nowhere is it actually validated that the “highly effective” teachers were accurately identified… nor does it validate that there was a newly positive effect for students.

      So it kind of misses the point of why we do all this.

      (and that’s aside from the elephant of scalability.)

      At the end of the day, kind of glossed over in the article is that Rhee resigned because her boss got trounced in an election, an election where her governance of the schools was a key issue. The community rejected her and wanted her gone. I think anyone looking to hire her, to enlist her organization, or to take her advice consider that extremely carefully before proceeding.

  6. RelDog says:

    There are no objective evaluators for these issues.

    Stanford is not a valid objective source. They are a conservative think tank that gives degrees. The same could be said for others like Cal which is a liberal think tank that gives degrees.

    We need a verifiably objective non-partisan source to make these evaluations. Good luck with that!

  7. el says:

    That said, though, I’ll highlight this commentary on the current status of Value-Added analysis and its reliability in measuring teacher effectiveness:

    http://schoolfinance101.wordpress.com/2013/10/16/the-value-added-growth-score-train-wreck-is-here/

    “I looked at New York City value-added findings when the teacher data were released a few years back. I would argue that the New York City model is probably better than most I’ve seen thus far and its technical documentation reveals more thorough attempts to resolve common concerns about bias. Yet, the model, by my cursory analysis still fails to produce sufficiently high quality information for confidently judging teacher effectiveness.

    Among other things, I found that only in the most recent year, were the year over year correlations even modest, and the numbers of teachers in the top 20% for multiple years running astoundingly low.”

  8. el says:

    If the State will fund us for an additional $25,000 a year for every teacher that we honestly evaluate as highly effective, with the goal that every teacher should be honestly so rated, sign us up.

    I’ll note in our district this would be a 50% raise for our median teacher.

  9. Hannah Katz says:

    I find the Rhee evaluation offensive to professional educators. I heard someone say that accountability is something no one enjoys, but which everyone needs. I disagree. We are the education experts, so we need no evaluations nor accountability. All the education profession needs is more generous funding. Lots more. Then let each teacher do what she feels is best and LEAVE US ALONE! Get the meddling parents and other non-professionals out of my classroom!

    1. CCB says:

      Forgive me, but are you aware of how bizarre this request is? I have huge respect for teachers and the work they do–my parents were teachers; I grew up surrounded by the teacher community and substitute taught myself. I will be the first to say that they are underpaid and under-appreciated in many contexts. But there is no profession in the world in which one isn’t subject to some form of either accountability or else market economics. If your product sucks, people will know, and no one will buy it. Or if you are a lousy employee and fail to achieve *reasonable* goals, you will get fired. I don’t think it’s crazy to think that teachers, who despite lousy pay still have enormous flexibility within the classroom AND comparative job security, should get measured on some level and held to some standard of accountability. We can debate what those measures and standards are, to be sure. But to ask for complete freedom from any oversight or consequences, when you’re teaching the next generation of citizens? Give me a break.

      1. Jimmy B. says:

        Agreed. Teachers should be evaluated. The question is how. You said there is no profession in the world in which one isn’t subject to some form of accountability right? So let’s evaluate a doctor on how well his patient sticks to his diet. Ok? Or let’s base a lawyer’s performance on whether his client gets thrown back in jail. I suggest that evaluating a teacher on how well his (or her) student performs on standardized tests is very similar when you consider the socio-economic forces in play here. Simply put, if a teacher’s students come from modest means and go to school hungry and/or aren’t confident in their living arrangements? Then to suggest that their teacher has a fighting chance of teaching them something that they’ll remember and be able to use during a standardized test? That’s as ludicrous as my suggestions for evaluating doctors and lawyers.

        1. Norma K says:

          AMEN! My daughter is a teacher with a Masters Degree. This is her 23rd yr. of teaching. She has taught every elementary grade…K-5, English as a special language, has been called in on many evaluations of students and also has had many student teachers. She is a single mom. She has one biological son and has adopted 2 more little boys that were foster children.Her own boys are in parocihal school, with alot of financial help, I might add. They are there because she knows the time a GOOD public school teacher has to devote to them. This yr. she has 28 kindergardners, a new principal that had been in the classroom for only 2yrs. Believe me, she is not there for the acolades(she has plenty of them) She is not there for the money! She is there to be treated as a professional, respected by parents, and the love of children. If we told a dentist…Your career will be based on how many children return with no cavaties. Do you honestly believe he can control that? Until we get the parents to care and be responsible no amount of money, testing etc. will help. We need to restore the family first. Believe me in parochical school the parents are involved! They have some money invested and they don’t always have the BEST of the Best teachers but those kids succeed! How many years did Ms. Rhee spend in the public school classroom?

  10. navigio says:

    Regardless of the problem of looking at only a subset of teachers in the study, it should be noted that a $25k pay increase is over 35% of the average teacher salary in CA. And of course the goal is for every teacher to be effective. If memory serves, I think MR even had to secure private funding to fund this program.