Pressure builds to change how California measures student progress on state tests

Credit: Alison Yin / EdSource (2014)

Credit: Alison Yin / EdSource (2014)

A group of education professors and dozens of student advocacy groups are urging California education officials to switch to a method that most states use to rate student progress on standardized tests. They say it will more accurately measure and compare schools’ performance than what they see as the flawed system the state uses now.

However, State Superintendent of Public Instruction Tom Torlakson is recommending that the State Board of Education stick for now with the current method, which compares the most recent scores to the scores of students who took the test the previous year. Torlakson and staff of the California Department of Education say they want to be sure they’re choosing the right alternative to a two-year-old school accountability system that parents and teachers are just getting used to. The State Board of Education may decide what to do at its meeting on Wednesday. (See Item 1, attachment 1, of the board’s agenda for the staff recommendation and links to related documents.)

California is required under state and federal laws to inform the public how students are performing on state standardized tests in reading and math each year, and whether they’re showing improvement. At issue is the best way to describe a school’s impact on student improvement for state accountability purposes, how much progress their children are making compared with students in other schools and, for parents in particular, whether their kids are on track to be proficient in math and reading.

To meet this requirement, California has adopted a formula that dozens of leading academics say may actually paint an inaccurate picture of how a school is serving its students.

“California has chosen basically the worst growth measure you can possibly choose,” said Morgan Polikoff, a professor of education at the University of Southern California. He wrote a letter to the State Board of Education urging California to use a method that better demonstrates a school’s impact on a student. It was signed by more than a dozen education scholars, including those from UC Davis, Harvard University and the University of Washington.

The current measure takes last year’s average scores by grade and subtracts it from this year’s average scores by grade to derive a schoolwide average. The difference becomes a key factor on the California School Dashboard, the system that creates color-coded ratings for schools, districts and student ethnic, racial and demographic groups.

“That doesn’t work for a lot of reasons,” Polikoff said. One reason is that this model fails to consider student mobility. Particularly in high-poverty neighborhoods, large numbers of students move, so a cohort of students by grade is often different from the year before, undermining the validity of the comparison. Paul Warren, research associate for the Public Policy Institute of California, reached a similar conclusion in a report released late last month.

Instead, more than 40 states track the growth in individual students’ scores from year to year on standardized tests like Smarter Balanced, which students take in grades 3 to 8 and again in grade 11.

“Growth measures, on the other hand, tell us how much each individual student is learning and how much of that learning can be attributed to the school. Many researchers agree that growth is a superior measure for purposes of measuring how well schools are doing,” wrote three dozen student equity and advocacy organizations, including Oakland-based Education Trust-West and Teach Plus California, a nonprofit in Los Angeles.

In fact, the ability to support a growth model was one of the criteria that the state board used in choosing Smarter Balanced as its math and English language arts test, the nonprofit Children Now noted in its letter to the board.

State board shows interest

Many states adopted a growth model to gauge student improvement when they received a waiver from sanctions imposed by the federal No Child Left Behind law during the final years of the Obama administration. They had to agree to use the results of student test scores as a factor in evaluating teacher performance, although some states either never fully implemented that aspect or subsequently abandoned the practice. Teachers unions opposed the idea, and many academics criticized using student test results as a statistically unreliable basis for evaluating teacher effectiveness. Gov. Jerry Brown and the state board opposed the idea and never sought a waiver.

Two years ago, the state board indicated it was interested in switching to some version of a growth model for the state’s new accountability system and asked ETS, a company that has done assessment work for the state, and state education department staff to look into some of the variations. It has done so and took the charge seriously, said Cindy Kazanis, the director of the Analysis, Measurement and Accountability Division of the California Department of Education.

A growth model can be used to predict an average school’s yearly growth, using average scores of different student groups — low-income students, English learners, ethnic and racial groups. The model can then be used to rank schools’ performance, on a scale of 1 to 100, based on how far below or above average their individual students actually scored. A half-dozen California school districts, including Los Angeles, San Francisco, Oakland and Long Beach, called the CORE districts, have been using this method for five years to informally guide school improvement, though not for accountability purposes with consequences.

Staff of the education department studied a slightly different growth method, which proponents say reveals how much each individual student has learned in a year and how much of that learning can be attributed to the school. “Parents and educators alike want the most accurate information on student learning, they want to know the truth about how their schools are doing, and they want to be held accountable based on fair measures,” the advocacy groups said in their letter.

But after applying two years of Smarter Balanced test data to one particular growth model, the department concluded for technical and policy reasons the state should hold off moving ahead this year — and perhaps consider looking at other forms of growth models. The state should examine one or more additional years of data to see if ranking schools based on a growth model is valid and reliable, the department said. Edward Haertel, an emeritus professor of education at Stanford University and a longtime member of a technical advisory group for the department, confirmed that he and others unanimously agreed with the department’s conclusion.

And state staff said that a system that simply compares schools’ performance omits what the current dashboard provides: It doesn’t tell parents and teachers how much a school’s score increased toward the goal of reaching proficiency in math or reading. And comparing schools’ growth also doesn’t tell you if or how quickly the achievement gap among student groups is closing, said David Sapp, deputy policy director and assistant legal counsel of the state board. That, too, is what teachers and principals want to know, he said.

Samantha Tran, senior managing director for education for Children Now, said growth models can provide that information, too; it’s not either-or. The state board and staff must be clearer on what they’re looking for, she said.

What is clear, Polikoff wrote, is that the state’s current system of rating school performance based on comparing test scores by different cohorts of students is “unacceptable.”

To get more reports like this one, click here to sign up for EdSource’s no-cost daily email on latest developments in education.