Citing methodology flaws, the State Board of Education on Wednesday revised criteria for rating performance on standardized test scores on the new color-coded California School Dashboard. The unanimous decision will reduce the number of districts and schools rated red, the lowest performing of the five color categories, but board members and state administrators insisted that was not the motivating factor (see previous story).
“It would be worse to do nothing; that would undermine credibility (of the system) and create more volatility” in test score ratings, said board member Ilene Straus, who monitors school improvement and accountability issues for the board.
The fix — which was proposed by a technical group of advisers to the California State Department of Education — was in response to what would otherwise be a big fluctuation in school and district color designations on the “academic” indicator, which includes test scores, on the school dashboard.
The next version is due out at the end of this month. Districts’ and student subgroups’ performance on the dashboard will determine which districts will receive assistance this year from their local county offices of education.
The technical problem is based on the department’s assumption, in creating the dashboard criteria, that students would do better than they did on the most recent Smarter Balanced tests in math and English language arts, which they took in the spring. But scores on average were flat statewide, and in many schools and districts, they declined. Without changing the criteria for setting dashboard colors, the number of school districts and schools designated as red would double, as would the number in orange, the second lowest of the five performance levels.
The impact is unclear, because districts and student subgroups would have to perform in the red for at least one other indicator other than test scores — suspension rates, graduation rates and the performance of English learners — to be designated as needing help. State officials said they haven’t calculated how many districts would qualify either overall or for one or more of their low-performing subgroups, such as students with disabilities.
The changes in parameters that the board unanimously approved will reduce the number of districts and schools rated red, while significantly increasing the proportion rated orange. The combined percentages of red and orange districts actually would be greater than under the current system — proof, said board member Sue Burr, that the intention “was not to game the system.”
Board member Feliza Ortiz-Licon agreed volatility had to be fixed and there was no intent to “fudge the data,” but the state’s memo to the board “sounded like” there was an effort to “mitigate the red numbers.” And, she said, there is no data laying out what the effect on districts and minority students would be.
Districts and subgroups may be eligible for assistance if results in both math and English arts are rated red, or if one subject is red and the other is orange. There is a good chance, however, that reducing the number of districts in the red performance level by nearly two-thirds under the new criteria will cut the number of districts and subgroups assigned help.
Other changes recommended by the Technical Design Group will lead to designate slightly more districts and schools “blue,” the highest performing level. More schools will also be “green,” the second highest performance category, which is the state’s goal, but far fewer district in the middle “yellow” level, where schools are slightly to moderately below grade level with no improvement from the previous year.
In a letter to the board, 14 civil rights and student advocacy groups acknowledge the need for some changes in criteria, but objected to re-designating low-performing districts whose scores dropped very significantly from red to orange.
And they objected to the hurried process and lack of public access to meetings of the technical advisory group, which meets privately and doesn’t make its minutes public.
“I feel rushed to make a big decision,” Ortiz-Licon said. “We may come back next year to the same conversation.”
Chances are board members will.
“This may not be the last time we make adjustments. We are going to get there but we are not there yet,” Straus said.
Cindy Kazanis, director of the department’s Assessment Development & Administration Division, said state officials made it clear when they set the initial academic indicator criteria that they would likely have to tweak it. It may take several years of results to produce consistent, predictable criteria for growth in test scores, she said.
On a related issue, Deputy State Superintendent Keric Ashley defended the reliability of the Smarter Balanced test itself. Some experts and advocacy groups suggested the lower test scores, particularly in English language arts, might point to a problem with the spring 2017 test itself, not California students’ performance on it. Students in all 14 states taking Smarter Balanced tests either declined or, like California, were flat, which is unusual.
But Ashley told the board that the executive committee of Smarter Balanced, which he serves on, turned to experts to look into that possibility. An initial study by the American Institutes of Research found no difference in the quality of the test questions or in how the test was conducted, compared with the year before, he said. That study has not been released, but the final report, due in February, will be public, he said.
“There is nothing wrong with raising questions, but it is not responsible to speculate” on the reasons why the scores were what they were, he said.
Source: California Department of Education
This chart shows the distribution of school districts by performance color for the academic indicator under the original criteria that the board adopted. It covers the 2016 and 2017 test results in English language arts in math. The number of districts in red would have doubled from 81 to 169 and nearly doubled in orange, had the board not reset the criteria.
Source: California Department of Education
This chart shows the distribution of districts under the new criteria for the academic indicator of the dashboard for English language arts and math, retroactively for 2016 and for the 2017 testing last spring. There will be fewer red and yellow districts, and a lot more orange and green districts than under the current criteria, shown in the chart above.
To get more reports like this one, click here to sign up for EdSource’s no-cost daily email on latest developments in education.