California is on the verge of finalizing what leading educators believe is the most ambitious attempt in the nation to use multiple dimensions to measure how well – or poorly – a school or district is doing, rather than focusing primarily on test scores.

“All across the country people are paying attention to what California is doing,” Linda Darling-Hammond, the president of the Learning Policy Institute, said at a recent California School Boards Association conference.

The deadline for approving the plan is barely two months away, as required by a state law championed by Gov. Jerry Brown that implemented the Local Control Funding Formula, which reformed both the way schools are funded and how progress will be measured.

The state’s goal has been to come up with a system that will require schools and districts to measure how they are doing on eight “priority areas ranging from test scores to less definable measures such as school climate.

But getting agreement on the double challenge of coming up with a more multidimensional system of accountability, and along with it a way to display school and district performance that is easy to understand, is proving to be a challenging undertaking, to say the least. 

One reason is that California is attempting to break away from the dominant way schools and districts have been assessed for at least a generation. Another is coming up with a system that meets the needs of a state with a larger and more diverse student population than in most countries, let alone states, as well as an array of education constituencies and interests. 

The change has been led by Brown, who has been critical of excessive testing throughout his administration. He is backed by the Legislature, the State Board of Education (whose members he appoints), as well as other key education constituencies such as the state’s leading teachers unions and professional organizations representing school administrators and school board members.

The old system had the advantage of being relatively easy to understand, but fell short because it did not give a complete picture of many aspects of what goes on in a school, and where a school was doing well and where it needed to improve.

“People want simplicity (that test scores provide) but simplicity hasn’t gotten us very far,” said former State Superintendent of Public Instruction Bill Honig. “We really have to look at the breadth of what is going on (in a school or district).”

“All across the country people are paying attention to what California is doing,” said Linda Darling-Hammond, president of the Learning Policy Institute.

Even President Barack Obama, whose administration for most of his presidency has pushed test scores as the primary way to measure school success, has begun to advocate for a less test-heavy system.

“Tests should be just one source of information used alongside classroom surveys and other factors to give us an all-around look at how our students and our schools are doing,” he said in a video posted on Facebook last fall. “Because learning is about so much more than just filling in the bubble.”

In fact, the Every Student Succeeds Act he signed last December will also require states to measure school and district performance on more than just test scores. In addition to scores on math and English, states will have to include graduation rates, progress of English learners, and one other “non-academic” indicator of school quality that states would be free to choose on their own.

California, however, is about to adopt an accountability system that has even more indicators of progress than will be required by the federal law. So far the state has identified seven indicators of performance, but the list may be expanded to 10 later on.

At its most recent meeting earlier this month, the State Board of Education reviewed a prototype chart that summarizes performance on each of these indicators. Progress, or lack of it, is shown by a range of colors, with red and orange on the low end to green and blue at the high end.

But judging from the reactions of board members, and public comments from close to 100 advocates, parents and others at the meeting, a good deal of work still has to be done to come up with a chart that will be viewed as significantly better — and at least as understandable — as the one that it is replacing.

Some board members, for example,  said that the multicolored design was too difficult for parents to understand.  Bruce Holaday said the state needs a graphic designer to make the display more understandable. “This is not parent user-friendly at this point,” he said. “

Fellow board member Trish Williams said she preferred an earlier version of the chart, which had bolder colors including one where “red was red.” That would signal more clearly to parents areas of “real concern” in a school or district, she said.

Patricia Rucker said she liked the chart, but said that it needed a more complete “narrative explanation” so that it is more understandable.

The multicolored chart also includes an “equity report” showing which student subgroups, based on racial and ethnic background, income levels, and so on, are lagging behind. Holaday suggested the equity report should also show subgroups that are doing well.

Taking a contrary view was board member Feliza Ortiz-Licon said Holaday’s suggestion would dilute the purpose of the new system, which is to clearly show achievement gaps among racial and ethnic student subgroups.

A new “College and Career Indicator” designed to rank how well schools are preparing students for college and careers also prompted a cascade of criticism. Several  board members felt that it was skewed too much toward measuring progress of college-bound students, and not sufficiently on those pursuing vocationally oriented careers.

Keric Ashley, the state’s deputy superintendent of public instruction, explained that the chart is a work in progress that could be revised as needed going forward. “This is a good starting point,” he said. “This is not supposed to be in place for a decade.” 

After hearing these and numerous other concerns, the State Board gave the California Department of Education the green light to move ahead with the new system and come back with  an updated design for its approval.

California Department of Education staff now have until the board’s next meeting on Sept. 8 to respond to the concerns raised at the meeting as  best they can.  

An editorial in the Los Angeles Times this week expressed extreme doubts that it would be possible. “These charts are unadoptable,” the editorial read. “The board must start over and create a simpler, clearer, more rigorous way to measure achievement.”

The State Board is required by law (Assembly Bill 104) to approve a more comprehensive assessment system for schools and districts by Oct. 1. But it appears to have more flexibility as to when it needs to adopt a chart to display that system.

Former state schools chief Honig believes that despite its imperfections, what California is coming up with is far better than the one-dimensional system it has used in the past. Test scores, he said, serve merely as a “warning light” showing schools and districts that they may have a problem. But test scores don’t tell them what the underlying issues are that produce those scores.

“Test scores basically help start the conversation, but they don’t tell you what is going on,” he said. “We might not have a perfect measure yet, but at least we are putting it on the table, so people can start looking at whether they are doing a good job.”

Said Darling-Hammond in her speech to school board members,  “This is something to be really proud of.”

SHARE ARTICLE

Comments (5)

Leave a Comment

Your email address will not be published. Required fields are marked *

Comments Policy

The goal of the comments section on EdSource is to facilitate thoughtful conversation about content published on our website. Click here for EdSource's Comments Policy.

  1. Chris Reed 11 months ago11 months ago

    Kind of incredible that this long look at what's being done ignores the theory offered by Dan Walters and others that the whole goal of this exercise is to reduce scrutiny by intentionally setting up a confusing system. http://www.sacbee.com/news/politics-government/politics-columns-blogs/dan-walters/article91842382.html This is consistent with this website's refusal to conclude the obvious about the Local Control Funding Formula being hijacked. Its original stated purpose was to specfically help English learners and foster students. Two years after its passage, Superintendent … Read More

    Kind of incredible that this long look at what’s being done ignores the theory offered by Dan Walters and others that the whole goal of this exercise is to reduce scrutiny by intentionally setting up a confusing system.

    http://www.sacbee.com/news/politics-government/politics-columns-blogs/dan-walters/article91842382.html

    This is consistent with this website’s refusal to conclude the obvious about the Local Control Funding Formula being hijacked. Its original stated purpose was to specfically help English learners and foster students. Two years after its passage, Superintendent of Public Instruction Tom Torlakson, with the blessing of the State Board of Education, said the money could go to raises for teachers.

    Is this site interested in hard truths? Or in selling the education establishment’s conventional wisdom?

    Replies

    • John Fensterwald 11 months ago11 months ago

      Chris: We always welcome your criticisms, though let's not confuse columnists' "theories" with "hard truths," and let's hope that those presenting "hard truths" base them on "hard facts." You may not like the multiple measures that the State Board has selected and you're not alone in criticizing the draft report card (emphasis on draft) that the board discussed this month, but let's not forget the context. • It's not just undefined "professional educators" who agree that … Read More

      Chris: We always welcome your criticisms, though let’s not confuse columnists’ “theories” with “hard truths,” and let’s hope that those presenting “hard truths” base them on “hard facts.”

      You may not like the multiple measures that the State Board has selected and you’re not alone in criticizing the draft report card (emphasis on draft) that the board discussed this month, but let’s not forget the context.

      • It’s not just undefined “professional educators” who agree that the use of test scores alone has given a misleading picture of school achievement and caused harm, particularly in low-income schools, where art, history and science were sacrificed to gin up math and reading scores. Hundreds, soon verging on thousands, of students and parents have trudged to Sacramento to demand that hard-to-quantify measures of school climate, parent engagement and access to equitable resources be included among the indicators that measure a school’s performance. They want to highlight the conditions and lack of opportunities in their schools that contribute to poor test scores. And, since the goal of schools is to produce students who are prepared for college and/or a career, test scores won’t tell you enough.
      • The Legislature and Gov. Jerry Brown, in creating the Local Control Funding Formula, were rebelling against the test-scores-only measure of performance under the No Child Left Behind law. They created no fewer than 8 priorities, of which student achievement is only one, to evaluate a school district. Conditions of learning, local performance goals, student engagement, delivery of academic standards, are some of the others. The law mandates that the state board create performance standards, such as a graduation rate all schools should aim for, and measure all schools, districts and subgroups in all eight priorities.
      • Congress, in passing the Every Student Succeeds Act, also is requiring that states use factors other than test scores to determine which schools must receive state assistance. They are the high school graduation rate, the rate in which English learners become proficient in English and an additional non-academic factor. Whether all of these factors must be combined to produce a single ranking or rating is an ongoing debate among conservatives and liberals.
      • Some of the same folks who are complaining about the “indecipherable” display presented to the board are also urging it to add on more indicators and the results of student subgroups to the summary display. I don’t know how the board plans to resolve that cognitive dissonance but I wouldn’t call you an apologist if you acknowledged the board’s dilemma.
      • The Local Control Funding Formula has a dual goal: to create a new school accountability system and to establish a more equitable distribution of funding. It’s a leap of logic, not a hard truth, to claim that a multi-dimensional and more complex accountability system is a deliberate effort to obfuscate efforts to shift money intended for low-income students and English learners to general pay raises for teachers, as you imply.
      • I doubt you would have learned about the Torlakson letter you refer to and the guidance that the department issued before that, had EdSource not reported both and the criticisms they spurred. I encourage you to read Torlakson’s letter closely. While it walks back on the original letter, it still places a high bar on when supplemental and concentration money can be used for general raises -– unless the result best serves high-needs students or increases or expands services and programs. Using financial incentives to hire and retain teachers may be a permissible use. But if school boards aren’t justifying the raises in their LCAPs and if county boards of education aren’t calling them on it, the fault lies with enforcement not the guidelines. The problem with local control is that no one really has a good handle on what’s happening in many of the state’s 1,000 districts.
      • EdSource has reported extensively on the reports by Public Advocates, the ACLU, Education Trust-West and Californians Together that have uncovered inadequate reporting or improper uses of supplemental and concentration money. EdSource reported one particularly problematic part of the law that could lead to abuse (Given the state board’s disinclination so far to address the issue, I admit the headline, which I wrote, may prove to be wildly optimistic).
      • You are not alone in criticizing the difficulty in tracking how money under the Local Control Funding Formula is being spent and in suspecting that the funding law is not fulfilling its promise. We’re always open to suggestions on how we can improve our reporting on complex issues.

      My apologies to you and readers for violating our guidance to keep comments brief.

  2. Jerry kaplan 11 months ago11 months ago

    I always appreciate your newsletter, but especially during this very difficult period of changing assessments under ESSA. The lead story today about CA struggling with indicators is a peek into the problems all states will have in coming up with reasonable assessments. It will not be easy!
    Jerry from NJ

    Replies

    • Bruce William Smith 11 months ago11 months ago

      I'm sorry to have to predict that it won't work at all, and in a few years we'll be back to conferences where people will agree that yet another attempt to improve state education through the use of metrics has failed once again to improve the lives of our young people. What is needed instead is a fundamental shift in direction in federal education policy, and ESSA is not it; therefore every family that can … Read More

      I’m sorry to have to predict that it won’t work at all, and in a few years we’ll be back to conferences where people will agree that yet another attempt to improve state education through the use of metrics has failed once again to improve the lives of our young people. What is needed instead is a fundamental shift in direction in federal education policy, and ESSA is not it; therefore every family that can afford it should opt out of state schooling whenever possible until No Child Left Behind’s failed strategy for social improvement via annual testing and publishing the results is abandoned entirely, and until Sacramento gets serious about subsidiary devolution, which implies that assessing and reporting on the results of local schools should be left to the local districts, whose citizens may have different priorities and values that the state and federal governments should learn to respect.

  3. Doug McRae 11 months ago11 months ago

    Not mentioned is the problem that currently available metrics, both test and non-test, are not yet ready to implement an ambitious multiple measures accountability system for California that features both single year status data and multiple year change data for all indicators. The change data for growth measurement will likely require 3-years of data for most indicators, and the CDE acknowledged at the July SBE meeting they are not planning to provide 2-year Smarter Balanced … Read More

    Not mentioned is the problem that currently available metrics, both test and non-test, are not yet ready to implement an ambitious multiple measures accountability system for California that features both single year status data and multiple year change data for all indicators. The change data for growth measurement will likely require 3-years of data for most indicators, and the CDE acknowledged at the July SBE meeting they are not planning to provide 2-year Smarter Balanced change data for accountability use this fall. It will likely be 2018-2020 before CA has the needed metrics to implement the planned new system. Such data quality considerations should come before finalizing how best to display multiple measures results for multiple audiences, which the article accurately indicates will be a daunting task even when quality data is available.