Stanford-Harvard ‘scorecard’ translates California’s test scores into months of missed learning

October 28, 2022

A massive data project by researchers at Harvard and Stanford universities, released Friday, has converted national and state test scores into the equivalent measure of learning decline during the pandemic for every school district and student group in California and 28 other states.  

The interactive data in the Education Recovery Scorecard confirm, in clear and stark terms, the disparities in performance and learning loss among states, districts within states and racial, ethnic and income groups within districts for grades three to eight.  (Go here to start with California and other states.)

For California, the data in math translated into a straight downward line on a graph, showing that wealthier districts, those with a small percentage of low-income students, experienced the least loss of learning — a matter of a few weeks in Orinda, Lafayette and Los Gatos in the Bay Area — compared with the loss of three quarters or more in a grade equivalent in Bakersfield City and Mountain View Elementary school districts. (Go here for an interactive graphic showing learning decline in math for California districts — the blue circles.)

“Every poor district lost a lot, and rich districts lost a little, and that is the picture leaders in Sacramento should respond to,” said Sean Reardon, a Stanford Graduate School of Education professor and a senior fellow at the Stanford Institute for Economic Policy Research, which led the data analysis.

The data also shows that states that were largely open for in-person instruction during 2020-21, the height of the pandemic, did not perform significantly better than those whose students learned remotely for most of that time. Students in all states declined in math relative to where they were before Covid, but the decline in California, the last big state to fully reopen, was, at the equivalent of 48% of a grade level. That was identical to Florida and slightly better than Texas, which had a 57% loss. In both states schools stayed open all year. The 12% decline in reading was nearly identical for all three states.

However, California’s loss of nearly a half-year of learning in math, compounded with previous years, still placed it nearly a year behind the average district score in math nationally.

By comparing the loss of learning with their share of $190 billion in federal Covid relief that states and districts received, researchers computed how far the funding could close the gap. Their conclusion: not nearly enough, even with strategies such as tutoring or more learning time. In two-thirds of districts nationwide, the equivalent in learning losses by grade exceeded what the funding they will get from the American Rescue Plan can likely provide. It is the largest source of federal money to address learning loss and runs out in two years.

That’s a critical message for parents to hear because surveys indicate most parents believe that students have already made up nearly all of what they hadn’t learned, said Thomas Kane, a professor and a faculty director of the Center for Education Policy Research at Harvard, which co-led the project. “There’s a disconnect. Parents think students are 95% recovered, yet the average pandemic loss is half a year. So no leader can propose unpopular things like extending the school day and year if parents think things are fine.”

For example, Alum Rock Union School District in San Jose lost the equivalent of 84% of a grade during the pandemic, but the American Rescue Plan provided enough to cover only 23% of the district’s instructional budget. It would take more than three times that amount to make up for the missed learning, assuming it was directed to effective learning strategies.  

However, the Stanford calculations don’t include states’ additional funding, which, in California’s case, is substantial: $11 billion, including $4 billion to extend after-school and summer learning for low-income districts, in this year’s budget, on top of the $15 billion from the American Rescue Plan. An additional $4 billion each was appropriated for the next several years to expand mental health services and to establish community schools, where additional money can be used to hire counselors and pay for social services as well as academics.

Source: Education Recovery Scorecard by research centers at Harvard and Stanford universities

All of these selected California districts fell behind in math from 2018-19 to 2021-22, although the equivalent decline in learning was less for wealthy districts like San Ramon Valley, as measured by the percentage of families qualifying for free and reduced priced meals (FRPL). Bakersfield City’s decline was nearly the equivalent of a year of learning. The state average was 48%.

Reardon said that funding is also important. “A decline in scores is a symptom of harm from the pandemic, but lots of other features of the pandemic affected kids’ lives. If we had mental health and well-being measures, we would probably see sizable declines across the board,” he said. “It’s important to think of the harm as not just a schooling issue.”

Kane and Reardon used the same national data from the National Assessment of Educational Progress or NAEP and Smarter Balanced results for California from pre-pandemic 2018-19 and 2021-22 that EdSource reported this week. However, researchers generally can’t compare test results among states. Reardon’s team at Stanford overcame the challenge by creating a scale that aligned states’ annual test results with scores from the NAEP fourth and eighth grades. NAEP served “as kind of a Rosetta stone” to measure district-level changes in academic skills, he said.

The scoreboard’s graphics and data enable identifying high and low performers among California districts. Fremont Unified, in Alameda County, where a third of its students are low-income, lost only a tenth of a year in math learning and increased 16% in reading, compared with the national average.  

Garden Grove Unified, in Orange County, where two-thirds of students are from low-income families, declined only 32% of a year in math, significantly better than the state and national averages, and was unchanged in reading. Monterey Peninsula Unified was another higher performer among low-income districts.

Nationwide, achievement declines varied sharply among districts in the same state. “The pandemic was like a band of tornadoes that swept across the country. Some communities were left relatively untouched, while neighboring schools were devastated,” said Kane. 

While the median school district lost the equivalent of 52% in math and 23% of a year’s expected growth in reading, 2.5% of students were in districts where math achievement rose and 5.3% of students were in districts where achievement fell by more than one grade level. The latter include Oklahoma City, New Haven, Connecticut, and, with a two-year loss, Richmond, Virginia.  

In reading, 14.8% of students were in districts where achievement rose and 1.4% were in districts that lost more than one grade level.

The scorecard covers 4,000 school districts and 12 million students in 29 states that had reported test data. More states will be added as states publish their scores, Reardon said.

To get more reports like this one, click here to sign up for EdSource’s no-cost daily email on latest developments in education.

Share Article