LAUSD’s stunning reading score on NAEP: Illusion, real or something in between?

November 28, 2022

Heliotrope Avenue Elementary students in Los Angeles Unified begin the school in person in April 2021 after spending the year in remote learning.

Los Angeles Unified’s unexpectedly big increase in eighth grade reading on the National Assessment of Educational Progress drew praise and skepticism after the release of the scores last month. The results may have warranted some of both.  

LAUSD was the only one of the 26 urban districts nationwide that took NAEP to see improvement in 2022. And its 9-point growth from pre-pandemic 2019 was by far the biggest increase among all states and school districts on any test. Most saw a significant decline in reading and record declines in math on the fourth and eighth grade tests.

Some data experts and testing analysts questioned the size of Los Angeles’ gain and the significance of any single year’s score in isolation. In 2019, the district had an unusually large drop in scores from 2017, putting the rebound in 2022 into context, said Brian Gill, a senior fellow and data expert at Mathematica in Princeton, N.J. There may be actual improvement in reading in the district, or there may be an unexplainable quirkiness in the data; that can happen with statistical outliers, he said. “A focus on outliers could cause you to get fooled by randomness.”

But Los Angeles Unified Superintendent Alberto Carvalho took offense at these criticisms. Carvalho, the former superintendent of Miami-Dade County Public Schools for the 14 previous years, hadn’t started work in Los Angeles to take credit for the district’s success; NAEP was administered between January and March, and his first day on the job was Feb. 14. 

During a webcast in Washington, D.C., the day after the scores’ release, Carvalho attributed many factors to Los Angeles’ success, among them higher levels of middle school attendance and engagement, rapid universal distribution of home computers during the pandemic, high-intensity tutoring and a “massive amount” of teacher training.

In a Nov. 10 tweet, he lashed out at doubters, likening them to flat-earthers: “When the validity of released ‘gold standard’ #NAEP data, after the detailed due diligence conducted to ensure it, is questioned, you might as well question sea-level rise, global warming, or gravity for that matter; then again some of these voices probably do just that.”

States and urban districts put a lot of faith in NAEP because it is the one common national test they take. Usually it’s given every other odd year, but because of the pandemic the 2021 test was pushed out to 2022. 

State tests preclude direct comparisons because states create their own tests with different methodologies and measures of progress. An exception is the Smarter Balanced assessment, which California and 10 other states use. 

Unlike Smarter Balanced, where gains and losses are commonly measured in percentages, NAEP changes over time are measured in points on a 500-point scale; for states, differences of 3 points are often deemed significant.

Los Angeles’ 9-point increase in the eighth grade reading test stood out. It happened in a year that NAEP scores in reading and especially math fell nationwide. The average score in eighth grade reading among the 26 urban districts dropped 3 points.

NAEP basics

Unlike state assessments, which the federal government requires all students in grades three to eight to take — more than 3 million in California — NAEP is given to far fewer students: hundreds in small states and about 4,000 in California. NAEP selects schools and students to reflect the demographics of states and districts.

About 1,400 fourth and eighth graders in Los Angeles took the test this year. One dozen to two dozen students from each grade were chosen from among sets of 60 schools. Schools were alerted in the fall that they would give the test, but they didn’t get to choose the students.

To limit the testing time required, students took only about 20% of the tests. For eighth grade reading, each student was assigned two of the 12 sections, each a half-hour long.

LAUSD’s eighth grade reading scores grabbed the attention of the National Center for Education Statistics, the federal agency that administers the test. Its post-mortem found the methodology was sound and there wasn’t a high rate of absences, said Grady Wilburn, a statistician with the national center. The center also asked the American Institutes for Research to replicate the results. In its unpublished report, the institute found no flaws in the data and that high enrollment loss during the pandemic — another variable — didn’t affect the scores, Wilburn said.

However, the analysis did find one significant disparity. The 2019 Los Angeles Unified NAEP assessments didn’t include the scores of affiliated charter schools, while other testing years, including the 2022 test, did. Unlike Los Angeles’ 300 independent charter schools, the 50 affiliated public schools are run by the district, and teachers fall under the union contract.

The district’s seven affiliated charter middle schools, mostly located in prosperous neighborhoods, are high achievers. On this year’s eighth grade Smarter Balanced English language arts tests, 66% of students on average made or exceeded the standard, compared with the district average of 41%.

In 2022, NAEP tested students in five of the seven schools. They apparently did well on NAEP, too. The national center attributed 2 of the district’s 9-point increase to the affiliated charters alone.

That still leaves a 7-point gain for the district, compared with no change in California’s score from 2019. There was a similar pattern, to a lesser degree, in the 2022 Smarter Balanced English scores for eighth grade language arts: Scores for the state declined nearly 3 percentage points from 2019, while Los Angeles’ gained about 1 percentage point.

What distinguished LA?

Austin Beutner, who was superintendent of LAUSD from May 2018 through June 2021, is confident that actions the district took during the pandemic led to improvement in the NAEP scores, although he acknowledges establishing cause and effect is problematic. “In public education, people want to attribute cause where it doesn’t exist, and they miss it where it does,” he said.

What is clear, he said, is that the district moved faster than most urban districts to create conditions conducive to learning despite the ravages of Covid and unemployment in low-income areas. All students got computer and internet service; the district set up the nation’s biggest and most cost-effective Covid testing system, he said, followed by vaccinations. It created the largest school food program for families. “Covid testing kept people safe and helped get teachers and students back into school classrooms and keep them there,” he said.

Middle school is where you would see the difference that a free computer and internet made, he said. “For the first time, their learning could go from school to home and back and forth in a way it never had before.”

There were intangible factors that made learning fun and engaging. The district worked with PBS  to create three TV channels, by grade, during the year of remote learning, watched by a quarter of a million people weekly, Beutner said. The musical instrument maker Fender distributed 15,000 guitars for free for after-school lessons for middle schoolers, who could also learn about animations and storytelling through a partnership with Illumination Entertainment.

But the biggest impact, Beutner said, was in literacy and numeracy skills for kindergarten to third graders through Primary Promise, an initiative that placed 300 reading specialists and 140 math specialists in high-need elementary school classrooms. They provided small-group instruction, at first on Zoom and then in person, expanding to hundreds of schools and thousands of students. The first group of 2,500 first graders was chosen because they were struggling academically, with only 9% reading at grade level, Beutner said. Within 12 weeks, 42% of these same students were at grade level, he said.

“To me, literacy is the holy grail of public education, and Primary Promise is the biggest story of what we did,” he said.

These strategies may hold promise, but they didn’t make a huge difference on the fourth grade NAEP reading score. It was flat, while the nation, state and the urban districts’ average dropped 2 to 3 points. On the Smarter Balanced English language arts test, the district’s and the state’s scores both fell 4 percentage points.          

Alison Yoshimoto-Towery, who was the chief academic officer of Los Angeles Unified from 2019 until this September, said changes that were less visible to the public but important included the adoption in 2019 of a second literacy curriculum for K-6 — Core Knowledge Language Arts or CKLA — which is praised by advocates of “science of reading” for its attention to phonics and other foundational skills. 

That adoption didn’t factor into the eighth grade NAEP reading scores, but the introduction in 2021 of diagnostic or formative assessments and digital literacy tools could have had an impact, she said. Formative assessments provide status reports on students’ strengths and weaknesses, to guide teachers’ instruction, which the summative Smarter Balanced tests cannot do.

“Many things together resulted in a stronger foundation for teaching and learning,” she said, including increasing access to virtual and in-person field trips that spark student interest. 

However, other claims of good attendance and widespread high-dosage tutoring appear to be a stretch. Intensive after-school tutoring in 2021-22 was largely a failure; money budgeted for it in schools went unused. And more than 30% of students in the district were chronically absent last year.

Likelihood of statistical errors

Morgan Polikoff, a professor at the University of Southern California’s Rossier School of Education, said he was withholding judgment on what might have led to the district’s gain on NAEP. “I don’t want to hold up anyone as a model for post-pandemic response right now. The jury is still out,” he said.

The progress in eigh grade test scores could instead reflect the impact of the Local Control Funding Formula, the funding law that California adopted in 2013. It shifted funding to districts based on enrollment of English learners, low-income students and homeless and foster children. “Research shows funding matters, and Los Angeles gained the most,” he said.

Polikoff, Gill of Mathematica and other researchers trust NAEP’s accuracy and its methodology.

“I find no problem with the number of kids tested for NAEP,” said Gill. “I think they are testing enough to produce useful results if interpreted appropriately.”

Problems arise when comparing states and districts with different sample sizes and levels of precision, Gill said. NAEP statisticians may determine that a 3-point drop in states’ reading scores is significant, while a 3-point drop in urban districts’ scores is not. But that does not mean that urban districts outperformed the nation, Gill said.

NAEP procedures for a representative sample of students are designed to eliminate bias. but they cannot solve the potential for random errors caused by testing conditions, unpredictable circumstances and differences in individual student test-takers from year to year. 

In 2019, LAUSD’s NAEP scores fell a rare 6 points. NAEP test-taking  overlapped with a six-day teachers strike and the weekslong tension that led up to it. The possible impact offers a different perspective on the 7-point rebound in 2022.  

“The strike took time and attention away from the classroom. I’m surprised nobody thought of it, when they looked at NAEP scores that year,” Beutner said.

For two decades, as national reading scores on NAEP stagnated, then declined, Los Angeles Unified showed slow, steady progress on eighth grade reading with the exception of one year, 2019. This year’s score, 3 points above 2017, was consistent with the pattern. For the first time, its score was higher than the urban school average. In 2003, it lagged the state by 17 points and the nation by 27 points. This year, the gap had closed to 2 points. Modest improvement may be the good news.

“The most likely reality is that whatever gain actually happened in LA was more modest than 9 points. And that it probably had something to do with changes in the district over time — instructional or other policy changes,” said Polikoff.

Beutner had a different take. “On a relative basis, apple to apple, NAEP to NAEP, LA did meaningfully better than other major school districts” in 2022, he said. “But nationwide the NAEP numbers haven’t changed in 30 years. That’s the bigger story.”

To get more reports like this one, click here to sign up for EdSource’s no-cost daily email on latest developments in education.

Share Article