Textbooks: Do we know which ones are effective?

Replies

• Ze'ev Wurman 2 years ago2 years ago

  Cathy Kessel is correct in that one of the references for the mediocrity of Common Core I quoted is from February of 2010, while the final version of Common Core was released only in June of 2010. So here is a report with similar finding on the final Common Core version, published in July 2010. https://pioneerinstitute.org/download/national-standards-still-dont-make-the-grade/ Kessel also finds an issue with the other paper I quoted, which argues Common Core "is fully two to three years … Read More

  Cathy Kessel is correct in that one of the references for the mediocrity of Common Core I quoted is from February of 2010, while the final version of Common Core was released only in June of 2010. So here is a report with similar finding on the final Common Core version, published in July 2010.

  https://pioneerinstitute.org/download/national-standards-still-dont-make-the-grade/

  Kessel also finds an issue with the other paper I quoted, which argues Common Core “is fully two to three years behind what is expected from students in high-achieving countries” and demonstrates that when it comes to integer addition, it is “only” one year behind. There are other standards and expectations where Common Core is more behind but, eventually, all this is now moot, when we have actual empirical evidence of Common Core mediocrity.

  On PISA international comparison, US declined between 2009 and 2018 by 9 points (from 487 to 478). On long-term NAEP our age 9 student declined 9 points (almost a grade level) between 2008 and 2019. And the C-SAIL report is the clincher, where the researchers embarked to find Common Core superiority and instead found that:

  “Our study revealed that the adoption of the new college- and career-ready standards had significant negative effects in grade 4 reading. Specifically, our analysis suggests that had the “treatment” states stuck with their prior standards, their grade 4 NAEP reading scores would have been 2.3 to 3.8 points higher on the 0-500 NAEP scale during the seven years after the adoption of the new standards. We also found a significant negative seven-year impact on grade 8 mathematics. While most of our results were statistically non-significant, they tended to be in the negative direction.”

  https://www.the74million.org/article/song-did-common-core-standards-work-new-study-finds-small-but-disturbing-negative-impacts-on-students-academic-achievement/

  • Cathy Kessel 2 years ago2 years ago

    Re "demonstrates that when it comes to integer addition, it is “only” one year behind”: Does “Include: addition and subtraction of numbers up to 4 digits” mean “fluency with addition and subtraction of numbers up to 4 digits using a standard algorithm”? I don’t see anything in the Singapore teacher’s guides that says so. Also, note that grade 3 addition and subtraction expectations for Russia (then USSR) and for Singapore differ: one includes 5-digit numbers, … Read More

    Re “demonstrates that when it comes to integer addition, it is “only” one year behind”: Does “Include: addition and subtraction of numbers up to 4 digits” mean “fluency with addition and subtraction of numbers up to 4 digits using a standard algorithm”? I don’t see anything in the Singapore teacher’s guides that says so. Also, note that grade 3 addition and subtraction expectations for Russia (then USSR) and for Singapore differ: one includes 5-digit numbers, the other does not. I believe this discrepancy may be due to an error in the 2017 “mediocrity” report’s account of Russian mathematics education history, and will link an account of details here: https://mathedck.wordpress.com/2013/08/04/comments-on-milgrams-review-of-final-draft-core-standards/.

    On the page about the C-SAIL “report” (“Effects of states’ adoption of college- and career-ready standards on student achievement”), the reader is told that it was a working paper (that is, not peer reviewed). I write “was” because the link for the paper is dead. There is a more recent peer-reviewed article with a somewhat similar name (“Did States’ Adoption of More Rigorous Standards Lead to Improved Student Achievement? Evidence From a Comparative Interrupted Time Series Study of Standards-Based Reform”), somewhat different conclusions, and two of the same authors. Its abstract says:

    “This study was designed to assess the effects of states’ adoption of more rigorous standards as part of the current wave of standards-based reform on student achievement using comparative interrupted time series analyses based on state-level NAEP data from 1990 to 2017. Results show that the effects of adopting more rigorous standards on students’ mathematics achievement were generally small and not significant. The effects on students’ reading achievement were also generally small, but negative and statistically significant for Grade 4. The study also revealed that the effects of states’ adoption of more rigorous standards varied across NAEP subscales and student subgroups.” https://journals.sagepub.com/doi/abs/10.3102/00028312211058460

    Similarly, there were mixed results when, years ago, the Singapore books were used at four US sites. Here is an excerpt from the report (which was published in 2005 by AIR, the American Institutes for Research):

    “The Montgomery County [Maryland] outcomes were positively correlated with the amount of professional training the staff received. Two Singapore pilot schools availed themselves of extensive professional development and outperformed the controls; two other pilot schools had low staff commitment coupled with low exposure to professional training and were actually outperformed by the controls. Professional training is important in helping teachers understand and explain the nonroutine, multistep problems in the Singapore textbooks. Teachers also need preparation to explain solutions to Singapore problems, which often require students to draw on previously taught mathematics topics, which the Singapore textbook, in contrast to U.S. textbooks, does not reteach.” https://www.air.org/resource/report/what-united-states-can-learn-singapores-world-class-mathematics-system-exploratory

    After noting the variability in test site outcomes, the AIR report says: “our study shifted from merely assessing the results from the U.S. textbook pilots to developing a broad comparison of the coherence and quality of the Singapore and U.S. systems for delivering mathematics instruction.” For example, it notes “Singapore’s teachers must take a stringent examination before being accepted to education school, and while they are students, they are paid a teacher’s salary.”

    Also, “Singapore’s professional development program offers sustained learning opportunities through a modularized approach that adjusts to teachers’ learning needs and is integrated into a continual learning process that includes experiential training experiences in nonschool settings. In contrast, professional training in U.S. school systems focuses on short-term workshops that fit into teachers’ released time. Evaluations suggest that these experiences are not likely to improve teachers’ performance in mathematics; the teachers themselves agree and do not seem to value this training very much.”

    A recent PD example from Singapore: https://www.science.edu.sg/for-schools/teacher-professional-development/teacher-work-attachment.
    A recent PD example from the US: https://edsource.org/2022/we-need-to-fix-professional-development-for-teachers/678632

    As with the AIR study, the mixed results of the C-SAIL study may have prompted more focus on the US educational system, rather than a single component of the system. The Center on Standards, Alignment, Instruction, and Learning (C-SAIL) conference in 2020 was on “What’s next for standards-based reform? With the release of the 2019 National Assessment of Educational Progress results in math and reading, it became clear that standards-based reform has not moved the needle on student achievement. This may be due, in part, to how districts, schools, and teachers are making sense of and implementing college- and career-readiness standards.” https://www.c-sail.org/videos

    Transforming expectations (such as standards) into instruction is no easy matter—as shown by three decades of standard-based reform. “We failed to prioritize the hard work of translating standards into rigorous curriculum materials, instructional strategies and teacher training,” says a recent essay (https://www.future-ed.org/unfinished-agenda-the-future-of-standards-based-school-reform/).

    • Ze'ev Wurman 2 years ago2 years ago

      Cathy Kessel and I could continue this back-and-forth for a long time, yet a few facts are indisputable: The key ones are that the regular (Main) NAEP results started to level off and drop since the 2017 NAEP, after decades where they consistently grew. Similarly, the results for Long-Term (LTT) NAEP dropped for the first time between 2008 and 2019. PISA results, particularly in math, did the same for US students. US TIMSS achievement increased by 22 (!) points … Read More

      Cathy Kessel and I could continue this back-and-forth for a long time, yet a few facts are indisputable:

      The key ones are that the regular (Main) NAEP results started to level off and drop since the 2017 NAEP, after decades where they consistently grew.

      Similarly, the results for Long-Term (LTT) NAEP dropped for the first time between 2008 and 2019.

      PISA results, particularly in math, did the same for US students.

      US TIMSS achievement increased by 22 (!) points between 1995 and 2015, yet *dropped* by by 4 points between 2015 and 2019.

      All this occurred *before* the pandemic but after Common Core adoption.

      Kessel is correct when she quotes “Transforming expectations (such as standards) into instruction is no easy matter – as shown by three decades of standard-based reform,” particularly in systems similar to ours where teachers are often encouraged not to follow recommendations based on robust research “invent” curricula and pedagogies on-the-fly behind their closed doors. Yet such consistent *national* (as opposed to local and regional) results across multiple national and international tests, of actual *decreases* in achievement since Common Core entered the mainstream of our classrooms, cannot be “explained” away as just “complexities of implementation,” as Kessel attempts.

      There are many other issues with Kessel’s attempts to show that standards like Singapore’s or other high achieving countries are “similar” to Common Core, but this is inside baseball to most so I will not engage in it here.

      • Cathy Kessel 1 year ago1 year ago

        People concerned about children’s education might appreciate clarification of claims about high-achieving countries. The 2017 “mediocrity” report Wurman cites says “[Common Core 4.NBT.4 Fluently add and subtract multi-digit whole numbers using the standard algorithm] is fully two to three years behind what is expected from students in high-achieving countries” (page 16, https://pioneerinstitute.org/pioneer-research/academic-standards-pioneer-research/mediocrity-2-0-massachusetts-rebrands-common-core-ela-and-math/). In contrast, Wurman is a co-author of a 2020 report that says “MA.3.NSO.2.1 Add and subtract multi-digit whole numbers including using a standard … Read More

        People concerned about children’s education might appreciate clarification of claims about high-achieving countries.

        The 2017 “mediocrity” report Wurman cites says “[Common Core 4.NBT.4 Fluently add and subtract multi-digit whole numbers using the standard algorithm] is fully two to three years behind what is expected from students in high-achieving countries” (page 16, https://pioneerinstitute.org/pioneer-research/academic-standards-pioneer-research/mediocrity-2-0-massachusetts-rebrands-common-core-ela-and-math/).

        In contrast, Wurman is a co-author of a 2020 report that says “MA.3.NSO.2.1 Add and subtract multi-digit whole numbers including using a standard algorithm with procedural fluency” is “exemplary” (page 2, Better Than Common Core, https://www.independent.org/publications/article.asp?id=13176).

        Perhaps a future Independent Institute report will explain this apparent contradiction.

        NAEP results: As a commenter has pointed out, Minnesota did not adopt Common Core. Its grade 8 NAEP scores follow the national pattern.

        https://nces.ed.gov/nationsreportcard/subject/publications/stt2019/pdf/2020013MN8.pdf

        Despite the predictions of the “mediocrity” report (full title is Mediocrity 2.0: Massachusetts rebrands Common Core ELA & Math), grade 8 NAEP scores in Massachusetts were higher than Minnesota’s:

        https://nces.ed.gov/nationsreportcard/subject/publications/stt2019/pdf/2020013MA8.pdf

        One author has suggested that state investment in teachers is an important part of the explanation:

        “Even more relevant for us is the experience of Massachusetts, which in the early 1990s had mathematics achievement that was fairly typical of the US, but which decided that that was not good enough, and embarked on a long-term effort to improve. The Massachusetts Education Reform Act of 1993 provided for increased funding to make opportunities to learn more equitable, substantial professional development, and higher mathematics requirements for teacher certification (thus impacting teacher preparation programs) [6]. It worked. Massachusetts now, considered as a separate country, has test scores significantly above the US average [20], [16], [11], [7]. Massachusetts also comes out at the top in the National Assessment of Educational Progress (NAEP), the US’s own instrument for assessing student achievement. For example, in the 2019 edition of NAEP, Massachusetts eighth-graders had the highest overall average of any state, and perhaps more significantly, 48% of them scored at the proficient or advanced level [12]. Only two other states had over 40% at these levels. The rest of the country should take some lessons.”

        For references and the rest of the article, see https://www.ams.org/notices/202006/rnoti-p842.pdf.