(This commentary first appeared in TOP-Ed.)
I’ve read with horror – as many of us have – story after story about the cheating mess on tests in Atlanta, focusing on 2009 state tests, and in other cities and states.
But then I wonder, what is “cheating”? What does that word mean? As I see it, it’s a way of pretending that a reported test score is valid and that the score actually tells us how the student performed and what that student actually knows on that test. Apparently, these cheating scandals show that scores had been tampered with in various ways and were not valid. Reported scores were too high – and did not actually show what students know and can do. Tragic.
And to this mess, I would add my concerns about state and federal testing policies that allow students, especially students with disabilities and English language learners, to take tests that may not be valid from the get go. These invalid tests include those that have been modified in significant ways and no longer measure what they purport to measure: providing a calculator for a student on a math computation test; having an instructor read a reading test to a student; giving extended time on a test that measures results under time pressure. Yet, score reports on these modified tests are issued and claim to show valid results. In these situations, we can’t blame teachers or students or any of the players on the ground. These are policies set at the top – by companies or states or other test makers. Examples?
How about the fact that the SAT and ACT now allow extended time on those critical tests for some students, without letting anyone know that the tests were modified?
How about reading a reading test to a student who can’t read or giving a calculator to a student who can’t add and subtract – and then reporting the tests as if they are valid. See, for example, Massachusetts policy.
How about the current California situation, where the use of the California Modified Assessment (CMA) has inflated the state’s accountability measures (API). The facts are: the CMA is easier than the regular test, more students with disabilities are using it, and their scores are added to the state measure. These issues have been discussed in prior TOP-Ed blogs.
Isn’t this, too, a form of “cheating”? Certainly it cheats students out of knowing what they can and cannot do. Also it cheats schools, taxpayers, and parents from getting a valid measure of student achievement.What do we actually mean by “cheating”? Who is cheating whom?
Miriam Kurtzig Freedman, author of Fixing Special Education, is an attorney, writer, consultant, and authority on special education. She is of counsel to Stoneman, Chandler & Miller LLP in Boston. She divides her year between Boston and Palo Alto, where she is a Visiting Fellow at the Hoover Institution. For more information and her blog, visit www.schoollawpro.com.
To get more reports like this one, click here to sign up for EdSource’s no-cost daily email on latest developments in education.
Miriam Kurtzig Freedman 12 years ago12 years ago
You may wish to check out “Disabling the SAT” at http://www.Educationnext.org for the story on how/why the College Board stopped flagging the SATs…
Cal 12 years ago12 years ago
The SAT and ACT are legally required to leave off the asterisk. They used to put it there, but a lawsuit deemed it discriminatory. You can't just get extended time, you have to apply for it. The SAT's standards are far easier to meet than the ACT's standards, but that's because extra time on the SAT does very little to help your scores, while on the ACT, time is a major factor (it shouldn't be, … Read More
The SAT and ACT are legally required to leave off the asterisk. They used to put it there, but a lawsuit deemed it discriminatory. You can’t just get extended time, you have to apply for it. The SAT’s standards are far easier to meet than the ACT’s standards, but that’s because extra time on the SAT does very little to help your scores, while on the ACT, time is a major factor (it shouldn’t be, but that’s a different issue).
The “help”, too, is not something that states do to increase test scores, but because IEP’s and 504 policies *legally require them* to provide this assistance.
You seem very unaware of the legal requirements, which makes your post a bit silly. Schools aren’t doing this to cheat. They’d rather not do it, for the most part. But if they don’t do it, they are sued for discrimination under the ADA.
el 12 years ago12 years ago
I personally think that the tight time limits on the SAT have always been one of its biggest weaknesses as an assessment instrument. I think they should give everyone extra time.
Doug McRae 12 years ago12 years ago
OK -- replies to Eric (# 5 above) and Miriam (# 8 above): Eric, I didn't mean to include you in an anti-testing or anti-accountability group. I know your views enuf not to do that. And I too would love to have a balanced assessment system with a board range of formats. But I'm not as convinced as you that a broad range of formats would substantially fix "teaching to the test" malpractices. And, … Read More
OK — replies to Eric (# 5 above) and Miriam (# 8 above):
Eric, I didn’t mean to include you in an anti-testing or anti-accountability group. I know your views enuf not to do that. And I too would love to have a balanced assessment system with a board range of formats. But I’m not as convinced as you that a broad range of formats would substantially fix “teaching to the test” malpractices. And, constructing a balanced assessment system with a broad range of formats is very likely to run afoul the current test administration time and cost constraints that are the reality for present day K-12 assessment system designs. Both federally financed assessment consortia are now promising the moon with their “next generation” assessment systems, but their political rhetoric / marketing promises have not banged heads with test administration time and/or implementation cost constraints as yet. When that happens, the rhetoric and promises will very likely have to be downscaled substantially. I would tend to agree with your suggestion for reduced emphasis on CSTs and APIs, especially a recognition that they are only one indicator that should not be taken as a sole or primary indicator of a school’s success or quality — good human judgments should always supercede rote quantitative indicators. And perhaps “explosive” was not a good choice of words — I would agree that “cheating” via malpractive test prep is an elephant in the testing dining room.
Miriam, you have more recent information on extended time for SAT and ACT than I have, and you have more detailed information on MCAS than I have. My experience with extended time for SAT goes back to 15 years ago when I was an advocate for an extended time SAT for my Spec Educ daughter, and interacted with ETS executives to insure she was able to take the test with appropriate extended time — I fully agreed with the emerging ETS policy to mark such administrations with an * on score reports to college admissions offices. If SAT and ACT have changed their policies on reporting of extended time administrations since the mid-90’s, then I’d like to know why . . . . as far as I know, SAT and ACT still have time limits and any untimed or extended time administrations clearly should be marked by some sort of qualifier or explanation so that folks can appropriately interpret the scores. Re the MCAS issue — if MA is reporting scores that result from true modifications of test administration protocol [note, modifications that change the construct being measured, not accommodations that do not affect the construct being measured] as true valid scores, then that is very poor educational measurement practice [I would call it educational measurement malpractice]. On this element, California has better practice than Massachusetts — CA does not report scores obtained via test administrations with modifications as true valid test scores. I appreciate your clarification that your use of the word “cheating” was intended to be provocative. My hesitation to use such a provocative word is that the kind of educational measurement malpractice you cite, while serious and very much in need of correction, doesn’t seem to be in the same bucket of behavior as things like adults changing answer sheets after have tests have been administered, or prompting students to change answers during test administration, or the other examples of clear adult misbehavior that seem to be the case in the Atlanta and DC situations that have received wide media coverage.
el 12 years ago12 years ago
On getting high school students to take the tests seriously: Our local high school created a policy where a score of proficient or advanced on the STAR test would raise that student's grade by one letter grade in the corresponding class. So, no penalty for doing poorly, but a significant carrot and reason to study for the test and make a serious effort. The end result seems to have been positive - students were happy, and … Read More
On getting high school students to take the tests seriously:
Our local high school created a policy where a score of proficient or advanced on the STAR test would raise that student’s grade by one letter grade in the corresponding class. So, no penalty for doing poorly, but a significant carrot and reason to study for the test and make a serious effort. The end result seems to have been positive – students were happy, and scores went up. It made the STAR test into an opportunity for them, rather than just a pointless chore.
Miriam Kurtzig Freedman 12 years ago12 years ago
Just to clarify, I was responding specifically to Doug McRae’s thoughtful comment. I thought it would appear right after his…
Miriam Kurtzig Freedman 12 years ago12 years ago
Interesting response. I leave to others to decide if this in quotation ‘cheating’ is intended to mislead. I used the term to provoke this issue: What do we call a state or test policy, whereby tests are administered in ways that provide misleading, invalid results, not honest feedback to students, parents, schools, and the rest of us? Invalid results do not measure what the test purports to measure; often they are created by the use … Read More
Interesting response. I leave to others to decide if this in quotation ‘cheating’ is intended to mislead. I used the term to provoke this issue: What do we call a state or test policy, whereby tests are administered in ways that provide misleading, invalid results, not honest feedback to students, parents, schools, and the rest of us? Invalid results do not measure what the test purports to measure; often they are created by the use of modifications. Modifications fundamentally alter the purpose of the test. For example, while a reading test is supposed to measure reading, not listening, the use of a reader alters that purpose. While a timed test is supposed to measure how a student performs within a given time constraint, allowing extended time alters that purpose. You get the idea. [Whether timing on tests should matter is an important question, but one I leave for another day. Here, I focus on how tests that are designed to be administered in specific ways—are not.]
When we change the way tests are given and get invalid results, what should we call that policy? At the very least, it cheats everyone from knowing what students know and can do.
Two other specifics. In 2003, the ACT and SAT removed the asterisk and stopped flagging scores taken with extended time. Now, some students do take these tests with extra time, and no one knows it. For the story behind this unfortunate policy, please see my 2003 “Disabling the SAT” at http://www.educationnext.org. And see Nightline’s follow-up coverage. http://www.youtube.com/watch?v=GfGYrSudW6s. Note however, that this TV story fails to point out the reason for the push to use extended time—that the asterisk or flag had been removed.
The Massachusetts issue I raised deals with the standard MCAS, not the Alt-MCAS. (Please see pages 7-23, and 25). The policy allows the ‘standard’ MCAS to be administered with modifications, called ‘non-standard accommodations.’ These may include a reader for the reading test and a calculator for portions of the MCAS that specify ‘no calculator’ use. The scores obtained from these tests are then counted along with those that were administered with no accommodations or with standard accommodations. So, do we now know if the student can read or if she can add or subtract? No. Simply stated, invalid results do not provide honest feedback.
Whatever we choose to call these practices and policies, I believe they are damaging and cheat us all from knowing what students actually know and can do.
Clearly, my blog touched a nerve. Thank you for that! Maybe someone has a better term for this policy and practice. The bottom line: we need honest test reporting.
Randall Delling 12 years ago12 years ago
What is tragic is the acceptance of an accountability system that creates injustice and promotes non proficiency. What is tragic is the willingness of self anointed "reformers" frothing at the bit to use bad science. I am going to speak for high schools. Students in high school do not take the test at alarming rates. High school students report that they fake taking these tests at a rate of 66%. They know the test counts … Read More
What is tragic is the acceptance of an accountability system that creates injustice and promotes non proficiency. What is tragic is the willingness of self anointed “reformers” frothing at the bit to use bad science. I am going to speak for high schools. Students in high school do not take the test at alarming rates. High school students report that they fake taking these tests at a rate of 66%. They know the test counts for nothing, no classroom grade, no college entrance, no graduation, nothing. If you ask a high school student about this, they will ask you to tell them why they should take the test seriously. When you say that the new so-called “reformers” are firing teachers over the test results they will respond with, and I quote, “Why should I care about some stinking teacher’s job?” If you are not aware that this data is extremely flawed for these reasons, then you have spent no time on a high school campus and do not know what you are talking about. The zeal with which a “reformer” will use this bad science to label teachers, principals, and school communities as failing is truly disturbing. No one is saying there should be no accountability, but when you hold dedicated, hard working, caring educators responsible for exam results that students are held harmless against, you have created injustice. If you create or accept injustice, you are not a leader. You are a user of bad science to promote a political agenda that is all about stealing public money and firing good people and nothing else. I know when a game is being run on me. I have the highest test scores in Los Angeles, and I am ashamed of myself for buying in to a system that lets students pass from grade to grade without being proficient and lets them graduate while their teachers and principals are unjustly criticized by “reformers” who are nothing of the kind. True reformers would be talking about putting iPads into all students hands with internet access, all textbooks, periodic and progress monitoring assessments, and the State assessments given on those iPads. A true reformer would be in favor of getting results back to schools immediately so that non proficient students could get another chance to take the test, and another, and another until they are proficient. A true reformer would hold students equally accountable for results as they would hold teachers and principals. This is bad science used for political gain at the expense of students, parents, teachers and principals. How tragic that we are so willing to jump in and make generalized comments about results of data that is literally worthless. Anyone who has talked to high school students at length is aware of this issue. They would also be aware of our public policy makers arrogant refusal to even open the dialogue regarding this reality. Tragic indeed.
Ariel Paisley 12 years ago12 years ago
Please consider the possibility that the real cheating going on here has not been discussed in these postings at all. Our students are being cheated out of rich personal futures because they are being denied access to a well rounded education by folks who push our educational system towards the narrow focus on testable learning. Our children are being cheated out art, music, career ed., PE, recess, and other non-academic activities. Our kids are being cheated … Read More
Please consider the possibility that the real cheating going on here has not been discussed in these postings at all.
Our students are being cheated out of rich personal futures because they are being denied access to a well rounded education by folks who push our educational system towards the narrow focus on testable learning. Our children are being cheated out art, music, career ed., PE, recess, and other non-academic activities. Our kids are being cheated out of their chance to fall in love with learning due to this perverse preoccupation with measurable outcomes and quantifiable data.
Oh yes, there is tremendous cheating going on in our public educational system, but the most egregious cheating is perpetrated by people who do not work in the classrooms with our kids. Rather, it is the policy makers who deny our children access to a wholistic, child-centered, slow-paced, discovery-driven education that are really cheating our next generation.
Of course, these policy-making folks will have all kinds of justifications for depriving kids of the whole-child education they really need. I wonder how many of our ‘top-level’ educational decision makers have ever spent a few years in the classroom with kids, let alone a couple of decades. What really concerns me is the utter cluelessness exhibited those who are under-educated about how children actually learn. The mere fact that some folks actually care more about test scores than they care about whole-child education betrays a profound ignorance of children and their true needs.
Heaven forbid that we would put childrens’ real developmental needs ahead of their test scores.
Eric Premack 12 years ago12 years ago
For the record, I'm neither anti-test, nor anti-accountability. Quite the contrary. Rather, I believe standardized, closed response "#2 pencil bubble" tests like the California Standards Tests do have an important role, but as only one component of a larger and balanced assessment system that includes a broad range of formats. A balanced range of formats will never fix all problems, but presumably would make it significantly more challenging to cheat while increasing incentives to teach to … Read More
For the record, I’m neither anti-test, nor anti-accountability. Quite the contrary.
Rather, I believe standardized, closed response “#2 pencil bubble” tests like the California Standards Tests do have an important role, but as only one component of a larger and balanced assessment system that includes a broad range of formats. A balanced range of formats will never fix all problems, but presumably would make it significantly more challenging to cheat while increasing incentives to teach to a broader range of skills and aptitudes. Our current unbalanced system begs teachers, schools, and districts (and charter schools) to cheat–it should be no wonder why so many do.
If we’re not willing to invest in a more broad and balanced system, we should significantly reduce the emphasis placed on the CSTs and the API, recognizing that they’re simply one narrow indicator among many, and not, in and of themselves, a sufficient base on which to judge a school’s success or quality.
Nor did I intend to be “explosive” about anything, but instead seek remind folks about the cheating elephant sitting in the middle of our collective living room.
Doug McRae 12 years ago12 years ago
A couple observations on Miriam's scathing commentary and Eric's explosive comment -- For Miriam: The word "cheating" implies intent to mislead, and I'm not sure some of the practices you cite rise to that level. Rather, it rises to the level of educational measurement malpractice, which for professionals should be just as damning a characterization. For extended time, ACT and SAT have put asterisks with explanations on their score reports for extended time … Read More
A couple observations on Miriam’s scathing commentary and Eric’s explosive comment —
For Miriam: The word “cheating” implies intent to mislead, and I’m not sure some of the practices you cite rise to that level. Rather, it rises to the level of educational measurement malpractice, which for professionals should be just as damning a characterization. For extended time, ACT and SAT have put asterisks with explanations on their score reports for extended time administrations at least since the mid-90’s, and before then strictly prohibited extended time administrations (which in my opinion was poor policy). The Massachusetts policy to which you link, on page 25 of the link, shows that MA uses differing performance level labels for their alternate tests for Spec Educ students — those differing labels with their definitions help folks know that the achievement levels obtained is somehow different for mainstream MCAS tests vs MCAS-ALT tests. That is better educational measurement practice than CA’s practice of using the same label [i.e., “proficient”] to mean different things for CA’s CST, CMA, and CAPA versions of the statewide STAR program that measure distinctly differing levels of achievement. CA’s practice invites misinterpretations in the trenches. For CA, methinks Spec Educ STAR test administration practices rise to the level of “cheating” in some districts when they overuse/abuse the testing options available for Spec Educ kids, but misleading information at the state level is just poor educational measurement practice that rises to the level of malpractice when it is pointed out to state officials and policymakers and they do nothing to remedy the misleading information that is promulgated.
Eric’s comment raises a larger and much more troublesome issue — the unfortunate response to high stakes tests in terms of truly lousy “test prep” approaches to curriculum and instruction in our schools. If we had testing police and prosecutors in CA, indeed Eric is right that many widespread practices targeting material on the statewide STAR tests would violate the state law he quotes. What bothers me is blaming the test for these unfortunate teaching practices. The argument by anti-testing anti-accountability folks is essentially — get rid of tests since they “cause” ineptitude and malpractice in curriculum and instruction. That’s like — let’s get rid of interscholastic athletics because they “cause” coaches to require excess practice time and parents to do bad things to the kids who participate. A subtext for this conversation is to bash the maligned multiple-choice test question format, and claim that other formats (open-ended, portfolio, whatever, which by the way result in more time and dollars devoted to the testing enterprise) will correct the problem. Balderdash — open ended testing formats are just as susceptible to bad test prep and test manipulation tacktics as multiple-choice item formats — witness Science Fair and Soap Box Derby events manipulated by overeager parents. I’m afraid that basic human frailities are the root cause for poor curriculum and instruction practices targeting what high stakes tests seek to measure — the fraility to look for shortcuts, the easy way out, attempting to boost test scores without doing the hard work of well rounded teaching efforts designed to address the full range of content expectations included in rigorous academic content standards. This human fraility is not unlike the fraility of greed as it affects our financial markets. When asked whether we should attempt to regulate greed in our financial markets, Warren Buffet said “No — I don’t think that’s possible” but we can expose it when it occurs and motivate individuals to take a higher moral ground in their decisions. I think we need a similar strategy to address the “cheating” phenomena that Eric so accurately describes.
Molly 12 years ago12 years ago
Eric, I completely agree with you. Though I will admit that I have worked in two other states (IL and AZ) where the teachers are required to structure their curriculum and by extension their lessons, around the state tests, as opposed to the state standards. At the last school I taught for (in Arizona) they went so far as to provide professional development for the teachers about which strands, or which finite standards, are heavily tested … Read More
I completely agree with you. Though I will admit that I have worked in two other states (IL and AZ) where the teachers are required to structure their curriculum and by extension their lessons, around the state tests, as opposed to the state standards. At the last school I taught for (in Arizona) they went so far as to provide professional development for the teachers about which strands, or which finite standards, are heavily tested in the state tests and how to adhere our lesson plans and curriculum more to those finite standards. Their theory was, the better they score on the big sections, the little sections don’t matter. In that regard, I completely agree with your ideas about cheating and only testing certain strands of standards. It seems completely unfair to me as a teacher and can only imagine how the students feel when they do very well on that one state test, but fall far below expectations on the others.
Now I am an ELL/ELD teacher and I see the same thing occurring again. They structure their curriculum and thereby their lessons each student “needs to know” in order to be considered “successful” according to the school.
It is an unreal practice and I think if administrators and the public at large began to take a serious look around, they would (much to their dismay) discover that it is not only Atlanta with cheating problems, but other states as well.
Eric Premack 12 years ago12 years ago
Arguably, cheating is extremely widespread in California and brazenly practiced by the vast majority of schools and districts. There is seemingly little enforcement of anti-cheating laws and guidelines. One of the most pervasive forms of cheating in California is the widespread use of textbooks and instructional materials that deliberately contain tools that facilitate the narrowing of focus on those standards that are actually tested to the apparent exclusion of a broader base of knowledge, or what … Read More
Arguably, cheating is extremely widespread in California and brazenly practiced by the vast majority of schools and districts. There is seemingly little enforcement of anti-cheating laws and guidelines.
One of the most pervasive forms of cheating in California is the widespread use of textbooks and instructional materials that deliberately contain tools that facilitate the narrowing of focus on those standards that are actually tested to the apparent exclusion of a broader base of knowledge, or what assessment geeks call the domain. For anyone who has perused a modern California-approved textbook, this takes the form of built-in highlighting of those concepts that are tested, lists of “key,” “must-know” items at the end of each chapter, test questions at the end of the chapter that are designed to focus on and align with the types of items that appear on the California Standards Tests–and to the apparent exclusion of the larger domain.
Another pervasive and common form of cheating is the use of automated cheating software. This software tests students relative to California’s specific content standards, and can even spit-out standard-by-standard information on a given student’s knowledge of a given standard, and then can spit-out instructional activities narrowly designed to bring the student up-to-speed on that narrow list of standards–again to the exclusion of the larger domain.
Is this “cheating,” or is it merely “standards-based instruction?” The line, it seems, has been blurred.
Arguably, the types of widespread practices described above violate state law, which provides that districts/schools may not “not carry on any program of specific preparation of pupils for the statewide pupil assessment program or a particular test used therein.”
If it doesn’t violate state law, it certainly seems to violate what I was taught constitutes “cheating” when I studied education measurement as a graduate student some years ago. The current “Standards for Educational and Psychological Testing” state that, “The integrity of test results should be maintained by eliminating practices designed to raise scores without improving performance on the construct or domain being tested.” I believe these are currently undergoing a re-write and it will be interesting to see if the gurus who develop these standards will take on this issue.
This tacitly-sanctioned and widespread use of cheating strategies is perhaps one reason for the remarkably large increases in statewide California Standards Test scores while California students’ performance on other assessments that speak to the broader domain (e.g., NAEP) is relatively flat.
For all we know, much of the run-up we’ve seen in state API scores of late is due to narrowing of the curriculum, test-prep, and other cheating activities and we know little about broader achievement beyond California’s rather peculiar standards.
el 12 years ago12 years ago
Why is a student’s performance under time pressure something that is important to measure? Is a child who can do a math problem in 2 minutes instead of 1 minute dangerously un-proficient?
And have you looked at the CMA test? Do you believe a child without knowledge and learning can score proficient on it?