STAR test scores decline for first time in a decade

Replies

• Doug McRae 10 years ago10 years ago

  Actually, Navigio, the one percent change I was referring to above is not the difference between zero and the decrease, but rather the difference between the previous year's gain (which was +2.0 points) and the current year's gain (which was -0.65 points) or a difference of 2.65 percentage points. That difference is not only statistically significant, but also educationally meaningful. The aggregate gain statistic I calculate for my initial observations on STAR results is kinda … Read More

  Actually, Navigio, the one percent change I was referring to above is not the difference between zero and the decrease, but rather the difference between the previous year’s gain (which was +2.0 points) and the current year’s gain (which was -0.65 points) or a difference of 2.65 percentage points. That difference is not only statistically significant, but also educationally meaningful.

  The aggregate gain statistic I calculate for my initial observations on STAR results is kinda a poor man’s API, an unweighted average of only the E/LA gains for 10 grade levels and the Math gains for 6 grade levels. It can be done in a few minutes after release of STAR results, rather than waiting for a month or so for the more complex weighted API data to be released. The rationale behind my “A” thru “F” grading system for these gain scores is much the same as the rationale behind the API goal of gains equal to 5 percent of the distance between the previous API and the statewide goal of 800, but for my poor man’s API that number comes out roughly to roughly 2.0 percentage points for raw unweighted gain scores — thus I assign a “C” grade to gains of 2.0 percentage points involving the close to 5 million students contributing to those gains. My statement that one percent is both statistically significant and educationally meaningful for this statistic is support for assigning a GPA-like meaning to the one percent differences between each grade level in the “A” thru “F” system.

  It would be pretty straightforward for a psychometrician or statistician to provide the numbers for different district sizes (or school sizes) at which the changes become statistically significant. But, as noted above, statistical significance is not a good criteria for interpreting these differences. Whether a gain score is educationally meaningful or not is a much more nuanced decision and not so amenable to straight statistical calculations. But, as you note above, most of the interpretations of STAR results are not solely based on statistical considerations, but rather are “spin” designed to carry a message a policymaker wants to communicate. K-12 tests operate in this kind of public policy environment, so I don’t criticize policymakers for providing these kinds of interpretations, but when needed I’m willing to point out when the “spin” is inconsistent with the reality of the numbers.

  • Doug McRae 10 years ago10 years ago

    Navigio: Your comment asked "What do we have to base the assumption that a gain is supposed to occur?" Then it went on to talk about actual results being a feedback loop to the cut score process and resetting cut scores and new test questions chosen based on previous year answers, etc. Perhaps the best way to approach your questions is to talk about how "standards-based tests" were conceptualized about 20 years ago -- the term … Read More

    Navigio: Your comment asked “What do we have to base the assumption that a gain is supposed to occur?” Then it went on to talk about actual results being a feedback loop to the cut score process and resetting cut scores and new test questions chosen based on previous year answers, etc.

    Perhaps the best way to approach your questions is to talk about how “standards-based tests” were conceptualized about 20 years ago — the term “standards” was chosen to represent expectations or goals for what folks wanted to the taught, and standards-based tests were designed to measure desired curriculum and instruction. This concept was in sharp contrast to the prevailing notions behind both national norm-referenced tests designed to measure relative achievement on what was currently being taught (not necessarily what folks wanted to be taught) and criterion-referenced tests which were designed to be more fine grained measurement for achievement on specific content objectives (not more broadly based “content standards”) again on what currently was being taught rather than what folks wanted to be taught. Thus, standards-based tests were conceptualized as a new paradigm for K-12 tests, designed to measure expectations that were higher than current established curriculum and instruction practices. The notion included fixed cut scores (or, using more technical language “performance standards”) which would serve as the basis for measuring achievement progress over time toward the expectations or goals set out by approved “content standards.” So, to directly answer your question, the notion that gains were supposed to occur over time was built into the conceptualization of standards-based tests and standards based curriculum and instruction programs, which were promulgated by the feds in the late ’90’s via IASA and then mandated via NCLB in 2002. The current state-by-state standards-based tests, then, are the result of this conceptual progression with expected increases set by the feds AYP program and in California by our accountability API program.

    On the operational aspects for our STAR California Standards-Based Tests, the so-called actual results feedback loop for setting cut scores occured only once in the life of the STAR tests, and that was back in 2000 and 2001 when the cut scores for STAR CSTs were originally set. During those standards-setting, the process included looking at actual item-by-item raw score results as part of the standards-setting process. Once cut scores were set back in 2000 and 2001, the cut scores have not been reset or changed. New items have been added to the tests to replace retired items over time on a programed schedule, primarily for test security purposes. The new questions have not been chosen based on previous year results; rather, they have been chosen to maintain the integrity of the CST blueprints which include not only content specifications but also psychometric specifications for items. When new items are installed, the overall test form that results may be a little bit easier or a little bit harder than the test form it replaced, and to insure that the “cut scores” are equivalent from year-to-year, equivalency or equating adjustments to the cut scores have been made from year-to-year.

    I hope this explains both where the expected gains notion originated for standards-based tests, as well as some of the mysteries around how fixed cut scores are set and maintained for our CSTs so that the CSTs can indeed measure actual gains (for comparisons against expected gains) over time.

    • Manuel 10 years ago10 years ago

      Thank you, Doug, for this very complete narrative on the CSTs. It does, however, confirm what I've been clumsily ranting about: ETS created a norm-referenced test where the "norm" was the entire state while basing all questions on the state's standards. Presto-chango, we got a "standards-based test" that is reproducible year after year. No wonder the histograms are so robust: they barely change and we argue endlessly over the merits of a point or two (or three!) … Read More

      Thank you, Doug, for this very complete narrative on the CSTs.

      It does, however, confirm what I’ve been clumsily ranting about: ETS created a norm-referenced test where the “norm” was the entire state while basing all questions on the state’s standards. Presto-chango, we got a “standards-based test” that is reproducible year after year.

      No wonder the histograms are so robust: they barely change and we argue endlessly over the merits of a point or two (or three!) change on the average of the distribution.

      Meanwhile the media keeps saying that “nearly half of California students are not on grade level.”

      Good grief…

      • Doug McRae 10 years ago10 years ago

        Sorry, Manuel: The facts are that ETS did not create a norm-referenced test and call it a standards-based test. The facts are that Harcourt Educational Measurement (the STAR vendor in the early 2000's) created a standards-based testing system with fixed cut scores that have been maintained by ETS since 2003 and employed to allow CA to measure achievement change over time. The underlying histograms you cite only reflect an undisputed point since academic achievement … Read More

        Sorry, Manuel: The facts are that ETS did not create a norm-referenced test and call it a standards-based test. The facts are that Harcourt Educational Measurement (the STAR vendor in the early 2000’s) created a standards-based testing system with fixed cut scores that have been maintained by ETS since 2003 and employed to allow CA to measure achievement change over time. The underlying histograms you cite only reflect an undisputed point since academic achievement tests have been in existance, that achievement is normally distributed regardless of how the test is designed, whether it is norm-referenced relative measurement with changing normative cut points over time or criterion-referenced with fine grained cut scores only for fine grained instructional objectives, or standards-based with fixed cut scores designed to measure gains or growth over years. Accurate media reports say that only about 1/3 of CA students were proficient when the CSTs were introduced in 2003, but that a little more than half are now proficent. If we project that growth out over another 10 years, accurate media would say that more than 3/4 of CA students are proficient. [I’m not suggesting we continue with STAR for another 10 years . . . we do need to change the statewide tests to reflect the Common Core standards that have been adopted, and over time we need to move to computerized tests just to get into the 21st Century.] My point is there is nothing in the standards-based system that requires results to remain at roughly 50 percent proficient; the system is not designed that way.

        • Manuel 10 years ago10 years ago

          Indeed, that is the arcana of testing: the responses will eventually approach a normal distribution after multiple administrations. Thank you for bringing that to the front of the discussion. But that happens only if the behavior of the test takers does not change over time. Instead, there has been a relentless drive to pump up the scores which has gone into a frenzy over the last few years. Under those conditions, if the tests questions were solely focused … Read More

          Indeed, that is the arcana of testing: the responses will eventually approach a normal distribution after multiple administrations. Thank you for bringing that to the front of the discussion.

          But that happens only if the behavior of the test takers does not change over time.

          Instead, there has been a relentless drive to pump up the scores which has gone into a frenzy over the last few years.

          Under those conditions, if the tests questions were solely focused on the standards, by now the entire distribution would have moved at least 60 points to the right because teachers would have been teaching to the test. They haven’t as the change has been roughly 30. And California teachers can’t be that bad, can they?

          Conversely, if the “evolution” of the test is organic and, indeed, the mean of the histograms is moving up very slowly (about 2 to 3 points per year for the mean), then the CSTs are an impediment to academic growth because they maintain the proficient population artificially low. That, on the average, that population is poor should give anyone pause.

          Thus, it doesn’t matter if I am correct or you are correct. Allowing this type of test (whether it is the CST or the Common Core) to be the source of many high-stakes decision is morally wrong. Why do continue to allow it?

    • navigio 10 years ago10 years ago

      Thanks Doug. I hope you don't mind me continuing to respond. This is a discussion I've been wishing would be had for years now. I understand that setting a fixed performance standard that was different than (above) existing ones would tend to provide a path for improvement toward that standard, but I'm not clear what changes from year to year that would cause that to happen forever. Perhaps you are simply saying that a norm-referenced methodology … Read More

      Thanks Doug. I hope you don’t mind me continuing to respond. This is a discussion I’ve been wishing would be had for years now.

      I understand that setting a fixed performance standard that was different than (above) existing ones would tend to provide a path for improvement toward that standard, but I’m not clear what changes from year to year that would cause that to happen forever. Perhaps you are simply saying that a norm-referenced methodology tends to lack ‘incentive’ to improve (actually, by definition, it lacks an improvement indicator, but I wont go there for now). In contrast a fixed standard methodology can always act as a baseline, and that alone provides the incentive to do something different in order to increase the number of kids over that cutoff. If that’s what you’re saying, I have to admit thats pretty interesting, and I guess would explain the fact that it aligns with the idea of accountability being useful simply because someone is paying attention.

      However, my question about improvement was not necessarily about incentive, but about teaching. In 2013, a 2nd grader will have been alive for about 7 years, and will have lived their entire life within the span of this fixed standard system. Even in 2010, the same could be said for a 2nd grader. And if one takes only school years (K-2, since thats where the vast majority of the test-based mechanism has its impact), the same could be said for 2nd graders from 2005 forward. Thats 8 years of 2nd grade results. However, looking at those results, they have been more or less on a constant increase since that time. Is the assumption that early on, maybe schools didnt really care much about increasing proficiency rates (perhaps because the NCLB 100% proficiency bomb was still a long way off?). Or perhaps the reason is that its taken us 10 years to figure out how to teach to those standards? (In theory, the blip we are seeing this year might be the leveling off point? Or perhaps we are not done and this is the result of something else). From a pedagogical standpoint, assuming everyone cared, I cant really see a reason why the scores would increase forever (or even for more than a test-takers lifespan, after which all test takers will have lived under the same influences).

      The interesting thing about all this is that it aligns with the apparent arbitrariness of NCLB thresholds, ie it doesnt really matter what the thresholds are, as long as they exist and are enforced (assuming the ‘incentive’ mentioned in the first paragraph is valid). Personally, I cant see how the Fed’s limits were a result of any kind of realistic expectation, or involved study or logic. Rather, they seemed to draw a point at the current proficiency rates, and then a line from that point to 100% and that was to be the cutoff for acceptable increases each year (is there any research that shows what the expected increase should be if the impact is simply one of having a fixed threshold? And did it match those arbitrary yearly increases? I would expect the pressure to increase as the cutoff date is neared, though that seems not to have happened either). The only way those cutoffs might have been the result of something thoughtful was in that they let the states define what proficiency meant, but even then, the standards would have had to be ‘aligned’ with those arbitrary increases in order to make any sense. Something tells me they werent (nor I expect would we actually want them to be). I guess in common core, one idea is to do away with differing state-set proficiency standards. And ironically, some of the early common core assessment results are pretty close to where we were back in 2003 or maybe a bit earlier. And although I probably shouldn’t have to ask this if I’d paid enough attention, I assume the common core assessments will ‘remain’, fixed-standard? Will there be a ‘resetting’ as in 2000-01 with CSTs?

      I hope I understood correctly, and that I am making even a little sense.. and I very much appreciate your patience.. 🙂

      • el 10 years ago10 years ago

        I also am skeptical of the notion that scores can or should go up every year, and would remind us all of the peril of comparing negative changes with positive changes sometimes creating artifacts. I would also point out that giving an F for a -1 that puts you at the second highest score to date kind of ties your hands compared to a result of -10 or more that was more comparable with the first … Read More

        I also am skeptical of the notion that scores can or should go up every year, and would remind us all of the peril of comparing negative changes with positive changes sometimes creating artifacts.

        I would also point out that giving an F for a -1 that puts you at the second highest score to date kind of ties your hands compared to a result of -10 or more that was more comparable with the first years they were given. 😉

        I’ve talked to several administrators who have mentioned anecdotally the stress the recession has caused, not just in terms of lack of school staff and resources, but parents that are stressed and tense, more relocations, and how that has reflected back to school discipline and culture. You can make up any story you want for why these numbers are what they are, but any given year there are a multitude of possibilities for positive and negative influences not just inside individual schools but across the state at large.

      • navigio 10 years ago10 years ago

        Ok, maybe I am asking too many questions? :-) Doug, my mind was going a 100mph when I wrote that and I wrote it on a cell phone, so I admit its a bit chaotic. I will try to distill my 'point' in this comment. Hopefully it is clearer. You said, "So, to directly answer your question, the notion that gains were supposed to occur over time was built into the conceptualization of standards-based tests and … Read More

        Ok, maybe I am asking too many questions? 🙂

        Doug, my mind was going a 100mph when I wrote that and I wrote it on a cell phone, so I admit its a bit chaotic. I will try to distill my ‘point’ in this comment. Hopefully it is clearer.

        You said, “So, to directly answer your question, the notion that gains were supposed to occur over time was built into the conceptualization of standards-based tests and standards based curriculum and instruction programs…”

        Saying this is ‘built into’ the tests would imply that we make the tests easier each year in order to increase proficiency (obviously not what is happening if its a criterion-referenced test and if these tests correlate exactly from year to year). How could it be ‘built-in’?

        Saying that is ‘built into’ the standards based curriculum and instruction programs is odd because that would also imply that either those standards and/or programs are still evolving (ie we’ve only been introducing them slowly–why in the world would we be doing that??!) or that there are outside influences that carry over from previous portions of a student’s life that the new standards and programs are still trying to ‘overcome’. This would have been a valid point a few years ago, but the reason I keep mentioning how long a test-taker has been alive is that if the curriculum, the instruction programs, the tests and the proficiency references have all intentionally remained static for the past however many years, then there is no way we could ever expect increases in student test scores as a result of that system itself. At least not for 2nd graders since 2010, and probably not for any kids since about 2005. So something must be changing.

        Simply saying its ‘built in’ is not really what I was looking for. 🙂

        (I still concede that you may have meant the simple fact that we have a fixed cutoff gives people incentive to improve, whereas a relative one does not. However, I still dont know if that’s what you meant to imply. If it is, then it also means the changes in test scores over the past decade are mostly a result of people not caring before, and caring more now. That seems like a problematic conclusion. Regardless, it would be good to see the research upon which that ‘theory’ of improvement is based.)

        • Doug McRae 10 years ago10 years ago

          Nagigio: The inference or interpretation for increasing test scores is certainly not based on tests being easier from year-to-year [CA and its vendor spend a lot of time and money insuring the tests are equivalent from year-to-year]nor even that the curriculum and instruction programs are still evolving [though that might be the case particularly early in the tenure of a standards-based testing program]. Rather, the inference or interpretation is that student achievement increases because … Read More

          Nagigio: The inference or interpretation for increasing test scores is certainly not based on tests being easier from year-to-year [CA and its vendor spend a lot of time and money insuring the tests are equivalent from year-to-year]nor even that the curriculum and instruction programs are still evolving [though that might be the case particularly early in the tenure of a standards-based testing program]. Rather, the inference or interpretation is that student achievement increases because instruction gets better over time — i.e, these tests are built to measure the results of instruction, and the expectation is that collective instruction for adopted content standards will improve with time. That is the inference or conclusion that the test designer is attempting to address with this type of test . . . . [Parenthetically, but not to open another can of worms, it’s quite another issue to attempt to attribute increases in achievment to the results of an individual teacher’s instructional efforts, and many if not most test makers voice concerns with the validity of these large scale tests when they are applied to inividual teacher evaluations, not because of the test properties per say but rather because of the validity of the attribution to individual teachers . . . . I wrote a commentary on this contentious issue some time ago for EdSource Today’s predecessor blog some time ago.]

        • navigio 10 years ago10 years ago

          Hi Doug. Thanks for the response. I have not been ignoring you, but doing some reading and thinking. You have really given me a lot to think about. :-) I think I mentioned elsewhere that I noticed an odd kind of change this year in the CSTs (YoY), specifically, the changes were different across grades. At first I was looking at my district and simply attributed this to mostly randomness (grade size is not that big), but … Read More

          Hi Doug.

          Thanks for the response. I have not been ignoring you, but doing some reading and thinking. You have really given me a lot to think about. 🙂

          I think I mentioned elsewhere that I noticed an odd kind of change this year in the CSTs (YoY), specifically, the changes were different across grades. At first I was looking at my district and simply attributed this to mostly randomness (grade size is not that big), but it got me thinking so I also looked at the state results by grade. Since entire state results are considered statistically significant at even less than one percent change, I figured statewide grade results could probably be considered such as well (there are almost a half million test takers per grade).

          The odd thing I noticed–and this seemed to be reflected to some extent at my district level–though not entirely–was the fact that there were drops in some grades and increases in others. Not just small numbers like less than 1%, but on the order of 5% in some grades.

          Specifically, in ELA there was about a 2.25 point drop in elementary grades, a 1 point drop in middle school grades and a 2.3 point increase in high school grades. But it gets even more interesting: The drop is pretty consistent for 2nd, 3rd, 4th, 5th, 7th and 8th (2 to 3 point), but there were increases in 6th grade, 9th grade and 10th grade. And surprisingly enough, 9th grade had a 5 point increase in ELA scores! That seems humongous at the state level.

          In thinking about the suggestion that CC might have something to do with that, I guess it might make sense that any CC changes might impact elementary more (those kids are also still more malleable). And I guess in theory, one might even decide to implement CC starting with the first class of each secondary school level (ie 6th and 9th). However, that would not explain why those grades increased, while elementary grades decreased. I was also struck by a recent UTLA poll of its teachers who did not seem to indicate much training for CC (that could mean either they are flying blind, or it could mean they are still working the old way).

          One thing I always do with test results is try to sanity check the demographics, since that can often explain more than anything. The general trend for all the major ethnic subgroups was pretty much identical (just with varying degree).

          Perhaps even odder, this trend was reflected in our mid size district (just under 20,000), ie large increases in 6th grade and 9th grade but even larger decreases in most of the other grades from 8th on down (an exception was we had an increase in 3rd and small decreases in 10th and 11th). Although I know we can discount our results due to the smaller sample size, the fact that they pretty much mirror state level results (except for degree) makes me think they are experiencing similar influences.

          Math is a bit of a different story. At the state level there was a relatively large (3 point) drop in 3rd grade and smaller ones for Geometry and Algebra II (2 and 1), while the rest of the middle and lower grades either held steady or increased slightly (including Algebra I). This basic pattern was reflected in the ethnic subgroup results. Similarly, there was a 2 point increase in Algebra I proficiency, with a 2 point decrease in Geometry and 1 point in Algebra II. I dont know if those results could be considered statistically significant, but if they are, they are noteworthy for their gap (4 point switch for 3rd grade compared to 2nd or 4th).

          The other odd switches were in science (also reflected in ethnic subgroups), where there was a 3 point drop in 5th grade science, but small increases in 8th and 10. Similarly, 3 point drops in biology and chemistry, but holding flat in physics. Those EOC science (and math) results are a bit trickier to analyze because not everyone takes them and participation can be influenced by other factors.

          Anyway, the point of all this mumbo jumbo was to highlight that when looked at with a finer granularity, there seems to be something very specific going on at the grade level, and more importantly, different things at different grade levels (it seems odd to imagine that different teachers have consistently gotten much better or worse at the curriculum in just one year–eg enough to explain an 8 percentage point switch between state level 5th and 9th grade ELA results. Perhaps there are some other policy related behaviors affecting these things, but then we would be measuring something else.

          Any thoughts on this? Do you think half a million students is a small enough sample size to explain these as more random occurrences than anything else? Does a mirroring at ethnic subgroup and partially district level give the results any additional statistical significance?

          On a related note, I noticed this past year ETS released the frequency distribution for the raw scores. I went back a few years and did not see these things released before. Do you know why they would not do that before? Is that giving away too much of the ‘special sauce’? 🙂 I asked ETS if they had any of this data and whether any of their other currently provided data can be provided in any form other than PDF. They said no.

        • el 10 years ago10 years ago

          Navigio, here is an idea to go with your very interesting observations. 5th graders and below have had their whole school career under tightening budgets and loss of CSR, loss of librarians, etc. Might be interesting to see if that trend holds up in higher resourced districts that had parcel taxes and other support beyond what was typical, and/or if districts that kept 180 days/CSR did better. High schoolers might be seeing that those early years of CSR … Read More

          Navigio, here is an idea to go with your very interesting observations.

          5th graders and below have had their whole school career under tightening budgets and loss of CSR, loss of librarians, etc.

          Might be interesting to see if that trend holds up in higher resourced districts that had parcel taxes and other support beyond what was typical, and/or if districts that kept 180 days/CSR did better.

          High schoolers might be seeing that those early years of CSR paid off.

          Or it might be that the test difficulty changed.

        • navigio 10 years ago10 years ago

          Hi El. Yeah, I'll try to see if I can find any big enough to matter. ;-) Actually, now that I think about it, in that EdSource survey, some of those large districts were pretty low-poverty. I'll see if I can find some time.. The odd thing is when looking at last year's state-level YoY results, essentially everything increased (and not insignificantly) except for 2nd grade math. I mean, literally across the board (state level, … Read More

          Hi El. Yeah, I’ll try to see if I can find any big enough to matter. 😉 Actually, now that I think about it, in that EdSource survey, some of those large districts were pretty low-poverty. I’ll see if I can find some time..

          The odd thing is when looking at last year’s state-level YoY results, essentially everything increased (and not insignificantly) except for 2nd grade math. I mean, literally across the board (state level, by grade). Those kids have been under those same pressures. That said, I have to admit with props 30 and 38 and the impending threat to slash education spending had 30 failed, we may have finally started to see the impacts (as mentioned before, our district kicked the can down the road, and even this year is having to cut millions) so maybe you’re right, but it’s just been delayed.

        • Doug McRae 10 years ago10 years ago

          Navigio: Responding to your lengthy reply posted at 3:34 pm yesterday (there was no reply button there): Actually, there is a big difference between the percent proficient data based on aggregating 10 grades of E/LA scores and 6 grades of Math scores [almost 8 million scores] and the statewide grade level by content area scores [involving about half million scores each]. My indication that one percentage point difference from year-to-year was meaningful … Read More

          Navigio: Responding to your lengthy reply posted at 3:34 pm yesterday (there was no reply button there): Actually, there is a big difference between the percent proficient data based on aggregating 10 grades of E/LA scores and 6 grades of Math scores [almost 8 million scores] and the statewide grade level by content area scores [involving about half million scores each]. My indication that one percentage point difference from year-to-year was meaningful was based on the aggregate of the almost 8 million scores. I would not say a one percentage point difference was meaningful for individual grade level content area scores differences from year-to-year — I’d venture you need a 3 to 5 point difference for statewide scores involving about a half million scores before the difference can be said to be meaningful. When you get down to district level scores, I recollect your district has about 20,000 students, or roughly 1600 students per grade level. Interpretation of grade level content area differences from year-to-year for your local district should use perhaps a 10 percentage point difference as the guideline for a meaningful difference.

          While I can provide some expert opinion on HOW to interpret meaningful differences for test results from year-to-year, when you get to the WHY part of interpreting scores, well, everybody is entitled to their own opinions. The Why question deals with information and perceptions for what’s going on in schools, from statewide to districts to individual schools, and that information and perception varies widely among K-12 school observers. It is the variation in those individual sets of information and perceptions that creates both interest and controversy when it comes to interpreting results of large scale testing programs. The EdSource Today venue provides a valuable outlet for individual opinions on the Why part of test results. As a test maker, I don’t claim any special expertise for answering the Why question, but I’m willing to weigh in with my opinions when I have them, just like everyone else in this space contributing their opinions on the 2013 STAR test results released by the SPI this year.

        • navigio 10 years ago10 years ago

          Thanks for your response Doug. I know my description was a bit chaotic. To be clear, there was a 5 point change in 9th grade ELA and 3 point changes in some other grades. For me the odd thing is that those were in the opposite direction from other grades, making the effective difference even much larger (ie what might have been a 1 of two point difference is an 8 point difference this year. … Read More

          Thanks for your response Doug. I know my description was a bit chaotic. To be clear, there was a 5 point change in 9th grade ELA and 3 point changes in some other grades. For me the odd thing is that those were in the opposite direction from other grades, making the effective difference even much larger (ie what might have been a 1 of two point difference is an 8 point difference this year. I know you keep saying the tests are dead on and not subject to variation but I cannot see how a specific policy behavior could have such an impact at just certain grade levels, at least not for a group as large as the entire state. I do want to make comparisons to other years to see whether there were similar variations before (I already did 2012 vs 2011 and nothing like this could be seen).
          Anyway, I’d love to hear your take on why. Not the entire state results, but the differences for 9th grade from other grades, for example. Thanks again.

        • Doug McRae 10 years ago10 years ago

          Navigio: OK, I've taken a look at the SPI Press Release tables (#2 for Math, # 8 for E/LA) again. Let's count how many differences greater than 3 points for the 16 grade level / content areas for which comparisons are apples-to-apples. For 2013, there is only one such difference, the grade 9 E/LA you mentioed. For 2012, I see two such differences, for grades 6 and 7 Math. For 2011, I … Read More

          Navigio: OK, I’ve taken a look at the SPI Press Release tables (#2 for Math, # 8 for E/LA) again. Let’s count how many differences greater than 3 points for the 16 grade level / content areas for which comparisons are apples-to-apples. For 2013, there is only one such difference, the grade 9 E/LA you mentioed. For 2012, I see two such differences, for grades 6 and 7 Math. For 2011, I see no such differences. My interpretation would be this pattern of differences from year-to-year for grade by content scores does not exceed what might be normal variations in scores, in other words not enough to search hard for reasons why these “meaningful” differences occured. If you know of specific circumstances for any of the individual grade levels / content areas, those circumstances might explain those specific results, but the pattern is not enough to say “Gee, these are large differences, let’s go search for reasons why . . . ” On the other hand, when you go back to 2005, you find that all 6 Math grades had greater than 3 point differences, and 6 of the 10 E/LA grades had greater than 3 point differences, with all of these differences in the + direction. That was a banner year for good STAR results. Those kind of differences were not normal variations in scores — they were unquestionably meaningful increases in achievement across the entire state. Searching for reasons why for 2005 was not only a reasonable but also a pleasant exercise, given that the results were uniformly positive.

        • navigio 10 years ago10 years ago

          Ok, I took a quick look at a couple large low poverty districts: clovis, saddleback, poway, capistrano. Each one had increases in 9th grade ELA. None had an increase in any ELA grade below 9th (in other words, grades 2 through 8 were either flat or decreased in every grade. In fact, 2 of the 4 had decreases in every single grade below 9th, while another had decreases in every grade below 9th, except for … Read More

          Ok, I took a quick look at a couple large low poverty districts: clovis, saddleback, poway, capistrano. Each one had increases in 9th grade ELA. None had an increase in any ELA grade below 9th (in other words, grades 2 through 8 were either flat or decreased in every grade. In fact, 2 of the 4 had decreases in every single grade below 9th, while another had decreases in every grade below 9th, except for one (and that grade was flat). Even though these are large districts, they are probably no longer statistically significant at each grade, so these exact same patterns must just be randomness. I guess I understand that CA level results are made up of individual district results, so seeing a similar pattern is expected, but this incessant decline everywhere but 9th is really starting to bother me.. much as my comments are starting to bother everyone else.. 😉 Did we put something in the water 15 years ago..?

        • Doug McRae 10 years ago10 years ago

          Navigio: Well, I'm not sure something in the water 15 years ago will hold water as an explanation for increasing test scores . . . .(grin). But if there was something statewide or even in certain kinds of districts / schools different for 9th graders last year than for other grade levels, then that would be a decent explanation. I can't think of any statewide initiative, or the way statewide dollars to districts were handled, … Read More

          Navigio: Well, I’m not sure something in the water 15 years ago will hold water as an explanation for increasing test scores . . . .(grin). But if there was something statewide or even in certain kinds of districts / schools different for 9th graders last year than for other grade levels, then that would be a decent explanation. I can’t think of any statewide initiative, or the way statewide dollars to districts were handled, that would fit that bill. The pattern of 9th grade vs other grades does raise some suspicion, but the non-sexy explanation that in any large kettle of fish (numbers) there will be a portion of those fish (numbers) that are not in concert with the other fish (numbers) in the kettle still might be the best explanation we have.

        • Manuel 10 years ago10 years ago

          Doug, there is one possibility that could explain navigio's observations: someone at ETS did not follow "the program" and the tests for 2013 "allowed" this outcome. The tests are, after all, created by humans and errors do happen... Read More

          Doug, there is one possibility that could explain navigio’s observations: someone at ETS did not follow “the program” and the tests for 2013 “allowed” this outcome.

          The tests are, after all, created by humans and errors do happen…

        • Doug McRae 10 years ago10 years ago

          Manuel: Yes, I agree, it's always possible there was a glitch somewhere in the the way the tables were loaded for scoring, or something like that. The CSTs forms used this last spring were repeats of previous forms used (no new items was my understanding) so the work ETS does to ensure comparability of forms when new or replacment items are added to forms would not seem to be a place where a glitch … Read More

          Manuel: Yes, I agree, it’s always possible there was a glitch somewhere in the the way the tables were loaded for scoring, or something like that. The CSTs forms used this last spring were repeats of previous forms used (no new items was my understanding) so the work ETS does to ensure comparability of forms when new or replacment items are added to forms would not seem to be a place where a glitch would materialize. But, the pattern of variation uncovered by Navigio would not seem to result from errors in the scoring tables or errors in data aggregations. Also, it’s also a possibility test security was violated, that someone out there had a copy of the previous form that was re-used and discretely disseminated it widely enough to cause 9th grade scores to be higher. That doesn’t seem probable given the number of higher scores needed to generate a +5% difference over years, but it has to be on a list of possibilities. You are correct, it’s always a possibility that something more nefarious could have caused the pattern that Navagio surfaced.

Replies

• el 10 years ago10 years ago

  Not to mention that these scores are better than 2011. The horror.

Replies

• Manuel 10 years ago10 years ago

  A true story: when I talked to a "highly placed Capitol staffer" on my CST histograms about two years ago, he confirmed to me that the "legislators" were warned that the distributions were not going to change over time if the test was to be designed the way it has been. They did not fully understand the advice, he said. He also said that this was the time to bring this to the front because … Read More

  A true story: when I talked to a “highly placed Capitol staffer” on my CST histograms about two years ago, he confirmed to me that the “legislators” were warned that the distributions were not going to change over time if the test was to be designed the way it has been. They did not fully understand the advice, he said. He also said that this was the time to bring this to the front because the law was sunsetting. Of course, we are now faced with a new set of assessments that we have no idea how they are designed and/or how they are going to work out.

  All we hear is “trust us.” Based on what has happened over the last ten years, I don’t think they have earned that trust. Hell, if I knew then what I know now I’d probably would have opted my children out.