Credit: John C. Osborn and John Fensterwald/EdSource TodayEarlier this month, a research institute at Stanford University affiliated with the Hoover Institution reported that students at independent charter schools in Los Angeles performed a lot better than their peers in traditional Los Angeles Unified District schools. The Center for Research on Education Outcomes (CREDO) has now released a report for California as a whole, and the results are mixed.
Data for six years of students in grades 2 to 11 in nearly 1,000 charter schools showed that overall they performed better in reading but did worse in math. In reading, 32 percent of charter students outperformed their peers at traditional schools while 21 percent underperformed. But in math, 37 percent underperformed while 29 percent outperformed traditional schools with similar students.
CREDO matched students with similar demographics in charters and nearby district schools, and, as it did with its other charter school studies, translated the differences in achievement into roughly equivalent additional days of learning (with the caveat that they should be “interpreted cautiously”). The result: charter school students gained 14 learning days in reading but lost the same number of days in math, based on a 180-day school year.
However, there were marked differences among charter schools. Students in urban locations, poor students and African-Americans who attended charters gained learning days compared to their peers at traditional schools. Students in rural areas, and Asian and white students enrolled in charters lost learning days (72 days lost in math learning for white students).
Results for Hispanic students were mixed, with seven learning days gained in reading and 14 days lost in math. However, low-income Hispanics in charter schools gained 22 days in reading and 29 days in math. A smaller proportion of English learners attend charter schools than traditional public schools (17 percent versus 24 percent statewide), but those who do attend charters perform better academically than their peers, with 36 learning days gained in reading and 50 days in math.
Elementary and middle school charters outperformed traditional schools, but charter high schools overall performed worse. At multi-level charter schools, serving elementary and middle grades or middle and high school students, students lost more than 100 learning days in math. “Unfortunately, more than half of the charter students in California attend high schools or multi-level schools, so their lack of growth has a large impact on the overall math results,” the study said.
Nearly all of the academic gains by charter schools statewide were by schools connected to charter management organizations. Source: CREDO
As in Los Angeles, charter schools that are part of a charter management organization, such as KIPP, Rocketship Education and Aspire Public Schools, excelled compared with unaffiliated charters usually started by parent groups, teachers or local non-profits. For charter management-connected schools, the gain was 36 days in reading and 28 days in math; for unaffiliated charters, the reading gain was 7 days; the loss in math jumped to 29 days.
CREDO published its first report on charters in California in 2009. In that study, charter school students on average had a seven-day learning gain in reading and a 22-day loss in math. The new report, using achievement scores in the same schools operating then and now, shows a doubling of the gain, to 14 days, in reading and a narrowing of the gap in math, to seven days.
To get more reports like this one, click here to sign up for EdSource’s no-cost daily email on latest developments in education.
Comments (27)
Comments Policy
We welcome your comments. All comments are moderated for civility, relevance and other considerations. Click here for EdSource's Comments Policy.
Don 9 years ago9 years ago
The insinuation that the neighborhood schools measure in SF would have increased segregation is bunk. You just admitted you don't know about the issue so why are you assuming the union was trying to prevent racism? Please don't make irresponsible charges like that. That's exactly the mentality the union employed. SFUSD schools are very segregated now because of the dynamics of the school choice assignment system. But because of the uniquely integrated quality … Read More
The insinuation that the neighborhood schools measure in SF would have increased segregation is bunk. You just admitted you don’t know about the issue so why are you assuming the union was trying to prevent racism? Please don’t make irresponsible charges like that. That’s exactly the mentality the union employed. SFUSD schools are very segregated now because of the dynamics of the school choice assignment system. But because of the uniquely integrated quality of SF neighborhoods, a neighborhood policy would increase integration. (Bayview Hunter’s Point is one of the most integrated neighborhoods in the City.) I don’t want to go further as I’m traveling far off topic.
Don 9 years ago9 years ago
I wasn't laughing, Gary. Your sarcasm aside, I taught high school for several years - that was 15 years ago. Maybe it has become more professional since then. I was always baffled that these self-serving "leaders" were allowed to take a portion of the wages in order to work their agendas, pay off legislators and generally prevent any change to the decades old lock-down on education and all to increase their ranks while unknowingly nurture fertile … Read More
I wasn’t laughing, Gary.
Your sarcasm aside, I taught high school for several years – that was 15 years ago. Maybe it has become more professional since then. I was always baffled that these self-serving “leaders” were allowed to take a portion of the wages in order to work their agendas, pay off legislators and generally prevent any change to the decades old lock-down on education and all to increase their ranks while unknowingly nurture fertile ground for an incipient charter “industry”. It funny how these leaders ensconce themselves for life with salaries that make the teacher salaries look even worse than they already are.
But “bosses” could be a loaded word and I’ll grant you that it wasn’t the best choice of words. OTOH, your comment wasn’t particularly respectful either. I’m wondering if your sarcastic and rather harsh response to blog repartee is indicative of the attitude you brought to the classroom. I know you were kidding but I doubt you were “mild-mannered”. It doesn’t fit your commentary style. I remember a lot of those kinds of teachers – bulies. Oh, ya. The ones that plaintiff’s witnesses talked about in Vergara. The ones that scream in children’s faces. But strangely enough this is the very reason I’m having trouble at a charter school. My perspective is not one-sided.
That aside, I’m sometimes embarrassed by the tactics used by the teacher unions to sway elections – elections and tactics that both result often in less learning for students – tactics that don’t help dispel the notion of union thuggery. On a local level, United Educators spewed a mountain of abject falsehoods to scare the public in its campaign against neighborhood schools. I saw Department of Elections personnel behind the counter wearing their union buttons on the night of the election. The opposition tore down our signs and showed themselves to be less than “professional”. A lot less. So your response doesn’t surprise me.
Replies
Gary Ravani 9 years ago9 years ago
Don: Of course you're not laughing. I'm laughing. The reason you are perceiving bullying is that you are unaware of the details of issues about which you preach with passion. When i point that out, it's deflating. You have a 1st amendment right to express yourself, and I have a similar right to respond. On to more deflation: You are, it appears, unaware that as a teacher and a member of a bargaining unit you had the right … Read More
Don:
Of course you’re not laughing. I’m laughing.
The reason you are perceiving bullying is that you are unaware of the details of issues about which you preach with passion. When i point that out, it’s deflating. You have a 1st amendment right to express yourself, and I have a similar right to respond.
On to more deflation: You are, it appears, unaware that as a teacher and a member of a bargaining unit you had the right to become an “agency fee payer,”and receive a refund of dues dedicated to”… work their agendas, pay off legislators and generally prevent any change to the decades old lock-down on education…” If that was your perception of what unions do to try to improve teachers’ working conditions which are, in turn, students’ learning conditions so be it. Your departure from the classroom must have been a bitter one.
I taught 31 years at middle school. No “bullies” can last at middle school, or really, in the profession. Interesting that what Vergara testimony exposed, if you’d been paying attention, is that at least two of the several teachers pointed out by the plaintiffs were “teachers of the year.” Kind of ironic, really. Bullies? Hardly.
My commentary style, which granted is aggressive, comes from long years of representing teachers’ interest because most teachers devote most of their energy to their teaching. It comes from having to constantly deal with repetitive and ritual bashing of teachers and their unions. And those kind of casual, unthinking, unsubstantiated comments like “tactics that both result often in less learning for students.” Know what I mean?
I am out of my area of expertise when it comes to event is SF, purported to be around “neighborhood schools.” It sound, though, like the union might have had concerns about school segregation. Segregated schools are having a very negative effect on CA students as documented by the UCLA Civil Rights Center.
Time to move on to other topics.
Don 9 years ago9 years ago
Gary, I don't think confusion explains why some might not agree with you and others who believe that the traditional 19th Century public school system ought to be sacrosanct. As a parent who is very displeased with my son's charter school I am no die-hard pro-charter advocate. But neither does that particular experience color my general view that the public education space ought not to be monopolized by employee unions which act primarily to serve … Read More
Gary, I don’t think confusion explains why some might not agree with you and others who believe that the traditional 19th Century public school system ought to be sacrosanct. As a parent who is very displeased with my son’s charter school I am no die-hard pro-charter advocate. But neither does that particular experience color my general view that the public education space ought not to be monopolized by employee unions which act primarily to serve their members not the students. Charters are an “industry’ – not a single entity hell bent on undermining the traditional schools. I know some element are just as there are forces intent on expunging every last charter. But let’s face it, powerful as some of the more extreme charter operators are, the real 800 pound gorilla is the one that works the halls of Sac and DC for their union bosses.
As a former teacher I’m no fan of what is happening to the profession in the name of certain reformist ideas OTOH or a dug-in anti-change union driven mentality, OTO. But charters are not the bogeyman even if slick operators are part of it, the laws need fixing or, as in my case, some schools lack a civic mentality. I see no philosophic, pedagogic, economic and sociologic rationale to view the bloated and immovable education establishment as the default position unless I simply can’t care anymore.
Replies
Don 9 years ago9 years ago
Correction : Charter are """NOT""" an 'industry' Read More
Correction : Charter are “””NOT””” an ‘industry’
Gary Ravani 9 years ago9 years ago
Don: If you don't think the charter industry considers itself an industry then you haven't done your homework. Log on to a few of their websites and see how they describe themselves. See how much of the business oriented media look at the potential to harvest public dollars for private pockets. Then your comments: " public education space ought not to be monopolized by employee unions which act primarily to serve their members not the students. " … Read More
Don:
If you don’t think the charter industry considers itself an industry then you haven’t done your homework. Log on to a few of their websites and see how they describe themselves. See how much of the business oriented media look at the potential to harvest public dollars for private pockets.
Then your comments: ” public education space ought not to be monopolized by employee unions which act primarily to serve their members not the students. ” and “800 pound gorilla is the one that works the halls of Sac and DC for their union bosses.” Nice pick up on the standard talking points of the anti-labor right.
The teacher unions, representing the views of several hundred thousand classroom educators, hold their own in Sacramento. Who else is going to express the view from the classroom? However, if it is your impression that the views of real teachers and their unions have had any impact on what goes on at the USDE level then your impressions are totally unformed. See NCLB and RTTT. Not only have federal policies of the last decade and a half been contrary to what teachers said should happen in school renewal, the policies have been empirically demonstrable flops.
If the “education establishment,” such as it is, were able to maintain itself in a “bloated” condition as you suggest, in a state that is 49th or 50th in funding national, well, that would be something of an accomplishment. Wouldn’t it? Actually the “establishment” has fewer teachers per student, fewer administrators per student, fewer librarians per student, fewer nurses per student, and generally fewer adults at schools to provide services to students. If that’s your definition of “bloat,” go for it.
Don 9 years ago9 years ago
I think what you mean, Gary, when you say I didn't do my homework, is that if I don't come to the same conclusion as you do it must be the result of laziness or inability. You go on to say that I am using right-wing talking points - a tired guilt-by-association rebuttal tactic that is at its heart a moral failing. If you have a substantive response just say it. You said in … Read More
I think what you mean, Gary, when you say I didn’t do my homework, is that if I don’t come to the same conclusion as you do it must be the result of laziness or inability. You go on to say that I am using right-wing talking points – a tired guilt-by-association rebuttal tactic that is at its heart a moral failing. If you have a substantive response just say it.
You said in response to union influence, “who else is going to express the view from the classroom?”
Certainly not the students. And thank goodness they don’t for the sake of teachers because students would never go along with allowing certain teachers to receive lifetime employment regardless of ability – a position contrary to the efforts made by the unions to ensure steady lifetime employment for all members effective or not. If teacher unions focused on the classroom and its students, there would be greater interest in the professionalism of the membership and internal regulation of it rather than a concentrated focus on compensation and benefits above all and for all, again regardless of ability.
I agree with your comments about RTTT and NCLB, but how are they relate to this topic other than that charter school opponents also oppose those destructive reform efforts?
I take exception to this notion that the charter industry is a monolith intent upon the dismantling of traditional schools. Yes, extreme elements have this view just as extreme elements on the other side believe that the traditional schools should be a monopoly. But the majority of charters are independents and of those that aren’t the majority are non-profits, not to say that entirely dismisses them from an overall bias, but it does remove the profit motive critique in large part.
The clear fact that education is underfunded does not in any way reduce the problem of waste, fraud and abuse. That less than half the appropriated dollars actually reach the classroom is a problem shared equally by charters and traditional alike.
Gary Ravani 9 years ago9 years ago
So, if i read you correctly terms like "bosses" and " unions which act primarily to serve their members not the students" are "substantive responses" and not just conservative/neo-liberal slogans. Right. And to point that out is akin to asserting guilt by association. Sure. And then "lifetime employment for members effective or not." Yikes. I see what you mean. It's practically a dissertation. You obviously have no clue about what unions do to improve professionalism. There are … Read More
So, if i read you correctly terms like “bosses” and ” unions which act primarily to serve their members not the students” are “substantive responses” and not just conservative/neo-liberal slogans. Right.
And to point that out is akin to asserting guilt by association. Sure.
And then “lifetime employment for members effective or not.” Yikes. I see what you mean. It’s practically a dissertation.
You obviously have no clue about what unions do to improve professionalism. There are almost constant conferences and workshops to do just that.
And yes, let’s put kids in charge. There could be a single ballot: Prom King & Queen at the top and Most and Least Favorite Teacher at the bottom. That’s the ticket.
Did you ever consider that every time teachers negotiate to decrease class size or agree to increase psychologists, nurses, librarians, etc. that they are effectively bargaining to take dollars off the table that could be devoted to compensation, but instead agree to support improved instruction for students?
Homework!
I recall my 19 years as a local union leader. I spent my mornings as a mild-mannered English/History middle school teacher and then every afternoon I put on my union hat and channeled the spirit of jimmy Hoffa and became a “union boss.” And “thug,” did I mention “thug?” Best years of my life, and that’s no joke.
Don 9 years ago9 years ago
I realize that this site is superlative because it delves down to the finer points and that the devil's in the details. But in this case it's seems abundantly clear, as Nora pointed out, the results of this study along with many others paint a clear if not ho-hum picture of indecisive results, unless widespread egregious bias by numerous well-respected researchers is omnipresent. So let me comment on a personal level which might be a … Read More
I realize that this site is superlative because it delves down to the finer points and that the devil’s in the details. But in this case it’s seems abundantly clear, as Nora pointed out, the results of this study along with many others paint a clear if not ho-hum picture of indecisive results, unless widespread egregious bias by numerous well-respected researchers is omnipresent.
So let me comment on a personal level which might be a refreshing break from rehashing the details, frequently, only to come to the same inevitable lack of any conclusive win for either pro or anti-charter forces. I have two boys in schools in SF, one in a tradition high school and the other in a charter middle school. I will tell you that parents don’t decide to go charter because of test scores. They go for the different opportunities, i.e, size, teacher-student ratio, different pedagogies, unique programs, location, etc. Many sidewalk discussions have led me to believe STAR and API results have not been significant factors in charter school choice.
I find the focus on large research results informative, but monotonous, though perhaps more interesting when focused locally. Still, anti-charter folks, while roundly bashing the validity of high-stakes testing, insist such invalid achievement metrics prove their cases when it pleases them and vice-versa for the pro camp – the battle being waged precisely due to lack of clarity.
Some percentage wants the alternatives (some say innovations)that TPSs often don’t provide. Charter opponents advocate closing lower performing charters and yet they consistently opposed the former NCLB shutdown turnaround model for TPSs as though traditional public schools are more legitimate even if equally as bad or worse. And charter schools, pupil for pupil, receive less public funding, although I heard LCFF has changed that.
Nowadays, school choice isn’t choice at all without charters in the mix because options are few.
Replies
John Fensterwald 9 years ago9 years ago
Don, Nora, Bill Honig:
Point well-taken on the study’s characterization of significant differences.
Gary Ravani 9 years ago9 years ago
Don: I can understand your confusion. It has been the charter lobby/industry that has established the rules of the game as presenting charters as the panacea to low test scores in regular public schools. It is, therefore, logical to point to the hypocrisy of the charter industrial complex when it resists is shutting down charters with low scores. It is not hypocritical in the reverse, as you suggest. It should be noted that one of the … Read More
Don:
I can understand your confusion. It has been the charter lobby/industry that has established the rules of the game as presenting charters as the panacea to low test scores in regular public schools. It is, therefore, logical to point to the hypocrisy of the charter industrial complex when it resists is shutting down charters with low scores. It is not hypocritical in the reverse, as you suggest.
It should be noted that one of the statewide charter organizations has called for shutting low performing charters down. Whether that is because they understand the conflicts with their own stated goals, or that they see it as an opportunity to replace one charter with one of their own brand of charter is not for me to say.
The hypocrisy writ large can be seen in the actions of many cities, states, and the USDE that contend replacing a regular public school with a charter is the chosen option to “cure” that low test achievement. It is obvious that the current research, even by avowed advocates of charters, finds it a slim chance of “substantial’ improvement occurring because of imposing a charter on a community. Again, the choice of test scores as the currency of discussion has been established by the charter advocates. Therefore, rightly speaking, whatever a “petard’ is, they are hoist by theirs.
navigio 9 years ago9 years ago
I dont think charter opponents necessarily advocate closing lower performing charters, rather to the extent they mention that, it is only in the context of consistency. (arguing that tps is failing based on test scores, then ignoring test scores in the response model is hypocritical). but I think many opponents would probably disagree with measuring schools primarily by test scores in the first place, regardless of the school type. i think its possible to … Read More
I dont think charter opponents necessarily advocate closing lower performing charters, rather to the extent they mention that, it is only in the context of consistency. (arguing that tps is failing based on test scores, then ignoring test scores in the response model is hypocritical). but I think many opponents would probably disagree with measuring schools primarily by test scores in the first place, regardless of the school type. i think its possible to want every school to succeed while still being against the policy that allows some of them to exist. in that context, charter policy is the bogeyman, if individual charters are not. an important distinction.
fwiw, open enrollment options are built into ed code. they are also built into nclb (though in an increasingly meaningless way there). we also have magnets and choice-only schools. there are even many options for attending out of district. so there are options for choice. it is true that charters have become the de-facto choice model, but not sure that is a good thing. not because charters necessarily fail, but because school choice is itself divisive and damaging. and charter-style choice is probably the most damaging of all.
I also believe the primary decision factor for charters is demographics. I believe the results of this study indicate that different even groups have different reasons for deciding on charters.
In the end, the ‘panacea’ for good schools is simply choosing to make them work. not some politically-based ‘alternative’.
Andrew 9 years ago9 years ago
I assume that some of the modest Charter School gains in the LA area resulted because Charters there were able to take their pick of the best of 20,000 or so newer teachers laid off through LIFO by the conventional public schools in that region during the years of layoffs. But Charters are notoriously poor at retaining teachers and limiting turnover, treating teachers as throwaways, rationalizing rather than remedying their turnover problem. … Read More
I assume that some of the modest Charter School gains in the LA area resulted because Charters there were able to take their pick of the best of 20,000 or so newer teachers laid off through LIFO by the conventional public schools in that region during the years of layoffs. But Charters are notoriously poor at retaining teachers and limiting turnover, treating teachers as throwaways, rationalizing rather than remedying their turnover problem. So unless Charters, spoiled by the temporary teacher glut, develop and implement new strategies for treating teachers well and keeping them for the long term, I expect the LA Charter gains will be short-lived as the excellent newer teachers migrate back into conventional schools as finances improve.
It is not surprising that Charter students tend to fall short in math, because rapid teacher turnover that characterizes Charters is not conducive to the consistent building of concepts on concepts needed for math mastery.
Charter schools exist only because of a legal bargain that was struck at their inception in California. Charter schools were given great freedom and were exempted from the bulk of legal regulation on the condition that they produce excellent outcomes in student achievement. In practice, on the whole, they have neither been regulated nor produced the outcomes that should be expected, especially given their self-selected students and parents. But they have developed a self-serving constituency adroit at rationalizing Charter shortcomings in teacher treatment and retention and modest academic achievement.
Replies
Paul 9 years ago9 years ago
Andrew, I think that this is a fruitful line of reasoning. I know my own response to charters, from when I was teaching. In some cases, at-will employment, low wages, and lack of innovation made me less interested in working for charters than for their sponsoring districts. A district school colleague whose son attended a charter from which I received a job offer tipped me off that there were no stars there, and that the school's … Read More
Andrew, I think that this is a fruitful line of reasoning.
I know my own response to charters, from when I was teaching. In some cases, at-will employment, low wages, and lack of innovation made me less interested in working for charters than for their sponsoring districts. A district school colleague whose son attended a charter from which I received a job offer tipped me off that there were no stars there, and that the school’s success was simply due to demographics. (She had placed her son there only because it was a safer environment for him, as an individual.)
On the other hand, I felt very privileged to teach algebra for another charter, because that school employs top-notch math educators, uses an innovative curriculum (CPM = Common Core math before there was Common Core), and demonstrates good results with a diverse population.
We have general statistics about charter school teacher retention, and they are awful, as you say. To complete the picture, I think it would be interesting to survey job seekers. Do they see charters as employers of last resort, or are they drawn to features that set charters apart? Presumably, we’d see differences between young and old candidates, and between autonomous charters and charter systems like KIPP.
School districts also treat teachers as throwaways. Layoffs received wide coverage, but the popular press has never said a single word about the thousands of temporary teachers who are automatically let go each year. The state doesn’t even keep a count! Teachers migrating from charters to district schools these days are likely to receive temporary contracts that lead nowhere, rather than probationary contracts that would set them on a path to a stable employment relationship.
As you say, rapid teacher turnover hinders consistent teaching of concepts.
Bill Honig 9 years ago9 years ago
John Fensterwald's introduction to this article states that the previously CREDO study of LA charters found that they "performed a lot better" than their peers. Actually, the effect size found was minimal-- .07 standard deviation for reading and .11 for math. In the research literature these results constitute negligible gains (excluding the possibility that even these small gains were inflated because of self-selection and high levels of attrition as one of the comments suggested.) For … Read More
John Fensterwald’s introduction to this article states that the previously CREDO study of LA charters found that they “performed a lot better” than their peers. Actually, the effect size found was minimal– .07 standard deviation for reading and .11 for math. In the research literature these results constitute negligible gains (excluding the possibility that even these small gains were inflated because of self-selection and high levels of attrition as one of the comments suggested.) For example, John Hattie, a well-respected researcher, in his study Visible Learning, reviewed almost all the major programs and strategies to improve education and listed the effect size of the top hundred. Almost all these studies had effect sizes substantially higher than .1 standard deviation. As an example, multi-tiered instruction or RTI (teach it right initially and then provide for rapid intervention for students still struggling) had an effect size of 1.2 standard deviations equal to 2-3 years of instruction or 12 times the LA charter study. Similar, but somewhat lower results, occurred for teacher capacity building, reciprocal teaching, or curriculum improvement.
Additionally, the CREDO report talks of “significant” gains. That word is misleading. In statistical terms it just means that there was a 95% chance that some change occurred and the results where not by chance, but does not mean it was substantial, the way the word is used in everyday communication. Unfortunately, many readers and commentators interpret statistical significance as meaning substantial gains.
So what’s the effect of these two studies? Some charters and charter organizations work well (Aspire) and should be supported and encouraged. Some work poorly and should be held accountable or not renewed. But, overall charter schools are not a panacea for improving public education or a substitute for comprehensive school improvement such as is now occurring in the state with the implementation of Common Core.
Nora Carr 9 years ago9 years ago
I will have to dig more deeply into the data to understand CREDO's conversion formula. The learning gains must not be applied to all charter students, but only to those who did better in charter schools, which is an important nuance. On the surface, if only 32 percent of charter school students performed better in reading, that means 68 percent performed the same or worse as those in traditional public schools, which makes the learning … Read More
I will have to dig more deeply into the data to understand CREDO’s conversion formula. The learning gains must not be applied to all charter students, but only to those who did better in charter schools, which is an important nuance. On the surface, if only 32 percent of charter school students performed better in reading, that means 68 percent performed the same or worse as those in traditional public schools, which makes the learning day gains seem less plausible. If 21 percent underperformed traditional schools in reading, that brings the percentage down to 47 percent performing about the same in either type school. Perhaps I’m missing something, but I wonder how many studies we need to show that results are mixed, despite the significant growth in the charter school sector? Some are better; most are no different; some are worse. So why are we doing this again?
Replies
navigio 9 years ago9 years ago
Money.
Manuel 9 years ago9 years ago
When I read the LA-centric study, I noticed that CREDO "matched" the students according to socioeconomics and other factors. I am, however, skeptic that such "matching" can account for school "culture," thus making the comparison somewhat meaningless. I also noticed that the claims of "36 days of learning gained" are peculiar. Does this mean that a student considered to be on grade-level advances 7 weeks into the next grade by the end of the year? What … Read More
When I read the LA-centric study, I noticed that CREDO “matched” the students according to socioeconomics and other factors. I am, however, skeptic that such “matching” can account for school “culture,” thus making the comparison somewhat meaningless.
I also noticed that the claims of “36 days of learning gained” are peculiar. Does this mean that a student considered to be on grade-level advances 7 weeks into the next grade by the end of the year? What classroom teaches material that belongs to the next grade?
Lastly, let’s not forget that these results are based on CST scores, which, after looking at the 2013 raw scores indicate a significant amount of data massaging. I would take any claims about “academic advancement” based on the CST with a large grain of salt.
GIGO, in other words…
Replies
John Fensterwald 9 years ago9 years ago
You and Nora Carr have raised good questions regarding the significance of the differences in results and the use of gains/losses in learning days, which may be an effort to amplify, in layman's terms, the small variations in standard deviations (.01 equals 7 days of learning growth, etc.) That said, the use of CST results, including high school end of year tests, and matching similar students from the same neighborhoods and feeder schools, are a … Read More
You and Nora Carr have raised good questions regarding the significance of the differences in results and the use of gains/losses in learning days, which may be an effort to amplify, in layman’s terms, the small variations in standard deviations (.01 equals 7 days of learning growth, etc.)
That said, the use of CST results, including high school end of year tests, and matching similar students from the same neighborhoods and feeder schools, are a credible methodology — not subjective like comparing school cultures and other factors. I must say I didn’t hear some of the same objections from charter skeptics/opponents five years ago, when CREDO’s first national study concluded that, on average nationwide, 37 percent of charters produced academic results that were worse than traditional public schools, while only 17 percent performed significantly better. For the rest of schools, there was no significant difference. Those figures were repeated as Gospel, not GIGO.
The only difference now is that updated studies by CREDO has found that charters are performing quite well for some subgroups of students and are producing notable results in some locations like Los Angeles.
CarolineSF 9 years ago9 years ago
Actually, in discussing and blogging about the previous CREDO study, which reflected poorly on charter schools, I often commented on the fact that it had been conducted by a pro-charter organization and thus had extra credibility. And in remarks on the current study I mentioned that regarding the previous study as well, and commented that Margaret "Macke" Raymond of CREDO was ethical enough to release the results of that previous study despite its findings. She … Read More
Actually, in discussing and blogging about the previous CREDO study, which reflected poorly on charter schools, I often commented on the fact that it had been conducted by a pro-charter organization and thus had extra credibility. And in remarks on the current study I mentioned that regarding the previous study as well, and commented that Margaret “Macke” Raymond of CREDO was ethical enough to release the results of that previous study despite its findings. She got a lot of angry pushback from her pro-charter colleagues, especially Caroline Hoxby (whose studies have also been widely misrepresented as impartial academic research, by the way).
So charges of inconsistency are inaccurate and invalid. It’s simply basic journalistic ethics to report that a study comes from a source with a partisan point of view.
Again, I have to note that matched demographics still don’t account for a student population self-selected or actively selected for overcoming certain hurdles or meeting certain requirements, compared with a student population that did not have to overcome those hurdles or meet those requirements.
navigio 9 years ago9 years ago
Whether the conclusions of a study support a political position is independent of the methodology used in the study. This may arguably be even more the case in situations where a study contradicts the political positions of your opponents. I also don't think anybody should be disqualified from critiquing methodology merely because they failed to do so in the past. You go as far as you need to go to make your point. Sometimes it's … Read More
Whether the conclusions of a study support a political position is independent of the methodology used in the study. This may arguably be even more the case in situations where a study contradicts the political positions of your opponents. I also don’t think anybody should be disqualified from critiquing methodology merely because they failed to do so in the past. You go as far as you need to go to make your point. Sometimes it’s not very far at all, other times it has to be quite far.
CarolineSF 9 years ago9 years ago
Journalistic standards and ethics call for disclosure when a study is conducted or funded by a source with a partisan interest in the outcome of the study. That's not about the methodology OR the results of the study; it's just basic standards and ethics. I did comment on all that in relation to the past CREDO study, as noted. I don't have the knowhow to critique methodology, but it seems apparent that the difference in admissions … Read More
Journalistic standards and ethics call for disclosure when a study is conducted or funded by a source with a partisan interest in the outcome of the study. That’s not about the methodology OR the results of the study; it’s just basic standards and ethics.
I did comment on all that in relation to the past CREDO study, as noted.
I don’t have the knowhow to critique methodology, but it seems apparent that the difference in admissions processes between charters and public schools is likely to result in differences in the student population that would confound the results of the study.
navigio 9 years ago9 years ago
Fwiw I was responding to John.
Manuel 9 years ago9 years ago
My point, navigio, in criticizing the "selection" methodology reported by John is that "no student is an island." The student's learning will be affected by her/his environment and simply because the student happens to fit a set of markers does not make her/him equal to others with the same markers. You could argue that point successfully when discussing mass-produced widgets, but not about human beings. To do so is folly, in my opinion. Anyway, what I … Read More
My point, navigio, in criticizing the “selection” methodology reported by John is that “no student is an island.” The student’s learning will be affected by her/his environment and simply because the student happens to fit a set of markers does not make her/him equal to others with the same markers. You could argue that point successfully when discussing mass-produced widgets, but not about human beings. To do so is folly, in my opinion.
Anyway, what I found fascinating about that Air Force Academy social engineering experiment is that segregation of students by prior academic record did not produced the optimum expected by the researchers. In fact, if the “best” solution is chosen, all low achievers are expected to flunk out. Thus, low achievers should never even be given consideration for admission because if kept within the mixed group, they will depress the achievement of the middle group.
That might work for producing the best military officers. But we can’t do that for students in K-12 and expect to have a fully functioning society. But that is what is being done by using charters as escape valves for those dissatisfied with public schools. What are we going to do with the low achievers that get stuck in public schools? Warehouse them as academic throw-aways? No, don’t answer that question right now.
Manuel 9 years ago9 years ago
Thank you for acknowledging that I raised good points. I did not want to get too technical, but allow me to illustrate what a "standard deviation" means in a "real world" example: the temperature of a gas is defined as the first e-folding of its energy distribution, i.e., the first standard deviation since the distribution is the Bell Curve. Thus, a 0.01 change in the standard deviation of a gas is, for all intents and … Read More
Thank you for acknowledging that I raised good points. I did not want to get too technical, but allow me to illustrate what a “standard deviation” means in a “real world” example: the temperature of a gas is defined as the first e-folding of its energy distribution, i.e., the first standard deviation since the distribution is the Bell Curve. Thus, a 0.01 change in the standard deviation of a gas is, for all intents and purposes, non-consequential. Same, I suspect, with these “7 days of learning.”
The fact that these researchers have accepted that as “valid” doesn’t mean that it is equally valid in the real world, just as has been pointed by others in this thread. Is their methodology accepted? Of course it is. But it is accepted by them. They have made the rules. But it doesn’t mean that they truly are describing reality. Yes, they have Ph.D.s, but they are in the “inexact science:” economics. So, yes, I sniff at this because the rules are made up and would be laughed out of town if, say, rocket science was run the same way. Yes, my bias is showing. Sorry.
As for their previous report, well, in my defense I must state that I never ever have cited it. Why? Because I have always felt that there are peculiarities with standardized testing that are never addressed because the majority of people in education are, in effect, bullied by those who wave mathematics around as if they were the Gospel’s Truth. Besides, in the last several years I’ve learned all kinds of things that confirm my visceral leanings: CSTs and most other standardized tests are GIGO.
As for “culture” being a difficult thing to quantify, take a look at this NPR news bit that was broadcast this morning. It talks about what happened to the academic achievement of cadets at the Air Force Academy. Very interesting stuff presenting a very intriguing dilemma.
navigio 9 years ago9 years ago
One thing I noticed that was missing from the matching criteria was parent education level. They did use prior test scores as one of the criteria so perhaps that was intended to be a proxy for parent education level, though I didn't sift through to see how specifically that particular matching was done. It's funny you mention the CST results. The first thing I thought of when I read this headline was exactly how miserably the … Read More
One thing I noticed that was missing from the matching criteria was parent education level. They did use prior test scores as one of the criteria so perhaps that was intended to be a proxy for parent education level, though I didn’t sift through to see how specifically that particular matching was done.
It’s funny you mention the CST results. The first thing I thought of when I read this headline was exactly how miserably the traditional public school system did in elementary and middle this past year. At the time I did ask whether anyone had charter school comparisons but I don’t remember whether we ended up having anything.
Obviously days of learning can’t mean great overlap. Rather what it probably means is performance variation with respect to some grade-specific baseline. I wouldn’t expect possible variation from that to be anywhere near linear, and more notably, it’s model would likely be different for different baselines. I expect that’s part of the problem with the method? I read the last study, but haven’t had a chance to read this one yet.
CarolineSF 9 years ago9 years ago
Thank you for meeting journalistic ethical standards with the disclosure about CREDO's being run by the Hoover Institution, though it also needs to be included in the disclosure that Hoover is a partisan organization that exists to promote free-market values and privatization policies. The study doesn't appear to address the issue of selectivity/self-selection of charter student populations, nor of attrition. In the absence of probing, questioning journalism about KIPP a few years ago, I had to … Read More
Thank you for meeting journalistic ethical standards with the disclosure about CREDO’s being run by the Hoover Institution, though it also needs to be included in the disclosure that Hoover is a partisan organization that exists to promote free-market values and privatization policies.
The study doesn’t appear to address the issue of selectivity/self-selection of charter student populations, nor of attrition. In the absence of probing, questioning journalism about KIPP a few years ago, I had to put my own daughter into the KIPP application process (as an education blogger) to confirm that KIPP schools gave tests to applicants. Confounding factors like that are critical points in evaluating this information.