EdSource_RedesigningAccountability_v2.2

The CORE Districts began in 2010 as a collaboration across school districts exploring ways to improve teaching and learning. In 2013, several school districts in the CORE consortium received a federal waiver from some provisions of the No Child Left Behind law and are working together to develop a new School Quality Improvement Index to provide more and better information about schools and the learning needs of students. The process for creating the Index has been challenging and complex, including finding the best way to measure social and emotional factors. In this piece Noah Bookman, the chief accountability officer for CORE Districts, explains how they dealt with this issue.

Part Four: Measuring Social-Emotional Skills

Noah Bookman, CORE chief accountability officer

Noah Bookman, CORE chief accountability officer

Social and emotional learning (SEL) is the process through which children and adults develop the ability to manage their emotions, achieve goals, show empathy, build relationships and make responsible decisions. Many educators, researchers and education policy makers have come to believe that positive social and emotional skills are critical to student success.

At CORE Districts, we agree, and are the first in the nation to include the measurement of social and emotional factors in a system of school improvement and accountability – our new School Quality Improvement Index. We have done so because we believe they offer schools and educators more and better information that furthers understanding of students and what they need to learn and succeed. This information can be used to inform and shape strategies to help students succeed in school and prepare them for success in college, careers and life.

Beyond narrow measures of academic achievement, measurement of social and emotional well-being offers schools another way to assess their efforts to help students. Measurement of indicators such as student attitudes toward personal growth and effectiveness may offer schools important clues about classroom instruction, classroom environment, and school climate and culture that can inform and guide efforts to improve them.

Taking on this groundbreaking work has not been easy. Our first challenge in designing the social-emotional measures of the School Quality Improvement Index was to determine which social-emotional skills to prioritize. For assistance, we turned to experts from Transforming Education, a nonprofit specializing in helping schools assess and develop students’ social-emotional skills.

 

Transforming Education helped us narrow down the choices by focusing on social-emotional skills with the greatest potential to impact student outcomes. They helped us identify skills that are “meaningful,” “measurable” and “actionable.” A meaningful skill is predictive of success in college, career and life. Measurable skills are those that can be measured in reliable, valid ways within schools. Actionable skills are those that can be strengthened in a school setting.

The work with Transforming Education was enormously valuable and made clear the challenges we faced. But their staff cautioned about including social-emotional measures in a school accountability system due to potential flaws in the existing tools used to measure social and emotional learning. They warned about misinterpretation of questions on surveys (which are the existing method of social-emotional skill measurement). Another caution was in regard to reference bias, whereby different groups may rate themselves differently on the same scale due to different understanding of given skills. These would all be threats to the quality of the measurement.

We carefully considered the reality that no perfect measure exists, but ultimately decided to try to include social-emotional skills in the School Quality Improvement Index. We think there are some good reasons to do so.

First, because our Index is about highlighting all the areas that matter for preparing students to be college and career ready, we remained steadfast in our pursuit to highlight essential skills that both research and our experience as educators suggest are as important as academic skills. While there may be some risk, and we recognize that social-emotional skills are complements, not substitutes for other important measures like academic achievement and graduation, we see these indicators as adding important perspective to understand what’s happening in the lives of students and in our schools, and we think these measures honor teachers who are doing work in this area every day.

Additionally, the “accountability” that CORE embodies is a far cry from the high-stakes punitive systems that often come to mind when one thinks of testing and accountability. Our School Quality Improvement System emphasizes the “right drivers” with the best available information. We’re developing a system of multiple measures that serves as a tool to identify strengths and challenges, rather than using accountability as a hammer, as has been done with the No Child Left Behind law.

In late 2013, school districts in CORE met to narrow down the long list of meaningful, measurable and actionable skills, and determined which were most essential and significant for students. During this session, policy, assessment and curriculum experts across all our districts made recommendations.

The CORE Districts Board, consisting of CORE’s 10 superintendents, agreed to focus on the following as the initial set of competencies to include in our School Quality Improvement Index:

  • Growth Mindset: A belief one’s abilities can grow with effort.
  • Self-Efficacy: A belief in one’s own ability to achieve an outcome or reach a goal.
  • Self-Management: The ability to regulate one’s emotions, thoughts and behaviors in different situations.
  • Social Awareness: The ability to understand others’ perspectives and empathize with them.

More comprehensive definitions can be found here. 

They also agreed to measure these competencies using common survey-based measures across all school districts participating in the School Quality Improvement System. Transforming Education helped us source the best survey measures for each competency from the leading researchers in the field.

Because we would be breaking new ground by measuring these skills on such a large scale, CORE developed a spring 2014 pilot study to address the measurement, communications and timing of the SEL Survey, as well as supports and interventions. Eighteen schools across six districts participated in this pilot, enabling us to collect approximately 9,000 student self-reports and 15,500 teacher reports on students’ competencies. Students responded to items like the following:

  • During the past 30 days, how often did you come to class prepared? (Self Management)
  • How true is this statement?: My intelligence is something that I can’t change very much. (Growth Mindset)
  • How true is this statement?: I can do well on all my tests, even when they’re difficult. (Self-Efficacy)
  • During the past 30 days, when others disagreed with you, how respectful were you of their views? (Social Awareness)

Teachers were also provided with descriptions of self-management and social awareness – two observable skills – and were asked to rate students on a scale from one to five. The findings of the pilot test were promising. In partnership with Harvard’s Center for Education Policy Research, we found that both the student self-reported results and teacher reports on students met several important tests of measurement quality. The findings told us that the instruments were measuring distinct skills, that they were doing so in a reliable fashion, and that these social-emotional skills are related to other outcomes of interest such as test scores, grades, discipline and attendance. We used the findings from the pilot study to make data-driven decisions about which social-emotional measures to use across all of the CORE schools in the 2014-15 school year.

While teacher reports on students were found to be more predictive of student performance than student self-reports, we ultimately decided that teacher reports on students’ social-emotional competencies would be optional for districts due to local concerns about the feasibility of having all teachers participate. (It is important to note that teachers already rate their students on behavior when they complete report cards, and I personally believe that traditional behavior and work habits marks should be replaced by measures of social-emotional skills.)

With the guidance of our partners at Transforming Education and Harvard, we continued to refine the survey and conducted a CORE districtwide field test in spring 2015. To our knowledge, it was the largest effort ever to measure students’ social-emotional skills. Almost half a million students participated in the self-report surveys, and just over 70,000 students in Fresno and Santa Ana participated in a substantial effort to have teachers assess their students for social-emotional skills.

Part of the goal of this field test was to gauge how some of the concerns expressed by the Transforming Education advisory board would play out. We’re currently analyzing the spring 2015 data and assessing any unexpected correlations that would raise red flags or indicate reference bias. We are planning to release our findings later this year. We believe the initial findings are promising. We are gauging the relationships between school-level social-emotional learning results and other measures of school quality such as academic achievement, chronic absence and suspension rates. And so far, we are also finding that higher school social-emotional ratings are correlating with higher school-level achievement, and lower levels of suspension and chronic absence.

Over the coming months, CORE will determine how to calculate and report on the data for the purposes of the School Quality Improvement Index score. We will also consider whether having teachers complete surveys is truly feasible given the workload involved in rating all students. And we’ll decide whether to report an overall social-emotional rating for each school or provide separate scores for individual skills (such as is reported for math and ELA) on growth mindset, self management, self-efficacy and social awareness. We expect to make these decisions this winter, based in large part upon the data and analysis from our partners at the Center for Education Policy Research at Harvard University.

As a learning organization we understand that we are working at the leading edge of innovation in deploying measures of social-emotional skills as part of our accountability index, and that there are risks to that effort. While we do not know if including these skills in our Index will work, we do know that fostering these skills in youth matters. We also believe that if these indicators are only secondary considerations beyond test scores and other academic outcomes that schools have traditionally focused on, they will continue to be thought of as “nice to have” rather than a “must have” of American schooling.

•••

Noah Bookman is the chief accountability officer for the CORE Districts.  He previously served as the director of performance management for the Los Angeles Unified School District.

The opinions expressed in this commentary represent those of the author. EdSource welcomes commentaries representing diverse points of view. If you would like to submit a commentary, please review our guidelines and contact us.


Comments

Leave a Comment

Your email address will not be published. Required fields are marked *


Comments Policy

The goal of the comments section on EdSource is to facilitate thoughtful conversation about content published on our website. Click here for EdSource's Comments Policy.

Expand Comments
Collapse Comments
  1. Don 7 months ago7 months ago

    The commentary isn't about social-emotional learning per se. It's about measuring social-emotional learning. It isn't made clear what it is that schools and teachers will do during the limited school day and year to foster this kind of learning with all the other challenges at hand. I would like to have heard more about what the teaching of it constitutes and how it would take place, rather than exclusively how it's measured. Many students of … Read More

    The commentary isn’t about social-emotional learning per se. It’s about measuring social-emotional learning. It isn’t made clear what it is that schools and teachers will do during the limited school day and year to foster this kind of learning with all the other challenges at hand. I would like to have heard more about what the teaching of it constitutes and how it would take place, rather than exclusively how it’s measured. Many students of great intellectual prowess may measure up poorly on a social-emotional assessment. Is it the role of public education to change normal variants of human behavior or is to teach students the fundamentals of the basic fields of knowledge? It is a slippery slope when you try to redefine what is “important” learning.

  2. Richard Cohen 7 months ago7 months ago

    We at the Restorative Schools Vision Project have been implementing SEL in schools since 2007 with great success. See our web site at restorativeschoolsproject.org for more information on how we do this.

  3. Laura H. Chapman 7 months ago7 months ago

    It is clear that no one putting this policy in place has read or digested this best-in-class research on WHY measures of SEL are not ready for prime time, and least of all for school accountability. Also, you should look at the two california grants from USDE for the very low threshold for evaluation of their first SEL grants. Strongly recommend this: Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes Angela L. … Read More

    It is clear that no one putting this policy in place has read or digested this best-in-class research on WHY measures of SEL are not ready for prime time, and least of all for school accountability. Also, you should look at the two california grants from USDE for the very low threshold for evaluation of their first SEL grants. Strongly recommend this:
    Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes
    Angela L. Duckworth1 and David Scott Yeager2
    Educational Researcher, Vol. 44 No. 4, pp. 237– 251
    DOI: 10.3102/0013189X15584327
    © 2015 AERA. http://er.aera.net

  4. Jonathan Raymond 7 months ago7 months ago

    Building social and emotional learning into how we measure children’s success in schools is a bold and necessary step if we are to prepare children for college, careers and life. What gets measured gets done. More important, however, are assisting and supporting our teachers and educators to be able to utilize the practices that help our children gain these critical skills.

  5. Melanie 7 months ago7 months ago

    Important for sure but I'm not certain about adding to the CORE which is presenting enough challenges right now. Maybe improve math and science first - then look at additions (or changes). As validated by research, emotions in particular are typically easier for the female brain to navigate (speaking of assigned gender). This is a nice piece to add but may have components that result in a potential for gender bias inherent (and maturation) and … Read More

    Important for sure but I’m not certain about adding to the CORE which is presenting enough challenges right now. Maybe improve math and science first – then look at additions (or changes). As validated by research, emotions in particular are typically easier for the female brain to navigate (speaking of assigned gender). This is a nice piece to add but may have components that result in a potential for gender bias inherent (and maturation) and normally such additional education would be recommended for a diagnosis on the spectrum or something similar where life skills are two grade levels below what is age/grade level appropriate.

Template last modified: