Incorporating social-emotional learning into school accountability

EdSource_RedesigningAccountability_v2.2

The CORE Districts began in 2010 as a collaboration across school districts exploring ways to improve teaching and learning. In 2013, several school districts in the CORE consortium received a federal waiver from some provisions of the No Child Left Behind law and are working together to develop a new School Quality Improvement Index to provide more and better information about schools and the learning needs of students. The process for creating the Index has been challenging and complex, including finding the best way to measure social and emotional factors. In this piece Noah Bookman, the chief accountability officer for CORE Districts, explains how they dealt with this issue.

Part Four: Measuring Social-Emotional Skills

Noah Bookman, CORE chief accountability officer

Noah Bookman, CORE chief accountability officer

Social and emotional learning (SEL) is the process through which children and adults develop the ability to manage their emotions, achieve goals, show empathy, build relationships and make responsible decisions. Many educators, researchers and education policy makers have come to believe that positive social and emotional skills are critical to student success.

At CORE Districts, we agree, and are the first in the nation to include the measurement of social and emotional factors in a system of school improvement and accountability – our new School Quality Improvement Index. We have done so because we believe they offer schools and educators more and better information that furthers understanding of students and what they need to learn and succeed. This information can be used to inform and shape strategies to help students succeed in school and prepare them for success in college, careers and life.

Beyond narrow measures of academic achievement, measurement of social and emotional well-being offers schools another way to assess their efforts to help students. Measurement of indicators such as student attitudes toward personal growth and effectiveness may offer schools important clues about classroom instruction, classroom environment, and school climate and culture that can inform and guide efforts to improve them.

Taking on this groundbreaking work has not been easy. Our first challenge in designing the social-emotional measures of the School Quality Improvement Index was to determine which social-emotional skills to prioritize. For assistance, we turned to experts from Transforming Education, a nonprofit specializing in helping schools assess and develop students’ social-emotional skills.

 

Transforming Education helped us narrow down the choices by focusing on social-emotional skills with the greatest potential to impact student outcomes. They helped us identify skills that are “meaningful,” “measurable” and “actionable.” A meaningful skill is predictive of success in college, career and life. Measurable skills are those that can be measured in reliable, valid ways within schools. Actionable skills are those that can be strengthened in a school setting.

The work with Transforming Education was enormously valuable and made clear the challenges we faced. But their staff cautioned about including social-emotional measures in a school accountability system due to potential flaws in the existing tools used to measure social and emotional learning. They warned about misinterpretation of questions on surveys (which are the existing method of social-emotional skill measurement). Another caution was in regard to reference bias, whereby different groups may rate themselves differently on the same scale due to different understanding of given skills. These would all be threats to the quality of the measurement.

We carefully considered the reality that no perfect measure exists, but ultimately decided to try to include social-emotional skills in the School Quality Improvement Index. We think there are some good reasons to do so.

First, because our Index is about highlighting all the areas that matter for preparing students to be college and career ready, we remained steadfast in our pursuit to highlight essential skills that both research and our experience as educators suggest are as important as academic skills. While there may be some risk, and we recognize that social-emotional skills are complements, not substitutes for other important measures like academic achievement and graduation, we see these indicators as adding important perspective to understand what’s happening in the lives of students and in our schools, and we think these measures honor teachers who are doing work in this area every day.

Additionally, the “accountability” that CORE embodies is a far cry from the high-stakes punitive systems that often come to mind when one thinks of testing and accountability. Our School Quality Improvement System emphasizes the “right drivers” with the best available information. We’re developing a system of multiple measures that serves as a tool to identify strengths and challenges, rather than using accountability as a hammer, as has been done with the No Child Left Behind law.

In late 2013, school districts in CORE met to narrow down the long list of meaningful, measurable and actionable skills, and determined which were most essential and significant for students. During this session, policy, assessment and curriculum experts across all our districts made recommendations.

The CORE Districts Board, consisting of CORE’s 10 superintendents, agreed to focus on the following as the initial set of competencies to include in our School Quality Improvement Index:

  • Growth Mindset: A belief one’s abilities can grow with effort.
  • Self-Efficacy: A belief in one’s own ability to achieve an outcome or reach a goal.
  • Self-Management: The ability to regulate one’s emotions, thoughts and behaviors in different situations.
  • Social Awareness: The ability to understand others’ perspectives and empathize with them.

More comprehensive definitions can be found here. 

They also agreed to measure these competencies using common survey-based measures across all school districts participating in the School Quality Improvement System. Transforming Education helped us source the best survey measures for each competency from the leading researchers in the field.

Because we would be breaking new ground by measuring these skills on such a large scale, CORE developed a spring 2014 pilot study to address the measurement, communications and timing of the SEL Survey, as well as supports and interventions. Eighteen schools across six districts participated in this pilot, enabling us to collect approximately 9,000 student self-reports and 15,500 teacher reports on students’ competencies. Students responded to items like the following:

  • During the past 30 days, how often did you come to class prepared? (Self Management)
  • How true is this statement?: My intelligence is something that I can’t change very much. (Growth Mindset)
  • How true is this statement?: I can do well on all my tests, even when they’re difficult. (Self-Efficacy)
  • During the past 30 days, when others disagreed with you, how respectful were you of their views? (Social Awareness)

Teachers were also provided with descriptions of self-management and social awareness – two observable skills – and were asked to rate students on a scale from one to five. The findings of the pilot test were promising. In partnership with Harvard’s Center for Education Policy Research, we found that both the student self-reported results and teacher reports on students met several important tests of measurement quality. The findings told us that the instruments were measuring distinct skills, that they were doing so in a reliable fashion, and that these social-emotional skills are related to other outcomes of interest such as test scores, grades, discipline and attendance. We used the findings from the pilot study to make data-driven decisions about which social-emotional measures to use across all of the CORE schools in the 2014-15 school year.

While teacher reports on students were found to be more predictive of student performance than student self-reports, we ultimately decided that teacher reports on students’ social-emotional competencies would be optional for districts due to local concerns about the feasibility of having all teachers participate. (It is important to note that teachers already rate their students on behavior when they complete report cards, and I personally believe that traditional behavior and work habits marks should be replaced by measures of social-emotional skills.)

With the guidance of our partners at Transforming Education and Harvard, we continued to refine the survey and conducted a CORE districtwide field test in spring 2015. To our knowledge, it was the largest effort ever to measure students’ social-emotional skills. Almost half a million students participated in the self-report surveys, and just over 70,000 students in Fresno and Santa Ana participated in a substantial effort to have teachers assess their students for social-emotional skills.

Part of the goal of this field test was to gauge how some of the concerns expressed by the Transforming Education advisory board would play out. We’re currently analyzing the spring 2015 data and assessing any unexpected correlations that would raise red flags or indicate reference bias. We are planning to release our findings later this year. We believe the initial findings are promising. We are gauging the relationships between school-level social-emotional learning results and other measures of school quality such as academic achievement, chronic absence and suspension rates. And so far, we are also finding that higher school social-emotional ratings are correlating with higher school-level achievement, and lower levels of suspension and chronic absence.

Over the coming months, CORE will determine how to calculate and report on the data for the purposes of the School Quality Improvement Index score. We will also consider whether having teachers complete surveys is truly feasible given the workload involved in rating all students. And we’ll decide whether to report an overall social-emotional rating for each school or provide separate scores for individual skills (such as is reported for math and ELA) on growth mindset, self management, self-efficacy and social awareness. We expect to make these decisions this winter, based in large part upon the data and analysis from our partners at the Center for Education Policy Research at Harvard University.

As a learning organization we understand that we are working at the leading edge of innovation in deploying measures of social-emotional skills as part of our accountability index, and that there are risks to that effort. While we do not know if including these skills in our Index will work, we do know that fostering these skills in youth matters. We also believe that if these indicators are only secondary considerations beyond test scores and other academic outcomes that schools have traditionally focused on, they will continue to be thought of as “nice to have” rather than a “must have” of American schooling.

•••

Noah Bookman is the chief accountability officer for the CORE Districts.  He previously served as the director of performance management for the Los Angeles Unified School District.

The opinions expressed in this commentary represent those of the author. EdSource welcomes commentaries representing diverse points of view. If you would like to submit a commentary, please review our guidelines and contact us.

EdSource in your inbox!

Stay ahead of the latest developments on education in California and nationally from early childhood to college and beyond. Sign up for EdSource’s no-cost daily email.

Subscribe