Researchers who examined data from five large California school districts have concluded that measures of students’ personal strengths and interpersonal skills are not reliable enough at this point to include in states’ and districts’ school accountability systems.
In a report last month, the research nonprofit Policy Analysis for California Education, or PACE, concluded that surveys of students about their habits of mind, like self-control, are useful and can point to schools that are succeeding in developing individual and interpersonal skills. But the study said that scores vary too much from year to year within schools and grades to be included with more traditional measures — standardized test scores and graduation rates — that are included on statewide grading systems like the California School Dashboard.
The authors are Hans Fricke, the director of Quantitative Research of the CORE-PACE Research Partnership; Susanna Loeb, former Stanford University professor of education who directs the Annenberg Institute for School Reform at Brown University; and Heather Hough, PACE’s executive director. They looked at three years of survey results of five unified school districts: Los Angeles, Long Beach, Fresno, San Francisco and Santa Ana
Rick Miller, executive director of CORE, a nonprofit data collaborative of California school districts that developed the study’s annual student surveys, said he was unfazed by PACE’s finding. Neither CORE nor the State of California planned to use the results to judge school performance. More importantly, he said, the PACE analysis reiterated that students’ soft skills like self-confidence matter: They can have a positive impact on student success and some schools are consistently good at developing them.
“We will keep investing in this learning,” he said. “We are on to something and will continue to look at these results.”
New frontier of education research
CORE has been in the vanguard of what’s broadly called social and emotional learning since it was founded by eight California school districts. CORE was the umbrella organization that in 2013 received a waiver from the federal No Child Left Behind Act to experiment with new accountability measures. Now, its four-year collection of survey data from more than 1 million students is “the largest measure of social emotional learning in the galaxy,” he told a webinar on the PACE study.
Research into social and emotional learning has burgeoned as California and other states broadened their test-based accountability systems to include measures of school climate and culture out of recognition that they are foundational for student achievement. These include rates of student suspensions and chronic absenteeism.
A national survey this year of 15,000 teachers and 3,500 principals by the RAND Corporation found that 72 percent of principals said that promoting social and emotional skills was their top or one of their top priorities, with more principals in high-poverty schools rating it as their top priority. More than 80 percent of teachers said that social and emotional learning programs can improve school climate and student behavior; 64 percent said school achievement can be improved as well.
About two-dozen districts are now using CORE’s annual survey of student perceptions of their own social and emotional learning skills, and many are delving into the differences in results by ethnicity, race and gender. PACE has established a research partnership with CORE to explore how scores should be calculated and reported and how educators can use social and emotional indicators to improve schools.
CORE surveys go to all students in grades 4 to 12. Students are asked to respond to statements that describe their views on four dimensions of social and emotional learning:
Self-management or self-control: the ability to regulate your emotions, thoughts and behaviors effectively in different situations. Survey prompts include: “I paid attention even when there were distractions” and “I stayed calm even when others bothered or criticized me.”
Growth mindset: the belief that you can view mistakes as opportunities and develop your skills through effort, practice and perseverance. Survey prompts include: “Challenging myself won’t make me any smarter” and “If I am not naturally smart in a subject, I will never do well in it.”
Self-efficacy: the belief you can achieve an outcome or goal and reflects confidence in the ability to exert control over your motivation, behavior and environment. Prompts include: “I can meet all of the learning goals my teacher sets” and “I can do well on all my tests, even when they’re difficult.”
Social awareness: the ability to take the perspective of and empathize with others from diverse backgrounds and cultures and to understand ethical norms for behavior. Questions include: “During the past 30 days, how well did you get along with students who are different from you?” and “During the past 30 days, how clearly were you able to express your feelings?”
All of the measures, particularly growth mindset and self-efficacy, are related, while gauging distinct skills. Growth mindset is the only skill that grew among students from 4th grade through high school, while average scores for the other three declined, starting in middle school. Self-management then plateaued in high school, self-efficacy rebounded slightly in 12th grade, but the mean social awareness scores declined every year after 5th grade.
Susan Ward-Roncalli, social emotional learning adviser and 30-year veteran teacher for Los Angeles Unified, said the drop in social awareness scores reflects students’ awareness of gender and sexuality issues as they reach puberty. Tensions emerge with people who are different from them.
Self-efficacy takes a hit in middle school, as students discover more complex tasks and material and realize they have to figure things out on their own and take charge of their learning, Ward-Roncalli said. In response, Los Angeles Unified is developing materials on self-efficacy that encourage teachers to break down learning into steps and not to see themselves as “a magician behind the curtain” who has all of the answers, she said.
“Self-efficacy drops for girls and never recovers,” Miller said. “We need to pay attention to that.”
Testing CORE’s methodology
CORE developed a model that uses the scores on each social and emotional measure of students in a school and compares those to scores of similar students in other schools, taking into account gender, race and ethnicity and other demographic characteristics. The variations in scores may accurately reflect the impact that a school and its teachers had on social and emotional growth. But that assumes that CORE’s model for predicting improvement produces consistent and reliable schoolwide scores.
In its latest study, PACE found that scores vary substantially for the same schools from one year to the next. This “lack of stability” raises doubts that survey-based social and emotional learning measures can consistently identify true differences in school performance over time, the researchers wrote, while acknowledging that a longer study might show more consistency.
Nonetheless, Miller and others were encouraged that the study did identify a significant number of schools with consistent scores in all four social and emotional learning measures at the top and bottom ends of the scale. Miller agreed that much could be learned from these schools’ practices and experiences that produced high scores. “There was an impact on growth — they did something different that is valuable,” he said.
As with others who see great promise from social and emotional learning, CORE’s goal is to measure what school strategies can improve students’ self-management and other skills and then determine to what extent the growth can contribute to improvement in other areas of achievement, from suspension rates to test scores.
Meanwhile, districts in the CORE data collaborative are providing schools with the survey results to identify gaps in performance among subgroups.
Los Angeles Unified has made it easy to look up the results for every school by grade and subgroup and compare with district averages. The district is providing online programs for middle school students, including videos on mindfulness practices and self-calming and the impact of growth mindsets on brain development.
Corona-Norco Unified in Riverside County, which has given the survey for two years, is piloting work at a middle school, looking at the relationship between the dip in growth mindset and struggles in math, said Jennifer Bourgeois, director of school improvement. The district is having conversations with teachers about their interactions with students.
“We’re seeing some preliminary information in students’ belief in themselves and corresponding growth on the academic side,” she said.