posted on August 23, 2017 by Tony Monfiletto
Richard Rothstein is a prolific writer about education reform. In Grading Education he documents the effect of perverse incentives on schools and other public institutions, everything from the Soviet Union to the Atlanta Public Schools. The cornerstone of the problem he identifies is the standardized tests we use to measure whether schools are effective. He quotes Donald Cambell in his central argument against them:
The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
He goes on to quote, “Such corruption occurs primarily because … an indicator that can be quantified often reflects only an aspect of the outcome of interest, so undue attention to this aspect will distort the balance of services being provided.”
This theory was supported by Linda Darling Hammond during a speech at the UNESCO conference in Paris. In her presentation she described perverse incentives from the reliance on standardized tests:
- The incentive is to serve students that are able to perform well on standardized tests. This creates a perverse incentive to force some students out of school or discourage them from enrolling. We see this with parent contracts, absence of school lunch and transportation services, and discipline/behavior policies that push non-compliant kids out of school.
- There is a tendency to broaden or narrow curriculum to focus on only the skills that are tested. Reading and Math are tested while higher order thinking skills, physical fitness, and emotional intelligence are not. Time is then shifted to what the state cares most about.
- Evaluating teachers on student performance builds incentives for teachers to be lone actors. Their job is at stake so they promote their success over or their colleague’s or school’s success.
- Over reliance on test scores focuses on evaluation rather than development. The attention and resources of the leadership is spent on judging whether teachers are getting results rather than building competency or on school-wide goals that require collaboration.
We have a few framing questions for our research:
1. What are the metrics that do not distort a school’s mission?
The Capacity Problem
The presence of perverse incentives is an effect of the constraints on policy makers and regulators. Our current metrics are driven almost entirely by the number (or proportion) of students that are proficient in reading or math in any school. This data is vital because it is the only objective information that can be assembled and applied across all schools. The sheer number of schools, students, and teachers is so overwhelming that we are left with the most basic of all assessments because it is all that we can comprehend. I am sympathetic to their challenge. It’s difficult to imagine any objective bureaucratic system that could develop a nuanced understanding of every institution and its unique context. It’s particularly challenging if you assume that the regulator is accountable to elected officials that expect objective information that can help them make decisions that serve the public interest.
We now find ourselves with an imperfect and blunt instrument that is unable to account for any school’s mission particular context. Unfortunately, standardized test scores now dominate any consideration of school quality and it is widely published and foundations use it to make their funding decisions and families use it to determine which school is best for their children.
The table has now been set and we have a simple and straight forward way to judge our schools. However, we are much less clear about what was sacrificed to get he scores that we desire. Critical thinking, art, science, social studies, physical education and a whole world of valuable experiences have been devalued in order to deliver on “accountability.” Even more troubling is that the drum beat for higher reading and math scores has drowned out any vision for the future of school. We cannot imagine a new frame of reference for school because we are bound by a set of metrics that is dictated by what we can measure.
2. Would a mutual accountability model where schools evaluate each other help develop capacity?
The Objectivity Problem
I also believe that regulators and funders fear that they will lose their objectivity if they engage too deeply with schools. They may be reluctant to expect qualitative evidence of success because it is soft and unreliable and they may have to use their judgment. Also, there is little patience for a school that is pursuing an institutional development strategy that will not bear fruit for years. If a regulator or funder considered school improvement with an institutional development frame, they would be skeptical of quick fixes— singular-one off tactics like canned curriculum or one-off after school programs. More importantly, they would be skeptical of schools that do not have mission driven strategies to improve their institutions. All worthwhile missions are subjective and regulators might fear the perception that they had a stake in the values that are at the core. Not only would they compromise their objectivity, they might also be implicated if the school fails.
When I met with the Secretary of Education in New Mexico in 2012, she was clear that her job was to judge performance, “I can’t engage in the inputs conversation. My job is to hold schools accountable of the outputs.” I got the sense from her that considering inputs (and context) would compromise her objectivity. In the end, she would be judged based on the data that she could show her boss, the governor, and the best she could do was demand better reading and math scores.
3. What is the right balance between embracing the values inherent in a school’s mission and holding it accountable for student learning?
The Tradeoff: Values vs. Metrics
As the education sector has evolved, schools have become increasingly specialized. They are developed with unique missions and districts have a growing preference for a diverse portfolio of choices in their communities. Intuitively, policy makers know that kids and communities differ. Similarly, foundations search out new ideas that can demonstrate innovation. Even our most conservative school districts point to innovation as the effort to create breakout schools and in Albuquerque we see this happening at an increasing rate (Next Gen housed at Del Norte High School, International Baccalaureate at Sandia High School, Early College at Central NM Community College). Meanwhile, the McCune and Kellogg Foundations and Daniels Fund make the bulk of their education grants to new start-up school. While government and the philanthropic community want innovation, they have defaulted to one dimensional metrics that assume all schools and children are the same.
Last month I visited Denver and I saw mostly traditional approaches to schooling even in schools that received the “Innovation” or charter designation which entitled them to waivers from nearly all state laws or local district policies. Like in most places the accountability system was focused on judging outputs and I spoke to many school leaders with one point of reference—the test scores that are published by the central office. They were asked to innovate, yet school leaders first looked to the regulators and asked “What are they looking for?” before asking “what are we trying to accomplish”? The school leader’s capacity to think differently was stifled by the overwhelming influence of standardized testing and the subtext that we need better versions of what we already have. Henry Ford put it best when he said the following about innovation, “If I would have asked people what they wanted, they would have asked for faster horses.”
4. What are the cross-cutting incentives that would propel innovation at our schools?
The New Metrics
Any new accountability system that we consider must account for the circumstance of each school and the create incentives to help drive mission-driven improvements in their organizations. We shouldn’t think about carrots and sticks, blame and praise, or other incentives that simply push our schools toward short-term fixes. Why would we want schools to develop programs that compensate for systems that are fundamentally broken? Incentives matter, and often policy makers and funders get them wrong because they do not account for the decisions at the district or school level that actually make our schools more reactive and less strategic.
Collaboration as an Example:
If we want a school to make a lasting improvement in student performance, then it must be a high functioning organization. There is no shortage of literature about “learning organizations” and their ability to adapt to future challenges and know and understand the needs of their clients. If one assumes that learning is a key element to highly effective organizations, then we can assume that the command and control structures of our current system are incapable of the adaptability needed to succeed in a fast changing environment. Collaboration is a fundamental building block to make this happen and I wonder if we can create accountability measures that incentivize it.
The Unselfish Gene, published in the Harvard Business Review by Yochai Benkler attempts to de-bunk the theory that people are hard-wired to act in their own interests. For example, he would argue that incentive pay for higher test scores would actually undermine success for the school overall. I chose this article because it points out the conditions that must be present for collaboration in mission driven organizations. I’m asserting that any effective school would have a purpose at the core of its work and an evaluation of school quality might look for these organizational attributes or incentives to create them:
- Communication, “people who can communicate are more empathetic and trusting, and they can reach solutions more readily than when they don’t talk to one another.
- Framing an Authentic Purpose, “framing can motivate, but people aren’t stupid and it’s important that the frame fit the reality”
- Empathy and Solidarity, “empathy for those affected by our actions alters the outcomes we all care about, and that, in turn, changes our behavior”
- Rewards and punishment, “do not turn the social interaction into a commercial exchange”
- Fairness and morality, “clearly defined values are crucial to cooperation; discussing, explaining, and reinforcing the right thing to do will increase the degree to which people behave that way.”
- Reciprocity, “systems relying on the pay it forward kind of reciprocity are enormously valuable”
- Diversity, “because we differ from one another, cooperative systems have to be flexible.”
Framing Question for our Research, “Does this mean we have to measure “collaboration”? Do we also need to measure the validity of the purpose of the collaboration? What are they collaborating on?”
Our work over the next year should be to develop accountability systems that propel our schools forward. Accountability could be re-framed from asking “what is the evidence of your actions” to “what are you doing to improve your organization.” This would be a forward looking stance that could at least give balance to our output dominated discussion.
I wonder if we could talk about the questions I’ve listed in each section? What other questions are there? What’s the next dimension of our thinking/where do we go from here?