Paperback(Reprint)
-
PICK UP IN STORECheck Availability at Nearby Stores
Available within 2 business hours
Related collections and offers
Overview
Today, organizations of all kinds are ruled by the belief that the path to success is quantifying human performance, publicizing the results, and dividing up the rewards based on the numbers. But in our zeal to instill the evaluation process with scientific rigor, we've gone from measuring performance to fixating on measuring itself—and this tyranny of metrics now threatens the quality of our organizations and lives. In this brief, accessible, and powerful book, Jerry Muller uncovers the damage metrics are causing and shows how we can begin to fix the problem. Filled with examples from business, medicine, education, government, and other fields, the book explains why paying for measured performance doesn't work, why surgical scorecards may increase deaths, and much more. But Muller also shows that, when used as a complement to judgment based on personal experience, metrics can be beneficial, and he includes an invaluable checklist of when and how to use them. The result is an essential corrective to a harmful trend that increasingly affects us all.
Product Details
| ISBN-13: | 9780691191911 |
|---|---|
| Publisher: | Princeton University Press |
| Publication date: | 04/30/2019 |
| Edition description: | Reprint |
| Pages: | 248 |
| Sales rank: | 391,863 |
| Product dimensions: | 5.40(w) x 8.30(h) x 0.70(d) |
About the Author
Read an Excerpt
CHAPTER 1
THE ARGUMENT IN A NUTSHELL
There is a cultural pattern that has become ubiquitous in recent decades, engulfing an ever-widening range of institutions. Depending on taste, one could call it a cultural "meme," an "épistème," a "discourse," a "paradigm," a "self-reinforcing rhetorical system," or simply a fashion. It comes with its own vocabulary and master terms. It affects the way in which people talk about the world, and thus how they think about the world and how they act in it. For convenience, let's call it metric fixation.
A key premise of metric fixation concerns the relationship between measurement and improvement. There is a dictum (wrongly) attributed to the great nineteenth-century physicist Lord Kelvin: "If you cannot measure it, you cannot improve it." In 1986 the American management guru, Tom Peters, embraced the motto, "What gets measured gets done," which became a cornerstone belief of metrics. In time, some drew the conclusion that "anything that can be measured can be improved."
When proponents of metrics advocate "accountability," they tacitly combine two meanings of the word. On the one hand, to be accountable means to be responsible. But it can also mean "capable of being counted." Advocates of "accountability" typically assume that only by counting can institutions be truly responsible. Performance is therefore equated with what can be reduced to standardized measurements. When proponents of metrics demand "transparency" they often insinuate that probity requires making explicit and visible as much information as possible. The result is the demand for ever more documentation, ever more mission statements, ever more "goal-setting."
The key components of metric fixation are
* the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics);
* the belief that making such metrics public (transparent) assures that institutions are actually carrying out their purposes (accountability);
* the belief that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay-for-performance) or reputational (rankings).
Metric fixation is the persistence of these beliefs despite their unintended negative consequences when they are put into practice. It occurs because not everything that is important is measureable, and much that is measurable is unimportant. (Or, in the words of a familiar dictum, "Not everything that can be counted counts, and not everything that counts can be counted.") Most organizations have multiple purposes, and that which is measured and rewarded tends to become the focus of attention, at the expense of other essential goals. Similarly, many jobs have multiple facets, and measuring only a few aspects creates incentives to neglect the rest. When organizations committed to metrics wake up to this fact, they typically add more performance measures — which creates a cascade of data, data that becomes ever less useful, while gathering it sucks up more and more time and resources.
In the process, the nature of work is transformed in ways that are often pernicious. Professionals tend to resent the impositions of goals that may conflict with their vocational ethos and judgment, and thus morale is lowered. Almost inevitably, many people become adept at manipulating performance indicators through a variety of methods, many of which are ultimately dysfunctional for their organizations. They fudge the data or deal only with cases that will improve performance indicators. They fail to report negative instances. In extreme cases, they fabricate the evidence.
A frequent feature of metric fixation is paying for performance, that is, offering individuals or organizations financial incentives to meet quantifiable criteria. That may work in organizations that exist for the single purpose of making a profit, though as we'll see, even in these cases it is rarely effective. It works even less well in organizations in which employees are oriented to a more idealistic mission, such as schools, universities, medical practices, and hospitals. Whenever reward is tied to measured performance, metric fixation invites gaming.
Because the theory of motivation behind pay for measured performance is stunted, results are often at odds with expectations. The typical pattern of dysfunction was formulated in 1975 by two social scientists operating on opposite sides of the Atlantic, in what appears to have been a case of independent discovery. What has come to be called "Campbell's Law," named for the American social psychologist Donald T. Campbell, holds that "[t]he more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." In a variation named for the British economist who formulated it, we have Goodhart's Law, which states, "Any measure used for control is unreliable." To put it another way, anything that can be measured and rewarded will be gamed. We will see many variations on this theme.
Trying to force people to conform their work to preestablished numerical goals tends to stifle innovation and creativity — valuable qualities in most settings. And it almost inevitably leads to a valuation of short-term goals over long-term purposes.
In situations where there are no real feasible solutions to a problem, the gathering and publication of performance data serves as a form of virtue signaling. There is no real progress to show, but the effort demonstrated in gathering and publicizing the data satisfies a sense of moral earnestness. In lieu of real progress, the progress of measurement becomes a simulacrum of success. We'll see that in the case of the educational "achievement gap."
Because belief in its efficacy seems to outlast evidence that it frequently doesn't work, metric fixation has elements of a cult. Studies that demonstrate its lack of effectiveness are either ignored, or met with the assertion that what is needed is more data and better measurement. Metric fixation, which aspires to imitate science, too often resembles faith.
All of that is not intended to claim that measurement is useless or intrinsically pernicious. One of the purposes of this book is to specify when performance metrics are genuinely useful — how to use metrics without the characteristic dysfunctions of metric fixation.
The next chapter, "Recurring Flaws," provides a taxonomy of the most frequent types of flaws in the use of performance metrics. Defining and labeling them will make it easier to refer back to them later. Then, in part II, we examine the origins of metric fixation and account for its spread and tenacity in spite of its frequent failures, in addition to exploring some of the deeper philosophical sources of its shortcomings. Part III comprises case studies that examine the more recent record of metrics, it successes and its shortcomings in a variety of fields, including K-12 education, higher education, medicine, policing, the military, business, and philanthropy and foreign aid. These case studies are intended to be suggestive rather than definitive. That is, they don't deal with every way in which the metric fixation manifests itself in each domain. Rather they provide concrete examples of recurring flaws and unintended consequences, as well as examples of the successful use of metrics from which we may derive lessons that can be applied in other domains. This section is followed by a brief excursus on the theme of transparency as the enemy of performance in certain realms. Finally, Part IVdraws upon the preceding analysis to enumerate the unintended negative consequences of metric fixation and offer some guidelines about when and how to make use of metrics without succumbing to metric fixation.
CHAPTER 2
RECURRING FLAWS
The drive to institute metrics often arises from the best of intentions, as a purported solution to real problems. And in some cases, as we'll see, it really does fulfill its promise to provide such solutions, or at least contributes to solving problems. But after decades of experience with the negative effects of metrics, as metric dysfunction threatens to cascade into yet more institutions, we should be able to anticipate the recurrent flaws. Here's a list to help identify and remember them. Of course, while we may distinguish them for purposes of analysis, these flaws often overlap in the real world.
Let's begin with problems of the distortion of information.
Measuring the most easily measurable. There is a natural human tendency to try to simplify problems by focusing on the most easily measureable elements. But what is most easily measured is rarely what is most important, indeed sometimes not important at all. That is the first source of metric dysfunction.
Closely related is measuring the simple when the desired outcome is complex. Most jobs have multiple responsibilities and most organizations have multiple goals. Focusing measurement on just one responsibility or goal often leads to deceptive results.
Measuring inputs rather than outcomes. It is often easier to measure the amount spent or the resources injected into a project than the results of the efforts. So organizations measure what they've spent, rather than what they produce, or they measure process rather than product.
Degrading information quality through standardization. Quantification is seductive, because it organizes and simplifies knowledge. It offers numerical information that allows for easy comparison among people and institutions. But that simplification may lead to distortion, since making things comparable often means that they are stripped of their context, history, and meaning. The result is that the information appears more certain and authoritative than is actually the case: the caveats, the ambiguities, and uncertainties are peeled away, and nothing does more to create the appearance of certain knowledge than expressing it in numerical form.
Campbell's Law and Goodhart's Law are warnings about the inevitable attempts to game the metric when much is at stake. Gaming the metrics takes a variety of forms.
Gaming through creaming. This takes place when practitioners find simpler targets or prefer clients with less challenging circumstances, making it easier to reach the metric goal, but excluding cases where success is more difficult to achieve.
Improving numbers by lowering standards. One way of improving metric scores is by lowering the criteria for scoring. Thus, for example, graduation rates of high schools and colleges can be increased by lowering the standards for passing. Or airlines improve their on-time performance by increasing the scheduled flying time of their flights.
Improving numbers through omission or distortion of data. This strategy involves leaving out inconvenient instances, or classifying cases in a way that makes them disappear from the metrics. Police forces can "reduce" crime rates by booking felonies as misdemeanors, or by deciding not to book reported crimes at all.
Cheating. One step beyond gaming the metrics is cheating — a phenomenon whose frequency tends to increase directly with the stakes of the metric in question. As we'll see, as the No Child Left Behind Act raised the stakes for schools of the test scores of their pupils, teachers and principals in many cities responded by altering students' answers on the test.
CHAPTER 3
THE ORIGINS OF MEASURING AND PAYING FOR PERFORMANCE
"Accountability," "metrics," and "performance indicators" have become cultural memes. Embracing them promises a seat on the train of historical progress, and no politician, agency chief, university president, or school superintendent wants to be left behind. When metrics becomes the coin of the realm, to refuse to use it is to risk bankruptcy. There is pressure from elected officials and from foundation managers to pay up.
How and why did this tyranny of metrics come about?
SOME ORIGINS OF PAYING FOR MEASURED PERFORMANCE
The idea that organizations outside the free market would be more efficient if they were paid based on measured performance seems to have occurred first to policymakers in Victorian Britain. In 1862, Robert Lowe, a Liberal member of parliament who oversaw the committee on education, proposed a new method for government funding of schools, which would be based on "payment by results." Lowe had distinguished himself in 1856 by shepherding through parliament a seminal piece of legislation in the history of capitalism. That was the Joint Stock Companies Act, which, together with legislation passed the previous year, the Limited Liability Act, set out a new law for corporations based on the principal of limited liability. From reforming the structure of business, Lowe turned to reforming government-supported schools.
Lowe's scheme was based on the premise that "the duty of a State in public education is ... to obtain the greatest possible quantity of reading, writing, and arithmetic for the greatest number." Schools were to be funded based on the performance of their students in the "3 Rs." Each school was to be visited annually by a school inspector, who was to quiz every student in English language and arithmetic. For every student who failed to appear or to answer questions successfully, a small sum would be deducted from the school's government funding. Lowe's reform was intended in part to cut costs, but above all to make school funding dependent on measurable results in the most basic and practical of skills, and to bring education into accord with his market-oriented principles by linking payment to performance.
Lowe's scheme was challenged by Matthew Arnold, the great cultural critic, whose day job was as a government inspector of the very schools Lowe set out to transform. Arnold warned consistently against extending the criteria appropriate to the market to other areas of life. With a dose of bravery, Arnold launched a public salvo against his political superior. In an essay entitled "The Twice-Revised Code," Arnold attacked the narrow and mechanical conception of education implied by the code. The ability to read intelligently, he pointed out, developed primarily not from narrowly tailored reading lessons, but from a more general cultivation, imbibed from the family or, failing that, from a school environment that created the mental desire to read. The goal of the schools, therefore, should be "general intellectual cultivation," without which the skills of reading and writing would not develop. The government, he lamented, sought to fund only the most rudimentary of educations instead of responding to "the strong desire of the lower classes to raise themselves." Since many impoverished students would inevitably be absent when the annual test was administered, or would fail the test itself, he predicted that the net effect of the proposed reform would be to reduce the funding of schools for the poor. The education of the people, he concluded, was to be sacrificed to "the friends of economy at any price."
Arnold frequently found himself inspecting schools where students ingested mountains of facts and arithmetic, but were bereft of analytic ability and utterly incapable of understanding sophisticated prose or poetry. They were taught not to reason but to cram. Both before and especially after the adoption of "payment for performance," he criticized such education for being "far too little formative and humanizing ... much in it, which its administrators point to as valuable results, is in truth mere machinery." This conception of education as machinery, tailored to the measurable production of reading, writing, and computation, and capable of being rewarded based on measurable output, ebbed and flowed in the decades that followed, reaching a flood tide at the end of the twentieth century.
At each subsequent wave, we'll encounter critics like Arnold, who pointed to the unmeasured costs of tying reward to standardized measurement.
MEASURING PERFORMANCE: TAYLORISM
There were traces of metric fixation in the school efficiency movement that rolled across the American educational landscape, starting in the 1910s and continuing for decades. In 1911, Simon Patten, an influential professor of economics at the Wharton School of Business, demanded that schools provide evidence of their contribution to society by showing results that could be "readily seen and measured." Other would-be reformers sought to bring to the school system the fruits of the industrial efficiency movement, founded by Frederick Winslow Taylor, an American engineer who coined the term "scientific management" in 1911. Taylor analyzed the production of pig iron in factories by breaking down the process into its component parts (through time-and-motion studies) and determining standard levels of output for each job. Workers who carried out their tasks more slowly than the prescribed time were paid at a lower rate per unit of output; those who met the expectation were rewarded at a higher rate. Taylor also advocated an elaborate system for monitoring and controlling the workplace. His goal was to increase efficiency by standardizing and speeding up work on the factory floor to create mass production.
(Continues…)
Excerpted from "The Tyranny of Metrics"
by .
Copyright © 2018 Princeton University Press.
Excerpted by permission of PRINCETON UNIVERSITY PRESS.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
Introduction 1
I THE ARGUMENT
1 The Argument in a Nutshell 17
2 Recurring Flaws 23
II THE BACKGROUND
3 The Origins of Measuring and Paying for Performance 29
4 Why Metrics Became So Popular 39
5 Principals, Agents, and Motivation 49
6 Philosophical Critiques 59
III THE MISMEASURE OF ALL THINGS? Case Studies
7 Colleges and Universities 67
8 Schools 89
9 Medicine 103
10 Policing 125
11 The Military 131
12 Business and Finance 137
13 Philanthropy and Foreign Aid 153
EXCURSUS
14 When Transparency Is the Enemy of Performance: Politics, Diplomacy, Intelligence, and Marriage 159
IV CONCLUSIONS
15 Unintended but Predictable Negative Consequences 169
16 When and How to Use Metrics: A Checklist 175
Acknowledgments 185
Notes 189
Index 213
What People are Saying About This
"In this clear and compelling book, Jerry Muller shows how our attempts to improve organizational outcomes through quantitative measures have metastasized into a culture of gaming and manipulation. Through carefully researched case studies on education, healthcare, and compensation, The Tyranny of Metrics makes a convincing case that we need to restore judgment and ethical considerations at a time when shallow quantification threatens the integrity of our most important institutions."—Rakesh Khurana, Harvard Business School"Have you ever wondered why universities make the mistake of hiring presidents with little or no experience in higher education, or why, nine times out of ten, these foreign imports fail? Then read Jerry Muller's new book and you will understand such folly as one more instance of an unhappy, massive trend—abandoning the situated judgment of experienced professionals in favor of the supposedly objective judgment promised (but not delivered) by the magic bullet of metrics: standardized measures and huge data banks touted as generating insight and wisdom all by themselves. Muller dismantles this myth in a brisk and no-nonsense prose that has this reader crying ‘yes, yes' at every sentence."—Stanley Fish, author of Winning Arguments and Think Again"Quantification, once only a tool, has become a cult. I can think of no better deprogrammer than Jerry Muller, whose renowned skills in dissecting political and social doctrines are evident here. The Tyranny of Metrics should be essential reading for managers and the managed alike."—Edward Tenner, author of The Efficiency Paradox: What Big Data Can't Do and Why Things Bite Back: Technology and the Revenge of Unintended Consequences"In The Tyranny of Metrics, Jerry Muller has brought to life the many ways in which numerical evaluations result in deleterious performance: in our schools, our universities, our hospitals, our military, and our businesses. This book addresses a major problem."—George A. Akerlof, Nobel Prize–winning economist"The Tyranny of Metrics is an important and accessible book about a growing problem. It comes as close as anything I've read to showing us how to break out of the dysfunctional cycle of measuring, finding out that measuring doesn't get us where we want to go, but then measuring some more."—David Chinitz, School of Public Health, Hebrew University Hadassah Medical School"Broad in scope and ambition, persuasively argued, and engagingly written, The Tyranny of Metrics is a very compelling book."—Mark Schlesinger, Yale University
Related Subjects
Customer Reviews
Explore More Items
