Evaluating teachers based on the performance of their students was in vogue for a little while before people got disillusioned with No Child Left Behind. The trouble with it is that a huge amount of the variance is due to the students the teacher is assigned rather than any attributes of the teachers themselves (I suppose the “fundamental attribution error” causes people to overlook that, at least when focusing on teachers). One way of getting around that is to not use absolute scores as the metric for a teacher, but their value-added. The trouble with that is that regression to the mean will reward teachers lucky enough to get students who happened to perform unusually badly the previous year and vice versa. That seemed to me a tough problem, but Robyn Dawes explained what to do in his “Rational Choice in an Uncertain World” from 1988. “The rational way of dealing with regression effects is to regress when making predictions. Then, if there is some need or desire to evaluate discrepancy (e.g., to give awards for “overachievement” or therapy for “underachievement”) compare the actual value to the predicted value – not to the actual value of the variable used to make the prediction.” I was planning on writing this post before I saw Steve’s follow-up on teacher quality, it’s just a coincidence.
A few pages later (though not in the same chapter) Dawes discusses the deinstitutionalization of the mentally ill and homelessness. Interesting to me since I’m a Szaszian libertarian and even the latter is often sufficient to get flack from both left & right for that event. He mocks people for overlooking the fact that the vast majority of homeless people are “poor, just plain poor” rather than “mentally ill” (quotes in the original), but gives little mention of the rate of homelessness among the deinstitutionalized, tossing off “My own observation is that “success” for all of us is a “sometimes” thing”. I found that disappointing.
April 28, 2010 at 8:53 am
You made a good point about how to account for randomness in teacher evaluations, but I want to step back for a second…
“The trouble with it is that a huge amount of the variance is due to the students the teacher is assigned rather than any attributes of the teachers themselves”
The idea that this is a serious problem is just naive. There is randomness in the universe and in our economic activities. It is ridiculous to act like this “problem” is unique to teachers, or that randomness is so terrible that it should push aside all other considerations (such as attempts to judge effectiveness). As a corollary, it is naive to treat economic success as prima facia evidence of economic “worthiness”.
Institutions can be structured to minimize the effect of randomness — for instance, a teacher should not be punished/fired if they have one bad year (whether it is due to bad luck with students or because they were stressed out by the illness/death of a family member). Also, the people running institutions should be looking for ways to measure “innate” contributors to success even in the presence of “extrinsic” randomness. This is worthwhile both for issues of fairness to employees and for the management of the institution.
It is possible that an institution has no way to effectively discriminate effective employees from ineffective ones, and in such a situation there would be no point in rewarding employees for good “performance” (whatever that would mean). However, the arguments against performance-based pay seem to rest on the idea that effectiveness cannot be measured perfectly, not that it is impossible to measure it at all.
As long as effectiveness can be measured to some extent, it should be considered by the management. As you pointed out, there will always be room for improvement, and anyone interested in these institutions should continue to look for better ways to evaluate their performance (and the performance of their employees).
April 28, 2010 at 1:41 pm
It’s worse than random, it’s perverse: the best teachers get assigned the worst students because, presumably, we can handle them.
April 28, 2010 at 9:51 pm
I don’t know how students are currently assigned, but if we naively implemented a value-added ratings system teachers would angle to be assigned the last years’ worst performers.
Because I don’t know how students are currently assigned, I didn’t give much thought into how they will be assigned under a different system. I expect there should be some data on whether poor students benefit more from good teachers than bad ones do, though I’m unaware of it. Robin Hanson suggests that poor students make good students worse off, so we should prefer taking the hit of relegating them to lousy classrooms.
April 30, 2010 at 6:26 pm
Slightly disappointing to me is your insistence, TGGP, against the notion of organic mental illness. But perhaps it’s not incomprehensible.
I accept of course that the objective case for such organic disease is circumscribed and not decisive. I also fully accept common critiques, regarding depression and ADD being overdiagnosed and overtreated, and so on.
But, why not at least remain open to the possibility that such organic states actually exist? I suppose that on one hand we have the epistemological humility of “what is now proved, was once only imagin’d.” On the other hand, there is the need for parsimony: one cannot remain open to all possibilities just because they are possible. I sense that the latter underlies your Szaszian stance. But I also sense that libertarian ideology is at least, uh, tinging your choice of stance.
There is a lot of subjective testimony in support of the concept of organic mental illness. Perhaps you are too positivistic to ascribe any significant weight to this sort of evidence.
Anyway, your post is a little opaque, on just what it was that you found disappointing. Was it that Dawes presented a view you agree with but he did not include the figures that would back up that view?
April 30, 2010 at 7:56 pm
“Was it that Dawes presented a view you agree with but he did not include the figures that would back up that view?”
Yes. I just finished it (except for some appendices) and I’ll also add that I disagree that we should be grateful for uncertainty since we’d have no hope without it. It is better to have justified confidence or assurance than hope, and better to have foreknowledge that can be used to prepare than a nasty surprise. I also disagree that ethics depends on uncertainty, many ethical thought-problems are framed with certainty of consequences.
I believe in alzheimer’s, syphilis, parkinson’s and so on. They are physical illnesses that happen to affect the brain. There are standards to be met in identifying physical illnesses and etiologies which relate the disease to its symptoms. “Mental illnesses” require no physical lesion, the symptom is synonymous with the disease, and what the symptom consists of is behavior disapproved of. In the absence of any standards of proof we could all be termed insane, and that which explains everything explains nothing. Some good examples from Rational Choice in an Uncertain World, there are certain fields of expertise we can be fairly confident are bogus because they are not even internally consistent. Handwriting analysis is one he discusses. The Rosenhan experiment is one of the most notable demonstrations of the lack of such validity in psychiatry.