As a reasonable and thoughtful academic of some experience, I judge that I am a skilled reviewer of other people's work, well able to take into account my own biases and render a fair and objective judgment for the benefit of the profession whenever I am called upon to do so.
I am wrong.
Those of you who entertain similar opinions of yourselves are equally wrong. Not, I am happy to say, disastrously wrong, but certainly wrong enough to be embarrassing and a source of concern for us all. Michèle Lamont is a sociologist and Faculty Advisor on Development and Diversity at Harvard, melding in this book her professional skill and her administrative concern. The title and subtitle of the book rather overstate what the book delivers, which is a shame inasmuch as it might leave a reader disappointed with a study that does not in fact disappoint.
What Lamont has done is study in detail the workings of peer review panels (the limitation is important, inasmuch as an entirely separate study would be needed of the ways journal and university press peer reviewing work, when the reviewers work alone and not in groups) for a group of important funding agencies in the humanities and social sciences: the American Council of Learned Societies, the Society of Fellows at "a top research university" (Lamont is at Harvard, recall), the Social Science Research Council, the Woodrow Wilson National Fellowship Foundation, and an unnamed foundation in the social sciences. She did detailed interviews with program officers, panelists, and program chairs.
The findings will be variously striking to various readers. For example, she teases out well the cultures of some representative disciplines: English, History, Anthropology, and Political Science. The historians come across phlegmatic, the literature scholars edgy and uncertain, and the anthropologists curiously isolated: but no summary in one or two adjectives is a fair representation of her discussion.
She is perhaps best, though, on the way reviewers bring to the table a host of judgments, expectations, and anecdotes that are not to be found in the folders they read, but that strongly influence their judgment. A sociologist admits, for example, that a good proposal on the table gets a better judgment because of the institution from which it comes. That institution is strong in that field, so this is more likely to be good work than if it came from (the sociologist's patronizing words) "some tiny little hole-in-the-wall college" (227). One grid that many bring with them has much to be said for it: "significance" and "originality" stand high on the list of things that reviewers seek in proposals, and I think I can live with that. But Lamont identifies as well a curious overlay of ethical inquiry and judgment: "humility," "determination", and "authenticity" are judged well, while "cultural ease" and "elegance" score points as well. How many of these categories are screens for finding "people like us" is a difficult question with no ultimate resolution.
For there is, on the one hand, some relationship between the recognized quality of the institution and the work that is likely to be done there, given the effort and resources that go into recruitment and selection of students and faculty; on the other hand, a bias that intrudes between the particular qualifications of the scholar and the merit of the proposal runs a real risk of unfairness.
The book that has resulted from these inquiries is not lively reading, but worth serious attention. At the same time, it will not be widely read or easily digested by many. Lamont's best prescription for improving the quality of the work that is done by many conscientious people is to raise the issue and focus the attention of scholars on their own practices and expectations. "Know thyself" remains good advice. An administrator reading the volume thinks about ways in which the unacknowledged forces that influence judgment can be made part of broader discussion, the better to improve the quality of judgment within institutions and beyond their walls. The topic deserves to remain open and remain the subject of continuing discussion.
Table of Contents:
1. Opening the black box of peer review
2. How panels work
3. On disciplinary cultures
4. Pragmatic fairness: customary rules of deliberation
5. Recognizing various kinds of excellence
6. Considering interdisciplinarity and diversity
7. Implications in the United States and abroad.