Why Measures of Scholarly Work are Anti-Intellectual and Basically Random With Respect to Incentivizing Quality

I should not be writing this! I absolutely have better things to do, by almost any measure. Yet I have been asked to clarify which of my publications are “refereed” and which not. Unless I decide simply to resign, which among other things would entail leaving my current students in the lurch, I cannot entirely refuse to respond.

It will be easier to respond at some length than to answer the question straightforwardly as requested— although I suppose I could flip a coin for all the grey cases, which would include about half of my scholarly output over the years.  (I cheerfully grant that people with other specializations may have different experiences, but I know what I am talking about for my area. I doubt that it is highly unusual.)

Let’s begin by recalling a few things about tenure. In addition to it being the actually existing system that ensures basic job security and quality of life for academics (if actually-existing alternatives to tenure did the same, we could discuss other pros and cons of tenure in good faith) it was established in some sizable part to give scholars a sort of shelter.

Behind this shelter, they had scope to engage in research, teaching, and writing that might not be considered fully legitimate as judged by certain outside-looking-in measures. For example, they might pursue inquiries inconvenient to religious orthodoxies, illegible or not-yet-proven in terms of reigning scholarly paradigms, politically undesirable as judged by ruling regimes, or simply (to evoke the precedent of Socrates, who sadly did not enjoy tenure) “corrupting the minds of the youth.” Important forms of the academic classroom, if allowed to unfold as they should with teachers having scope to challenge students and students having space to explore cultural critiques, are hard to sustain without such shelter.

Although I have understood some of the risks (perhaps vaguely like Socrates with exceedingly lower stakes) one of the pleasures of gaining tenure is that for years I have been able to largely disregard a forced, misleading, counterproductive, and all-around anti-intellectual and bogus hard distinction between the “refereed” and the “not-refereed.” (Once again I disavow knowledge of all fields, but insist that I know about mine.) This has unfolded partly on principle, but mainly as an evolving result of “following my nose” toward the best ways I could discover to nurture and disseminate my research, teaching, and outreach.

One example concerns  my most-cited article, an analysis of Madonna that appeared in an edited book on religion and popular culture. It was polished and edited more extensively than almost any of my publications—but was it “refereed” (read: legible to bean-counters as unambiguous scholarly “productivity”)? I have never been certain whether anyone who evaluates care answer. The book I suppose mainly counts as “refereed.” And in retrospect, the article, too, probably now reads to most colleagues as at least in the ballpark of  “real scholarship,” since the book has been widely cited and is from a top university press. But in real time this was ambiguous. Some colleagues thought I was wasting my time, and were at least tempted to lump this as “not-scholarly” and discount it when evaluating me—“non-refereed,” slightly embarrassing, and a bad choice measured in opportunity cost.

I also know this about the piece. Its earliest draft was not part of my core scholarly agenda—it was supposed to be a quick side project urging readers of Christian Century magazine not to be dismissive of a couple of Madonna videos. Likely because this journal was just as dismissive as I feared, my manuscript was rejected. (Let’s belabor the point, I am evoking what I wrote above about venturing beyond what is legible within “ordinarily legitimate” paradigms; that was a key point of my piece.) So I expanded it into a more “normal” article with footnotes. Then I received a similar rejection from a journal of feminist theology. Thus I failed both at “popular” and “refereed” productivity. But I am glad it wound up in the book (“refereed” or not) because by the time I had rewritten it, with excellent editing, it was better and far more widely disseminated.

In my experience, such complexity is more the rule than the exception. Let me state this point in general terms, after which I will return with a few more examples.

  • There is a continuum between writing of high intellectual quality—or not.
  • There is a continuum between writing that represents lengthy time investments and serious innovation—or not.
  • There is a continuum between journals that have editorial boards (or functional equivalents) that are distinguished and/or full of the sorts of interlocutors one would desire—or not.
  • There is a continuum between journals (or functional equivalents) that extensively vet and edit drafts of submitted work— or not. (Sometimes this is called “refereeing”— or not.  “Referred journals” by no means always edit.)
  • There is a continuum between journals (or equivalents) that have a crossover readership from academia and “the public”—or not. (This can be important or not, but in fields where I work it is important.)
  • There is a continuum between journals (or equivalents) that have readerships that one would desire as dialogue partners—or not.
  • There is a continuum between articles with long word/page counts—or not. (Very often it takes more work, resulting in higher quality and use-value, to make something longer into something shorter.)
  • There is a continuum between research undertaken in part for synergies with quality and breadth of teaching (a clear virtue but rarely a good way to maximize resume lines)—or not.

Not a single one of these continua corresponds to “refereed/not-refereed.” Every one includes a great deal of refereed work at the undesirable end of its continuum. Every one includes ambiguously refereed work on the positive end.

The common wisdom posits that a referreed-or-not continuum is a valid approximate composite measure.  This is what I am explicitly denying.

Perhaps other scholars’ mileage will vary, but in my long experience with dozens of journals, refereed-or-not is an independent variable that is close to random with respect to the above continua. A great deal of what I read that wastes my time by exhibiting lower quality, a lower time investment by the authors, a lower readership, non-optimum word counts (often grossly inflated due to a lack of tough editing, although sometimes overly compressed too), and lower value for teaching is refereed. A great deal of what I read that has higher quality, higher investment… let’s just say “worthwhile” and sometime to aspire to emulate (which is prerogative of judgment as a tenured scholar)… is either not overtly not “refereed” or we circle back to the coin flip method of deciding ambiguous cases.

Since I really do feel aggrieved to be repeating these points—I have been saying this for years, more than enough to suspect that everything I am writing here will be ruled out of bounds, thus becoming an instance of a recurring waste of time—I will offer only a few more examples with fewer details.

Several of my most important and prestigious articles have taken the form of lengthy review articles in widely read journals. These could be considered (at first approximation) “refereed journals”—or if not, then toward the valorized end of a “quality/prestige” continuum. These are short articles with pointed scholarly interventions—sometimes involving a great deal of work to make them short enough to fit in “review article” slots. On balance, I worked harder on and am prouder of them compared to most of the articles (refereed!) that were spin-offs from my first book. But the bean counters ask: are these “reviews” (read: we will discount them if we bother to count them at all)? Well, yes. Are they scholarly articles? Absolutely yes. Are they in “referred journals”? (I already said yes, but now I add that some cases are clearer than others, while in general the quality, word counts, and dissemination are comparable across the board.) Are they “refereed articles” in these journals? (I don’t know. It is unclear how to categorize invited review essays, except that it involves discounting.) After we sort all this out, we can turn to other review essays, similar in quality and not always less widely read, but not placed in prestige journals.

So here we have a huge bloc of work that is like a yo-yo—on the way up it is “real scholarship,” on the way down “he is doing mere reviews,” then up again to “real,” down to “non-refereed,” etc. Likewise we can play yo-yo with dozens of short reviews that I defend passionately as serious scholarly contribution.

How about invited work for reference works or encyclopedias? I have written such pieces that were quick and easy, either written for the occasion or spun out of my books. But I have also written several of the longest, most ambitious, most time-consuming, and in one case most prestigious, works on my resume. At times pieces on the whole spectrum have been counted roughly “the same.” Consider also that I agreed to write some of them because they could be building blocks for books. One such article is lengthy and not “encyclopedia-like” in form. Nevertheless it will continue to be rated a “mere encyclopedia article” (read: discounted if not disregarded) until it becomes the heart of an introduction to a refereed book (the holy grail!). “Objectively” it will have been a counter-productive waste of my time until the book comes out—except that this is part of my process for writing the book.

Meanwhile some of my less important and least read articles—representing the smallest time investment and value added—were refereed. When my university insulted and demoralized its humanities faculty by imposing a one-size-fits-all, staggeringly ugly, and outrageously time-wasting straitjacket of a productivity report website—with an apparent goal of making it easier for administrators to generate quantitative reports—they touted this as a time saving benefit to faculty. This was on the grounds that the website was supposed to search relevant databases and import data automatically. Guess what it found for me, from more than 100 articles and reviews? An exceedingly small number drawn entirely from this last category of my less valuable work.

It is no exaggeration, but a straight literal fact, that I have sometimes seen resume lines that only cost me a couple afternoons of work counted “the same”—sometimes more!—than lines describing projects that have cost me years of work. And several of the most egregious examples involved refereeing or not. On one side are the already-noted “write it in one afternoon” articles in the “refereed” column, and on the other side my two most ambitious book chapters—both synthesizing huge amounts of research. It would have cost me less trouble to expand these into books than it did to cut my first drafts back to article length. Yet both count as “mere invited articles” in edited volumes that might also be construed as reference works.

Turning to on-line spaces that are increasingly central to scholarly discussion—with boards that may or may not “referee” depending on semantics—similar themes tumble through similar ambiguities. (All of this is before we try to assess this blog—refereed by me and me alone!—which I have repeatedly called an experiment in progress.) I have many short and simple online publications. But I also I have many longish online essays that will be building blocks for my books. Which are “refereed articles”?  I predict that my deans’ numerical printouts will reflect zero of them—although there is no clarity about it and every reason to expect that definitions will be applied inconsistently. Meanwhile some of these online pieces have been vetted, edited, and read more–not less–than comparable pieces in “normal” journals. In fact, a few are on par in quality with the best things I have ever written, although I exercised my judgment to place them online.

Such exercise of judgment—whether about publishing online, accepting invitations to write for edited books, spending time on “mere review essays,” etc.—is not solely my prerogative as a tenured scholar. It is my responsibility as a scholar in a land-grant public university.

So let’s restate a point I made at the outset. On principle, as a tenured scholar, I consider it part of my intellectual and spiritual discipline to try (in most cases) to bracket and disregard the forced, misleading, counterproductive, and all-around anti-intellectual and bogus hard distinction between the “refereed” and the “not-refereed” in the fields where I work. Rather I will continue to aim for a composite quality score on all the above-mentioned continua—amid which the refereed/non-refereed distinction does not quite reduce to a completely random independent variable, but comes close.

2 thoughts on “Why Measures of Scholarly Work are Anti-Intellectual and Basically Random With Respect to Incentivizing Quality

  1. Pingback: Dog Park Sex and the Bankruptcy of “Refereed versus Non-Refereed” for Measuring Value | MBE: Mark's blogging experiment

  2. Pingback: Notes from a Dinosaur Who Cares About Reference Books | MBE: Mark's blogging experiment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s