Why Measures of Scholarly Work are Anti-Intellectual and Basically Random With Respect to Incentivizing Quality

I should not be writing this! I absolutely have better things to do, by almost any measure. Yet I have been asked to clarify which of my publications are “refereed” and which not. Unless I decide simply to resign, which among other things would entail leaving my current students in the lurch, I cannot entirely refuse to respond.

It will be easier to respond at some length than to answer the question straightforwardly as requested— although I suppose I could flip a coin for all the grey cases, which would include about half of my scholarly output over the years.  (I cheerfully grant that people with other specializations may have different experiences, but I know what I am talking about for my area. I doubt that it is highly unusual.)

Let’s begin by recalling a few things about tenure. In addition to it being the actually existing system that ensures basic job security and quality of life for academics (if actually-existing alternatives to tenure did the same, we could discuss other pros and cons of tenure in good faith) it was established in some sizable part to give scholars a sort of shelter.

Behind this shelter, they had scope to engage in research, teaching, and writing that might not be considered fully legitimate as judged by certain outside-looking-in measures. For example, they might pursue inquiries inconvenient to religious orthodoxies, illegible or not-yet-proven in terms of reigning scholarly paradigms, politically undesirable as judged by ruling regimes, or simply (to evoke the precedent of Socrates, who sadly did not enjoy tenure) “corrupting the minds of the youth.” Important forms of the academic classroom, if allowed to unfold as they should with teachers having scope to challenge students and students having space to explore cultural critiques, are hard to sustain without such shelter.

Although I have understood some of the risks (perhaps vaguely like Socrates with exceedingly lower stakes) one of the pleasures of gaining tenure is that for years I have been able to largely disregard a forced, misleading, counterproductive, and all-around anti-intellectual and bogus hard distinction between the “refereed” and the “not-refereed.” (Once again I disavow knowledge of all fields, but insist that I know about mine.) This has unfolded partly on principle, but mainly as an evolving result of “following my nose” toward the best ways I could discover to nurture and disseminate my research, teaching, and outreach.

One example concerns my most-cited article, an analysis of Madonna that appeared in an edited book on religion and popular culture. It was polished and edited more extensively than almost any of my publications—but was it “refereed” (read: legible to bean-counters as unambiguous scholarly “productivity”)? I have never been certain whether anyone who evaluated me could answer. The book, I suppose, mainly counts as “refereed.”  And in retrospect the article, too, probably now reads to most colleagues as at least in the ballpark of  “real scholarship”– although articles which are “book chapters” don’t fully count as “articles” — since the book has been widely cited and is from a top university press. But in real time this was ambiguous. Some colleagues thought I was wasting my time, and were at least tempted to lump this as “not-scholarly” and discount it when evaluating me—“non-refereed,” slightly embarrassing, and a terrible choice at least as measured in opportunity cost.

I also know the following about the piece. Its earliest draft was not part of my core scholarly agenda—it was intended as a quick side project urging readers of Christian Century magazine not to be so dismissive of a Madonna video. Likely because this journal was just as dismissive as I feared, my manuscript was rejected. (Let’s belabor this, I am evoking what I wrote earlier about venturing beyond what is legible within “ordinarily legitimate” paradigms.)  So I expanded it into a “normal” article with footnotes; this received another rejection from a journal of feminist theology. Thus I failed both at “popular” and “refereed” productivity. But I am thrilled that it wound up in the book (“refereed” or not) because by the time I had rewritten it, with excellent editing, it was better and far more widely read.

In my experience, such complexity is more the rule than the exception. Let me state this point in general terms, after which I will return with a more examples.  There are continua between all these poles:

  • Writing of high intellectual quality—or not.
  • Writing that represents lengthy time investments and serious innovation—or not.
  • Journals that have editorial boards (or functional equivalents) that are distinguished and/or full of the sorts of interlocutors one would desire—or not.
  • Journals (or equivalents) that extensively vet and edit drafts of submitted work— or not. (Sometimes this is called “refereeing”— or not.  “Referred journals” by no means always edit.)
  • Journals (or equivalents) that have a crossover readership from academia and “the public”—or not. (This can be important or not. In fields where I work it is important.)
  • Journals (or equivalents) that have readerships that one would desire as dialogue partners—or not.
  • Articles with long word/page counts—or not. (Usually it takes more work, resulting in higher quality and use-value, to make something longer into something shorter.)
  • Research undertaken in part for synergies with breadth of quality teaching (a clear virtue but rarely a good careerist move)—or not.

Not a single one of these continua corresponds to “refereed/not-refereed.” Every one includes a great deal of refereed work at the undesirable end of its continuum. Every one includes ambiguously refereed work on the positive end.

Common wisdom posits that a referreed-or-not continuum is a valid approximate composite measure.  I am explicitly denying this.

Perhaps other scholars’ mileage will vary, but in my experience with dozens of journals, refereed-or-not is an independent variable that is close to random with respect to the above continua. A great deal of what I read that wastes my time by exhibiting low quality, low time investment by the authors, low readership, non-optimum word counts (often grossly inflated due to a lack of tough editing, although often overly compressed too), and low value for teaching is refereed. A great deal of what I read that has high quality, high investment… let’s just say “worthwhile” and something to emulate (belaboring again: to emulate as a tenure scholar with the prerogative of exercising judgment )… is either explicitly “not refereed” or we circle back to a coin flip method for deciding ambiguous cases.

Since I really do feel aggrieved to be repeating these points—I’ve been saying this for years, more than long enough to know that everything I am writing will be ruled out of bounds and judged an annoying waste of time—I will offer only a few more examples with fewer details.

Several of my most important and prestigious articles have taken a form of lengthy review articles in widely read journals. These could be considered (at first approximation) “refereed journals”—or if not, then toward the valorized end of a “quality/prestige” continuum.  These are best understood as short articles with pointed scholarly interventions—sometimes involving a great deal of work to make them short enough to fit in “review article” slots. On balance, I worked harder on them and am prouder of them compared to most of the articles (refereed!) that were spin-offs from my first book. But the bean counters ask: aren’t they really “reviews” (read: we will discount them if we count them at all)? Well, yes. Are they scholarly articles? Absolutely yes. Are they in “referred journals”? (I already said yes, but now I add that some cases are clearer than others, but the quality, word counts, and dissemination are comparable across the board.) Are they “refereed articles” in these journals? (Time for a coin flip; it is unclear how to categorize invited review essays, except that it involves discounting.)

After we sort all this out, we can turn to other review essays of mine, similar in quality and sometimes more widely read, but not placed in “prestige” journals.

So here we have a huge bloc of work that is like a yo-yo—on the way up it is “real scholarship,” on the way down “he is doing mere reviews,” then up again to “real,” down to “non-refereed,” and so on.

Likewise we can play yo-yo with dozens of short reviews (clearly not “articles” but an immense amount of work) that I defend passionately as serious scholarly contribution.

How about invited work for reference works or encyclopedias? I have written some such pieces that were quick and easy, either written for the occasion or spun out of my books. But I have also written several of the longest, most ambitious, most time-consuming, and in one case most prestigious, works on my resume. Pieces on this entire spectrum have often been counted roughly “the same.” Consider also that I agreed to write some of these because they could be building blocks for my books. One such article is lengthy and not “encyclopedia-like” in form. Nevertheless it will continue to be rated a “mere encyclopedia article” (read: discounted, if not disregarded, if not treated as negative) until it becomes the heart of an introduction to a refereed book (the holy grail!). “Objectively” it will have been a counter-productive waste of my time until the book comes out—except that this is part of my process for writing the book.

So far I have been talking about throwing good energy after bad, as decreed by my university’s assessment regime. But then there is throwing bad energy after good, either for benign motives or to consciously game the system.

Some of my less important and least read articles—representing the smallest time investment and value added—were “refereed.” When my university insulted its humanities faculty by imposing a one-size-fits-all, staggeringly ugly, and outrageously time-wasting straitjacket of a website for reporting “productivity”—with an apparent goal of making it easier for administrators to generate quantitative reports—they touted this as a time saving benefit to faculty. This was based on the idea that this website could search relevant databases and import data automatically. Guess what that search found for me, from more than 100 articles and reviews? An exceedingly small number drawn entirely from this last category of my least valuable work.

It is no exaggeration, but a straight literal fact, that I have sometimes seen resume lines that only cost me a couple afternoons of work counted “the same”—or even more!—than lines reporting projects that cost me years of effort. The most egregious examples involved counting things as refereed or not. On one side were the already-noted “spin it off in an afternoon” articles in a “refereed” column, and on the other side my two most ambitious book chapters—synthesizing massive amounts of research. It would have cost me less trouble to expand these latter two chapters into books than it did to cut their first drafts back to article length. Yet both count as “mere invited articles” in edited volumes that might also be construed (read: discounted) as reference works.

Turning to on-line journals that are increasingly central to scholarly discussion—with boards that may or may not “referee” depending on semantics—similar themes tumble through similar ambiguities.  I have many short and simple online publications. But I also I have many longish online essays that will be building blocks for my books. All of this is before we turn to this blog—refereed by me and me alone!

Which are “refereed scholarship”?  I predict that my deans’ numerical printouts will reflect zero of them—although there is no clarity about it and every reason to assume that that definitions will be applied inconsistently. Meanwhile, some of these online pieces have been vetted, edited, and read more–not less–than my comparable pieces in “normal” journals. Some are on par in quality with the best things I have ever written, even though I exercised my judgment to place them online.

Such exercise of judgment—whether about publishing online, accepting invitations to write for edited books, spending time on “mere reviews,” etc.—is not solely my prerogative as a tenured scholar. It is my responsibility as a scholar in a land-grant public university.

So let’s restate a point I made at the outset. On principle, as a tenured scholar, it is part of my intellectual and spiritual discipline to try (in most cases) to bracket and disregard the forced, misleading, counterproductive, and all-around anti-intellectual and bogus hard distinction between the “refereed” and the “not-refereed” in fields where I work. Rather I will continue to aim for a composite quality score on all the above-mentioned continua—amid which the refereed/non-refereed distinction does not quite reduce to a random variable, but comes close.

2 thoughts on “Why Measures of Scholarly Work are Anti-Intellectual and Basically Random With Respect to Incentivizing Quality

  1. Pingback: Dog Park Sex and the Bankruptcy of “Refereed versus Non-Refereed” for Measuring Value | MBE: Mark's blogging experiment

  2. Pingback: Notes from a Dinosaur Who Cares About Reference Books | MBE: Mark's blogging experiment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s