Should quality peer reviewers be recognized by the journal?
March 23, 2013 Filed under Blog, Featured, Publishing, Reviewing
I came across this web post and thought this Editor’s idea for recognizing the top 8% of reviewers was a pretty good idea. I admit it might be some work to implement, but the Editor had a formula to do it, reducing the work involved.
I am a big fan of this idea for several reasons. First, peer reviewing is a volunteer job. As we showed in Golden and Schultz (2012), Reviewers take an average of 9.6 hours to perform a single review, and they review an average of 8 manuscripts a year. Anything that we can do to increase the recognition that these reviewers get is worth it.
Second, finding reviewers is getting increasingly difficult. I’ve been an Editor since 2004, and it is getting increasingly harder to find people who will take on the reviews. People are simply busier and making less time for volunteer work. Editors Awards (such as the AMS offers) are difficult to motivate people because their chance of winning is so small.
But, suppose reviewers could say that they were recognized as within the top 5-10% of reviewers in terms of quality, promptness, and number of reviews performed. Suppose that they could list this on their CV. Then, this kind of recognition is something more attainable. Now, perhaps these reviewers at the top are already motivated by their service to science and giving them an award for doing so isn’t going to change their behavior. But, it might help motivate the more reluctant early-career or mid-career reviewer who might not otherwise want to participate. And, it is a nice little acknowledgement that their contributions to peer review are valuable and recognized.
Loading ...Golden, Mary, and David M. Schultz, 2012: Quantifying the Volunteer Effort of Scientific Peer Reviewing. Bull. Amer. Meteor. Soc., 93, 337–345.
(Image from www.ruralintelligence.com)
Thanks for writing about our initiative – it has been quite well received. I think the 8-10% cut-off is about right, at least for us, as these people do over 30% of our reviews. If you try to reward a larger percentage it becomes quite hard to decide where to draw the line, as big groups of people all have the same value for the index we used. I think this would be roughly true no matter what index you used.
Tim
I think this is a good idea. However, I don’t think it would motivate me anymore because I am motivated for different reasons. First, as a meteorologist I feel honored to be called upon to perform reviews. Second, I think it is part of my duty as a professional meteorologist to perform reviews. Third, I enjoy reviewing papers because it lets me see new research before it is available to the broader community (and yes, I keep this to myself per the ethics standards of peer review). Fourth, it forces me to keep abreast of the literature by reading papers relevant to the articles I review. Having said all that, it wouldn’t hurt to get a little recognition, but I don’t believe that would give me much more motivation than I already have. Nevertheless, I do understand the difficulty of obtaining reviewers, so if this helps I am definitely in favor of it.
— Matt
This idea is intriguing. I see no harm in trying, especially for higher-volume journals and/or those with anonymous review, where the reviewer’s efforts otherwise go unrecognized.
For journals that publish open reviews, quality (or lack thereof) is self-evident. Still, as an editor of a low-volume, open-review journal (EJSSM), the thought has occurred to me that a few of our very best reviewers ought to be recognized *somehow*. I’m open to suggestions as to how.
As a reviewer, my motivations match Matt’s, so I voted accordingly in the poll. It’s a labor of love–sometimes, tough love. 🙂 Though not as much so as you, Dave, I probably would be considered an “experienced reviewer”, having reviewed ~100 papers over the past 15 years. Though thorough review is time consuming (most take well north of 10 hours apiece), I am generally comfortable with it, and have developed a fairly consistent regimen for doing so. Templates built on well-received past reviews are a big help, as is Erik Rasmussen’s evaluation-summary table (found in several EJSSM reviews).
Still, as a reviewer, I sometimes hit snags, desiring advice or clarity. I might recruit a trusted colleague’s opinion on a particular, small portion of the review, especially if I am either: 1) seeking greater expertise on a specific thought or reference brought up in the paper, 2) stuck as to how to respond best to a point of contention, or 3) unsure of the validity of my own counterclaim(s). It’s beneficial for me and for the author(s) in that the quality of the review is improved, and by extension, its potential benefit to the quality of the paper.