How to referee an academic paper

Berk Ozler does us all tremendous good by interviewing legendary QJE editor Larry Katz over at Development impact.

David McKenzie follows it up with his own tips on the same blog. He makes an observation I’ve heard too often, which is that older development economists eat their young.

For some time I’ve been meaning to write up my own thoughts on how to referee an empirical paper, inspired in large part by a photocopy Betty Sadoulet and Alainn de Janvry handed out in a development economics seminar. While not exactly the same thing, you can see my main suggestions in the last two pages of my causal inference and research design syllabus.

And to save all of us pain, I do have an old advice post on how to be a discussant on a seminar paper. there are many parallels.

6 Responses

  1. Nice post – I loved the details . Does anyone know if my assistant can obtain a sample IRS 1040 – Schedule M example to type on ?

  2. Speaking as a former junior academic development economist at a mid-level institution, I have to say I’ve been very much surprised and appalled by the way the journal refereeing system works in my few years in academia. Now that you can Google any paper you are asked to referee, there is clearly no longer any such thing as double-blind peer review. You would think this would provoke a rethink of how to referee journal articles, but it hasn’t- we just absurdly pretend that double-blind peer reviewing still exists. When I ask my colleagues why they chose a particular journal to submit a paper to, the most common answer I get is “because I (or my co-author or advisor) know one of the editors.”

    I have received plenty of very negative referee reports where the report makes it clear that the referee didn’t bother to read the paper carefully- probably they Googled me, saw that I wasn’t anyone important, and crossed the referee report off their to-do list as fast as they could. Similarly, I have received very positive reports where the referee then goes on to raise substantive concerns that essentially demolish the paper- but clearly they recommended R&R because the paper still gets published. I strongly suspect that in the latter case those are referees who have some kind of personal connection to me or people I know.

    I’m not sure what the solution is, but it should be obvious that continuing to pretend that we have double-blind peer reviewing when we really don’t creates a situation where nepotism trumps scholarship. Ultimately, if you’re a young economist whose goal is to make a substantive contribution to how people understand and do development, I would say stay away from academia- the priorities have run completely off the rails. You can have a much bigger voice working for a think tank, international organization, government, NGO, or even a contractor.

  6. Hello Chris,

    The link to your casual inference syllabus is broken. Well, at least I couldn’t open the pdf here. All the best and congratulations for the great blog :)