Randomized trials revisited

16720017-Abstract-word-cloud-for-Randomized-controlled-trial-with-related-tags-and-terms-Stock-Photo

Following my defense of randomized trials this week, come this thoughtful reader comment:

This response seems to miss, or perhaps obscure, the point. In my understanding, Hausmann is suggesting that development organizations take a Toyota-style approach to innovation, in which front-line workers have authority to adapt, make suggestions, and eventually change the way the organization works. In this case, power to innovate lies in the front-lines, among implementers.

In contrast, Blattman seems to depart from the premise that high-level managers, or academics, are the ones authorized to have ideas, and these ideas are then transmitted to the fieldworkers who implement them. Thanks to rigorous testing, the best ideas can be disseminated. Power is centralized, and held by the proper authorities.

So the debate is not about methods or rigor, it is about authority to innovate and power to decide.

I agree, but in that case we both have it wrong.

To see this, imagine a Toyota that gives customers a car whether they like it or not. Front-line worker innovation might or might not develop cars that people want to own. The problem is there’s no vote or market test where the ultimate users decide.

I think this is called the Lada.

This is the fundamental problem of aid. A bunch of planners have the power to decide and are only accountable to donors, most of whom seem happy to remain ignorant of the details or the actual success of their interventions.

People have put ahead randomized trials as an improvement over the current mess (including donors who quite fairly don’t know any other way to make this better).

Now, trial-and-error innovation combined with randomized trails could be more powerful. That was my point. But make no mistake: both are still the tools of the inept planner.

Of course, none of these are reasons that social scientists like randomized trials. They are interested in using these field experiments to try to test ideas or estimate parameters, sometimes to produce general knowledge for the public good. Maybe an adaptive mechanism could do this even better.

To extend the metaphor, I think that means we academics who do field experiments are the intellectual wing of the Communist Party, who have captured Lada’s production for our own purposes, both selfish and noble. The term randomista sounds more appropriate than ever?

27 Responses

  1. One important distinction that I haven’t heard you explicitly make here is that RCTs have very different roles if you are thinking about whether to do something versus how to do something.

    Academic-led RCTs have been more successful at the whether level. Whether donors should fund micro finance, whether we should form organizations to scale up deworming or bed nets or hiv/AIDS education or what have you.

    In many/most cases, it does not make sense for implementing organizations to think about the whether question as R&D because they are constrained by their mission statement. Micro finance organizations will always offer micro finance. The whether question is appropriate for donors and very large organizations.

    There have been many exciting “how” papers, eg group vs individual liability, gender targeting, commitment devices, flexible repayment terms. These questions are much more like R&D, and much less relevant to donor decision making. These are the areas where we need better frameworks for front line trial & error, a/b testing and other less rigorous approaches.

    So another reading of Hausmann is that our focus on whether questions, which are relevant to donors but not implementing organizations, who will carry on offering micro finance regardless, crowds out energy that could be spent improving the services that are going to be offered anyway.

  2. I think your metaphor is a bit flawed, because cars are things that don’t change after they are produced. If you view it as the production of recipes, rather than cars, while you can still critique poles of thought that are distant from the end user (it’s the only restaurant in each town, say, so there’s no consumer choice), it’s a lot harder to argue against the logic that chefs in restaurants should be doing more adaptive learning of what works with the ingredients in their vicinity, rather than having a central food lab testing new dishes through RCTs.

    If I understand your first article correctly, you seem to be arguing that the prominence of RCTs is not costly because prior to the randomista revolution, very few donors or large development players were investing in learning, and now there’s pressure to do so. It seems a curiously comfortable acceptance of a second-order solution to motivate change in that way – why not simply attack the problem of not enough learning through a first-order solution of requiring more investment in monitoring and learning, allowing methods suitable to the programming, and perhaps with more investment in cross-learning by those who can fund the relevant public goods (donors and academia)?

  3. My issue with Haussman’s article is that it is not called “how evidence-based policy could get even better” but “the problem with evidence-based policy”. If it was called the former, I would say that definitely a model whereby people can experiment on the field, get rapid feedback, twitch policies, get more feedback and so on is the ideal form of policy design (well, that combined with sound theory). However, because he chose the latter, he seems to imply that we are moving from a where these rapid feedback loops are happening towards a frozen model of policy. And that is just wrong. As you emphasized in the first post, the real tension right now is evidence vs. intuition or preconceived ideologies, and the second camp is winning by a landslide. The RCT movement is not about making policy slower, but about finding a way to learn what actually works. If Haussman knows some way to both learn what works and quickly adapt, I seriously doubt any of the leaders in the Evidence-based policy movement would object.