The problem with evidence based policy change is we don’t have evidence on the important policies

Peter Singer has a Boston Review piece telling us we should all be “effective altruists”—to make a difference by giving our time and our money, and giving only to causes that demonstrate their effectiveness through evidence.

There are many good replies, including from Angus Deaton and Daron Acemoglu. Here is Acemoglu:

More evidence is always preferred, but precise measurement of the social value of a donated dollar may be impossible. What is the social value of a dollar given to Amnesty International as opposed to Oxfam or an NGO providing vaccines or textbooks?

…But the problem is thornier still. A large body of research shows that economic development is the best way to lift millions out of poverty and improve their health, education, and access to public amenities. So one has to take into account how charities’ activities affect economic development, which is essentially impossible. If, as some economists and political scientists suggest, changes in political and economic institutions are critical for long-run economic growth, then watchdog organizations such as Amnesty may be essential for transforming dysfunctional regimes. Effective altruists don’t (yet?) see the importance of these more political organizations.

To his critique (and Deaton’s as well): Yes! At the same time, some reservations:

  • It’s hard to overstate how many stupid and dead end causes people give money to. Singer probably sees very rich people giving to idiotic boutique causes all the time. I avoid those people, but I have to contend with the World Bank and others spending billions on things like vocational training, which has basically zero impact. This is another way of saying Singer is right on the margin, Acemoglu is right as we move away from the marginal decision.
  • I’m not worried about “too much” effective altruism. It would be a problem if it happened, but the world won’t even get close. Aid donors and the very rich are (1) stubborn, and (2) don’t read.
  • But Acemoglu is right that institutional and political change are more important and the evidence-based crowd have done very little here. Most of that evidence is about anti-corruption or election monitoring or other things that I doubt change politics very much.
  • Meanwhile all the good political economy research (like Acemoglu’s) has no clear implication for social and political change in the world. There is a big disconnect. These scholars have mostly ignored this gap either because… I don’t know why. Maybe it’s too treacherous or hard, or they don’t find it interesting enough, or they are cynical about policy change. I don’t know. Someone explain it to me.
  • Actually, this is not entirely true. You could view a lot of research says “you should stop violent conflicts, and here are concrete steps to do so”. I can think of few better short-term investments. More work along these lines strikes me as a good thing.

39 Responses

  1. Nice post Chris, thanks. Agree with the comments here. Aware that even distributing bednets have slight impacts on the relationship between constituents and politics in developing countries. Perhaps there is a role for social scientists as you suggest in actually finding giving opportunities / thinking about how to address questions of institutional improvement that aren’t narrowly defined. The problem for me with ‘institutions matter’ in the developing world, is that advocating effectively for specific policy changes seems higher in expected value, more achievable, and less imperialistic. EAs are already funding this kind of thing with Project Healthy Children’s nutritional assessments and advocacy at the national level. SCI can also be seen as an advocacy org as it works so closely with departments of health. I would have thought that something like effective reform of policing might be interesting, and somewhere that evidence could bear on in terms of what works? The biggest issue with all this, though, is that (especially in many African states – a la bayart / soares de oliveira) politics is so context specific its hard to generalise across countries as to predicted effects (I think Bratton and Van de Walle get close to this but it takes a long time, is necessarily abstracted, only bites on very large and impactful policies like SAPS, and is often retrospective). This precludes scale unless you just focus on one nation that’s relatively large and get to know all the players and how it works. The level of knowledge involved is difficult again, then, because of the lack of transparency and ability to share what you know without changing what’s happening on the ground. So, for those reasons, I think its unreasonable to expect a movement mainly comprised of people under 30 with very little field experience or very little in the way of a strong connection to a single country, to be making the kinds of dramatic improvements they’ve already made in the field of politics, advocacy and institutions. Is that fair?

  2. “A large body of research shows that economic development is the best way to lift millions out of poverty and improve their health, education, and access to public amenities. So one has to take into account how charities’ activities affect economic development, which is essentially impossible.”

    If this is true, then the expected value of any particular charity *in terms of its impact on economic development* is zero. That is, we have no idea whether it is likely to do any good or any harm in terms of economic development.

    That being the case, the sensible thing would be to support those charities that are known to have a big positive impact in other areas, whether it’s health or education. These seem to be good things to support in their own right, independently of their impact on growth.

  3. Good blog post. The estimates of total NPV of gains from growth accelerations are in the trillions of dollars (e.g. China, India) (and losses from decelerations similarly large. Anything that even marginally changed those probabilities at modest cost would have fantastically high B/C ratios–but would have high uncertainty.

  4. It is telling how Acemoglu and Deaton and all other commentators to Singer in BR and Chris Blattmann ignore the altruist component with regard to non-human animals. I gotta ask do you buy and eat factory farm derived animal products Chris?

  5. ‘Meanwhile all the good political economy research (like Acemoglu’s) has no clear implication for social and political change in the world.’

    IMHO it seems the most significant implication of much of this PE research is that in developing countries rent-seeking, corruption and patronage aren’t cancers on the system which can be removed, they are the blood which keeps the political system stable and alive. Institutional development requires a particular subsection of elites to have both the incentives and power to enact and enforce institutional change over the interests of other more entrenched elites. These *reformist* elites are still going to be corrupt, possibly violent and have sources of political-economic power based on patronage. Who wants to be seen supporting such groups?

  7. Acemoglu’s response is quite glib. EAs can donate to GiveDirectly, and help someone with probability ~1. Or they can donate to causes aiming at large-scale institutional and political change, with a much lower probability of having a positive effect. Does he think the latter has an obviously higher expected value? EA’s are grappling with a tremendously difficult tradeoff between upside and P(success); it would be nice if Acemoglu even acknowledged that there is a tradeoff.

  8. “Meanwhile all the good political economy research (like Acemoglu’s) has no clear implication for social and political change in the world. There is a big disconnect. These scholars have mostly ignored this gap either because… I don’t know why.”

    If your goal is political change, not research about political change, then you operate like a political consultant, not a researcher. I don’t think scholars have studied what consultants/activists/lobbyists/politicians and others who actually deal with political institutions actually do. And there may not be a general theory here — the circumstances are highly particularized.

  9. Nils is right on the money. I think at the core of much of this is a distinction between the actions and culture of EA and the fundamental ideals of EA. Deaton and Acemoglu haven’t offered a case against the fundamental ideals of EA – they have simply made a case that those ideals call for more focus on political institutions than most EAs and EA culture evince. There’s actually a large and growing crowd of people (including myself) who say things like “I agree with the idea of EA but not the way it tends to be done.”

  10. Also, I should quote here the paragraph of Singer’s reply that specifically addresses Acemoglu and Deaton, since Chris refers to them:

    “Deaton, Daron Acemoglu, Iason Gabriel, and Jennifer Rubenstein all suggest that effective altruists are likely to neglect the large-scale political and economic reform that would treat the causes, rather than the symptoms, of poverty. It is true that we can’t assess such action by randomized trials, but if large-scale reform offers some prospect of reducing poverty, then effective altruists will try to assess its chance of doing good, and if the expected value of such action is higher than the expected value of more limited interventions, they will advocate working for the large-scale reforms.”

  11. I can confidently speak for most EAs when I say that we are acutely aware that policy and institutions is a potentially valuable mode of action. See https://80000hours.org/2015/07/effective-altruists-love-systemic-change/. Evidence for effectiveness doesn’t have to come from an RCT, but it has to have some basis in reason. It is ironic that Acemoglu’s argument itself serves as evidence that Amnesty might be a good investment. EAs try to weigh all these arguments. It’s not easy to do. Institutional change is difficult, so it’s not obvious whether a marginal dollar spent on institutional change is worth more than a dollar spent on bednets, despite the fact that global institutional change is worth orders of magnitude more than global distribution of bednets.