External validity and policy adaptation: from impact evaluation to policy design

policy design

With the growing number of impact evaluations worldwide, the question of how to apply this evidence in policymaking processes has arguably become the main challenge for evidence-based policymaking. How can policymakers predict whether a policy will have the same impact in their context as it did elsewhere, and how should this influence the policy's design and implementation? This paper introduces a simple and flexible framework to address these questions of external validity and policy adaptation. Failures of external validity arise from an interaction between a policy’s theory of change and a dimension of the context in which it is being implemented. I discuss the strengths and limitations of existing approaches to external validity, and develop a method of "mechanism mapping" that maps a policy’s theory of change against salient contextual assumptions to identify external validity problems and suggest appropriate policy adaptations. In deciding whether and how to adapt a policy, there is a fundamental informational trade-off between the strength and relevance of evidence on the policy from other contexts and the policymaker’s knowledge of the local context. This trade-off can guide policymakers’ judgments about whether policies should be copied exactly from elsewhere, adapted, or invented anew.

All are welcome at this CEBI seminar; please arrive early to guarantee a seat.