From Esther Duflo to the Counterfactual of Institutional Learning
What if we applied the same standards of causal reasoning to our own evaluations that we expect from the programmes we evaluate? This article proposes extending counterfactual thinking beyond the effects of development interventions to examine how evaluations influence decisions, learning and institutional change. After all, demonstrating what works is one thing. Getting organisations to act on that knowledge is quite another.
Duflo’s Revolution and an Uncomfortable Question
In 2019, Esther Duflo was awarded the Nobel Prize in Economic Sciences, alongside Abhijit Banerjee and Michael Kremer, for their experimental approach to alleviating global poverty. Their work helped establish a fundamental question in impact evaluation: how can we know whether an intervention actually produces the results we attribute to it?
The answer involves establishing a credible counterfactual: what would have happened if the intervention had never taken place?
Randomised controlled trials (RCTs) have made a decisive contribution to answering this question, allowing researchers to identify causal effects with a level of rigour that simple comparisons between beneficiaries and non-beneficiaries rarely achieve (Banerjee & Duflo, 2009).
In Poor Economics, Banerjee and Duflo (2011) also demonstrate how experimentation can challenge widely held assumptions about poverty and inform more effective development interventions.
But identifying a causal effect does not answer every evaluation question.
A programme may succeed in one context and fail in another. An average treatment effect may conceal important differences across population groups. And an intervention may deliver immediate results without transforming the institutions needed to sustain them.
I recently explored some of these questions on TripleAD, looking at how causal machine learning can help uncover heterogeneous effects and move us beyond the familiar «Does it work?» towards more useful questions: for whom does it work, under what circumstances, and with what implications for decision-making? (Rodríguez-Ariza, 2026a).
Yet there is another frontier that interests me even more.
We can become increasingly sophisticated at demonstrating what works without necessarily becoming any better at using that knowledge.
And this raises a rather uncomfortable question for those of us working in evaluation:
What would have happened if our evaluation had never taken place?
I do not mean whether we would have one fewer report, a few outstanding recommendations or one less presentation on the institutional calendar.
I mean something considerably more difficult: would the organisation have made different decisions? Would it have changed its priorities, reallocated resources, questioned its assumptions or addressed any of the institutional structures underlying the problems identified?
Perhaps it is time to apply some of the same causal scrutiny to evaluation itself that we routinely apply to the programmes we evaluate.
The Forgotten Counterfactual
Back in 2017, reflecting on how to strengthen evaluation use, I argued for making the theory of change of the evaluation process itself more explicit: how an evaluation might contribute to knowledge use, learning and decision-making (Rodríguez-Ariza, 2017).
Almost a decade later, the question remains relevant. Perhaps we can now frame it more rigorously.
The literature has reminded us for decades that producing evidence does not guarantee its use. Patton (2008) placed intended users and intended uses at the heart of evaluation design. Weiss (1998) showed that evaluation influence extends beyond immediate decisions to include new ideas and different ways of understanding problems. Kirkhart (2000) broadened this perspective further, conceptualising evaluation influence in terms of its source, intention and timing.
So we are hardly discovering that evaluations should be used. We already know that.
The more interesting challenge is to take this concern one step further.
Imagine two organisations receiving equally rigorous evidence that an intervention is ineffective.
The first revises its strategy, changes incentives and reallocates resources.
The second acknowledges the recommendations, prepares a management response, updates its tracking system and carries on much as before.
Both can demonstrate that they have responded to the evaluation. But have they learned equally?
The difference does not necessarily lie in the quality of the evidence. It may lie in who has the authority to make decisions, which incentives shape those decisions, whether institutional relationships allow assumptions to be challenged, or what consequences follow from acknowledging that something is not working.
This connects with a recurring concern on TripleAD: organisations can identify the same problems repeatedly without managing to change the mechanisms that reproduce them (Rodríguez-Ariza, 2026b).
This leads to a hypothesis worth exploring:
The influence of an evaluation depends as much on the quality of the evidence it produces as on the institutional conditions that allow that evidence to inform decisions and drive change.
This is not an argument for lowering methodological standards. Quite the opposite. It is about recognising that the rigour of evidence and an institution’s capacity to use it are two distinct dimensions of the same challenge.
What If Evaluations Had Their Own Theory of Change?
If we accept that an evaluation is also an intervention within a decision-making system, we should be able to identify the changes we expect it to produce and the mechanisms through which those changes might occur.
We could therefore work with two complementary theories of change.
The first would explain how the programme is expected to achieve its results.
The second would explain how the evaluation is expected to influence decisions, practices and institutional learning.
This second theory of change might follow a seemingly straightforward sequence:
Evidence → Interpretation → Deliberation → Decision → Changes in practice → Institutional learning
But we know that organisations rarely operate so neatly.
Between each stage lie concrete mechanisms: trust, legitimacy, authority, incentives, resources, power relations and political opportunities. Evidence may be ignored, reinterpreted, selectively used or put to purposes we never anticipated.
That is precisely why an evaluation’s theory of change should do more than describe how we hope its recommendations will be used. It should help us investigate what actually happens.
We might ask:
- Which decisions changed as a result of the findings?
- Who used the evidence, how and for what purpose?
- What would plausibly have happened without the evaluation?
- Which conditions enabled or prevented change?
- Which changes endured beyond the formal tracking of recommendations?
We could not always answer these questions through an RCT. Nor would we need to.
Contribution analysis, for example, offers a way to develop credible causal explanations by drawing on theories of change, empirical evidence and the examination of alternative explanations (Mayne, 2012). Process tracing and comparative case studies can also help us understand how an evaluation contributed — or failed to contribute — to particular decisions.
The point is not to mechanically attribute every decision to an evaluation. It is to investigate evaluation influence with the same critical mindset we bring to assessing programmes.
And here, an important distinction is needed.
An implemented recommendation is not necessarily a transformed decision.
It may represent meaningful change. Or it may simply be an administrative action that allows someone to tick a box.
Management response tracking systems are necessary, but they should not become substitutes for examining how evaluations are actually used and what influence they have.
Three Counterfactuals for Rethinking Evaluation
Perhaps we can organise this reflection around three complementary questions.
The programme counterfactual: What would have happened without the intervention?
This is the classic impact evaluation question. It helps us investigate the causal effects of an intervention and distinguish them from other influences.
The evaluation counterfactual: What would have happened without the evaluation?
This invites us to examine whether the evaluation contributed to changes in decisions, interpretations, priorities or institutional practices.
The institutional counterfactual: What would have happened if the organisation had different capacities, incentives or structures for learning?
This third question is particularly interesting. It shifts attention from the quality of the evaluation to the conditions that enable — or constrain — its influence.
These three counterfactuals are not equally easy to establish or test. The third, in particular, is more exploratory and may require comparisons across organisations, analysis of causal mechanisms and plausible alternative institutional scenarios.
Taken together, however, they offer a useful perspective: we need to understand not only whether an intervention works, but also whether an evaluation makes a difference — and why some organisations are better able than others to turn evidence into action.
From Producing Evidence to Changing Decisions
This reflection is particularly relevant at a time when artificial intelligence is dramatically expanding our capacity to produce and analyse information.
We can process vast amounts of data, identify patterns, explore heterogeneous effects and automate tasks that once took weeks.
But no amount of methodological sophistication can guarantee that an organisation is willing, able or prepared to change.
As I argued in an earlier reflection on artificial intelligence and the future of evaluation (Rodríguez-Ariza, 2026c), the value of our profession cannot be reduced to producing increasingly sophisticated information. It also lies in developing well-founded evaluative judgements, enabling deliberation and contributing to better-informed decisions.
Perhaps we need to complement our methodological quality assurance systems with much greater attention to the institutional conditions that make evidence use possible.
Not to hold evaluators responsible for decisions beyond their control. Nor to reduce the value of an evaluation to immediate, easily measurable changes.
An evaluation may help challenge assumptions, open up debates, reshape interpretations or strengthen capacities without producing an immediately identifiable decision. These forms of influence matter too.
And precisely because they matter, we should make a greater effort to understand and document them.
Evaluation does not end when we have convincingly demonstrated what happened. Its transformative potential also depends on what institutions are able to do with that knowledge.
Duflo helped us ask what would have happened without an intervention.
Perhaps it is time to turn that same causal curiosity towards our own practice.
Because if an evaluation rigorously demonstrates that a programme is not working, yet the organisation continues doing exactly the same thing, we are facing two different problems.
One is the programme.
The other may well be our evaluation system.
And that second problem deserves to be evaluated too.
References
Banerjee, A. V., & Duflo, E. (2009). The experimental approach to development economics. Annual Review of Economics, 1, 151–178. https://doi.org/10.1146/annurev.economics.050708.143235
Banerjee, A. V., & Duflo, E. (2011). Poor economics: A radical rethinking of the way to fight global poverty. PublicAffairs.
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 2000(88), 5–23. https://doi.org/10.1002/ev.1188
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–280. https://doi.org/10.1177/1356389012451663
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). SAGE Publications.
Rodríguez-Ariza, C. (2017, October 28). Potenciando el uso de la evaluación [Strengthening evaluation use]. TripleAD. https://triplead.blog/2017/10/28/potenciando-el-uso-de-la-evaluacion/
Rodríguez-Ariza, C. (2026a, October 9). Cuando el promedio no basta: Causal Machine Learning y una nueva frontera para la evaluación [When the average is not enough: Causal machine learning and a new frontier for evaluation]. TripleAD. https://triplead.blog/2026/10/09/cuando-el-promedio-no-basta-causal-machine-learning-y-una-nueva-frontera-para-la-evaluacion/
Rodríguez-Ariza, C. (2026b, October 14). ¿Cuántas veces (y por qué) necesitamos aprender la misma lección? [How many times (and why) do we need to learn the same lesson?]. TripleAD. https://triplead.blog/2026/10/14/cuantas-veces-y-por-que-necesitamos-aprender-la-misma-leccion/
Rodríguez-Ariza, C. (2026c, July 9). Inteligencia artificial y el futuro de la evaluación [Artificial intelligence and the future of evaluation]. TripleAD. https://triplead.blog/2026/07/09/inteligencia-artificial-y-el-futuro-de-la-evaluacion/
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21–33. https://doi.org/10.1177/109821409801900103
Note on the use of artificial intelligence: I used AI tools to explore ideas, consider different perspectives, identify references and refine the writing. The argument, critical reflections and conclusions are my own, and I take full responsibility for them.
