Moving from Studies to Streams: A More Radical Way to Avoid Floods of Evidence by Channeling Evaluative Efforts in Conditions of Complexity

Abstract

Background: Multiple evaluations of complex problems risk creating ‘floods’ of evidence that are hard for decision makers to make sense of. Using the metaphor of ‘canalizing’ evidence to create more manageable streams of evidence, it is argued that the evaluation community can create more usable knowledge to support decision makers and strengthen accountability. ‘Downstream’ systematic reviews and clearing houses are not enough to address this issue. The problems arise ‘upstream’ from multiple, weakly connected evaluations being conducted in relative isolation, rather than focusing evaluative efforts on the key challenges facing complex policy spaces.

Purpose: It is argued that evaluation policies need to move ‘upstream’  and be consciously designed to both build on existing evaluations and take the needs of decision-makers more fully into account. This cannot be achieved by ‘one off’ improvements to individual evaluations but requires a coherent response that would support effective and meaningful retroductive realist syntheses that reveal the patterns found in evaluations of heterogeneous interventions.

Setting: Evaluations are almost never at the heart of policy debates today and we should at least be curious about why this is the case. It is argued that evaluative efforts are poorly used, unfocused and require ‘canalization’ to replace floods of evidence with coherent sense-making.

Intervention: not relevant

Research design: Review and reflection

Data collection: not relevant

Analysis and findings: Urgent action is required across the community of evaluators, commissioners of studies, and users of evaluative evidence to make better use of the resources dedicated to evaluation. Streams of evidence should be curated to promote learning and improvement, involve multiple stakeholders, in ways that require the evaluation community to take stock of our competencies and organisational forms. Whether this takes us towards a single answer, and identifies ‘best practice’, or whether it takes us towards heterogeneous insights, and identifies more ephemeral ‘good practices’, is for the evidence to decide. How this evidence is channelled will be critical.