A Mechanism-Centred Approach to Evaluating Complex Aid Interventions: The Case of Accompanying Measures to Budget Support
Main Article Content
Abstract
Background: Current methodological debates related to theory-based evaluations (TBE) centre around questions how to improve the explanatory strength of these approaches and how to integrate mechanisms as analytic concept. Particularly in complex aid interventions, when multiple elements are expected to interact and thus create an added value, exploring mechanisms as an analytical tool can be promising.
Purpose: This paper seeks to contribute to the discussion on the use of TBE for evaluating complex aid interventions by sharing experiences from a recent evaluation of accompanying measures to general budget support.
Setting: Nine countries of sub-Saharan Africa which have received German budget support, namely Burkina Faso, Ghana, Malawi, Mali, Mozambique, Rwanda, Tanzania, Uganda, Zambia.
Intervention: Accompanying measures (mainly in the form of technical assistance and capacity development) are one element of the budget support package, which further encompasses financial contributions, policy dialogue, and conditionalities.
Research Design: We focus on interrelations between different elements of budget support and apply a mechanism-centred approach to programme theory building,
Data Collection and Analysis: After defining accompanying measures and integrating them into the intervention logic of budget support used in recent multi-donor evaluations, key mechanisms were identified on an explorative mission to Mozambique, validated in an online survey, and further discussed in expert interviews and during field research in Tanzania.
Findings: For the specific example relating to two elements of budget support (policy dialogue and accompanying measures), some of the hypothesized mechanisms were found to create an added value and thus increase the effectiveness of budget support as a package. The applied approach helped generate a more comprehensive implementation theory and provided insights into potential benefits and challenges of combining different elements in one programme. Beyond its use for future evaluations in the field of budget support, we argue that TBE of complex interventions can benefit from adopting such a mechanism-centred approach to create a better understanding of how different elements of the programme interact. Moreover, the focus on mechanisms when analysing programme implementation enables evaluators to improve their empirical inquiry on the identified mechanisms and to draw valid conclusions on the programme’s contribution to the observed outcomes.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Astbury, B., & Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363-381. https://doi.org/10.1177/1098214010371972 DOI: https://doi.org/10.1177/1098214010371972
Beach, D., & Pedersen, R. B. (2013). Process-tracing methods: Foundations and guidelines. Ann Arbor, MI: University of Michigan Press. https://doi.org/10.3998/mpub.2556282 DOI: https://doi.org/10.3998/mpub.2556282
Betts, J. (2013). Aid effectiveness and governance reforms: Applying realist principles to a complex synthesis across varied cases. Evaluation, 19(3), 249-268. https://doi.org/10.1177/1356389013493840 DOI: https://doi.org/10.1177/1356389013493840
Blamey, A., & MacKenzie, M. (2007). Theories of change and realistic evaluation: Peas in a pod or apples and oranges? Evaluation, 13(4), 439-455. https://doi.org/10.1177/1356389007082129 DOI: https://doi.org/10.1177/1356389007082129
BMZ. (2008). Konzept zur Budgetfinanzierung im Rahmen der Programmorientierten Gemeinschaftsfinanzierung (PGF). Bonn: Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung.
Chen, H-T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage Publications.
Chen, H-T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective. Thousand Oaks, CA: SAGE Publication. https://doi.org/10.4135/9781071909850 DOI: https://doi.org/10.4135/9781071909850
DANIDA. (2013). Guidelines for development contracts. Retrieved from http://amg.um.dk/en/~/media/amg/Documents/Technical%20Guidelines/Guidelines%20for%20Development%20Contracts/Guidelines%20for%20development%20contracts%20final%20June%202013.pdf
de Kemp, A., Faust, J., & Leiderer, S. (2011). Between high expectations and reality: An evaluation of budget support in Zambia - Synthesis Report. Bonn; The Hague; Stockholm: BMZ; Ministry of Foreign Affairs; SIDA.
DFID. (2011). Implementing DFID's strengthened approach to budget support: Technical note. Retrieved from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/214161/Strengthened-approach-budget-support-Tecnical-Note.pdf
Dijkstra, G., de Kemp, A., & Bergkamp, D. (2012). Budget support: Conditional results - review of an aid instrument (2000-2011). IOB Evaluation. The Hague: Ministry of Foreign Affairs; Policy and Operations Evaluation Department (IOB).
Donaldson, S. I. (2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum. https://doi.org/10.4324/9780203809730
European Commission. (2012). Budget support guidelines executive guide: A modern approach to budget support. Retrieved from http://ec.europa.eu/europeaid/sites/devco/files/methodology-budget-supportguidelines-executive-guide-201209_en_2.pdf
European Commission & IEG. (2015). Joint evaluation of budget support to Uganda (2004-2013): Final report. Retrieved from http://ieg.worldbank.org/evaluations/joint-evaluation-budget-support-uganda-2004-2013
European Court of Auditors. (2014). EuropeAid's evaluation and results-oriented monitoring systems (Special Report No. 18). European Union.
European Union. (2014). Mapping of support to PFM reforms in Mozambique. Unpublished document.
Faust, J., Leiderer, S., & Schmitt, J. (2012). Financing poverty alleviation vs. promoting democracy? Multi-donor budget support in Zambia. Democratization, 19(3), 438-464. https://doi.org/10.1080/13510347.2012.674357 DOI: https://doi.org/10.1080/13510347.2012.674357
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2012). Program evaluation: Alternative approaches and practical guidelines. Upper Saddle River, NJ: Pearson Education.
Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco, CA: Jossey-Bass.
George, A. L., & Bennett, A. (2005). Case studies and theory development in the social sciences. Cambridge, MA: MIT Press.
Gerring, J. (2010). Causal mechanisms: Yes, but…. Comparative Political Studies, 43(11), 1499-1526. https://doi.org/10.1177/0010414010376911 DOI: https://doi.org/10.1177/0010414010376911
Hedström, P., & Ylikoski, P. (2010). Causal mechanisms in the social sciences. Annual Review of Sociology, 36, 49-67. https://doi.org/10.1146/annurev.soc.012809.102632 DOI: https://doi.org/10.1146/annurev.soc.012809.102632
High Level Forum. (2005). Paris declaration on aid effectiveness: Ownership, harmonisation, alignment, results and mutual accountability.
Keijzer, N. (2013). Who's the boss? Strengthening the effectiveness of capacity-development support (Briefing Paper). Bonn: Deutsches Institut für Entwicklungspolitik.
Krisch, F., Schmitt, J., & Dörr, U. (2015). Accompanying measures to general budget support in Sub-Saharan Africa. Bonn: German Institute for Development Evaluation. Retrieved from https://www.deval.org/en/evaluation-reports.html
Lawson, A. (2015). Synthesis of budget support evaluations (Vol. 2): Review of methodological issues emerging from 7 country evaluations of budget support.
Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24(1), 5-20. https://doi.org/10.1016/S1098-2140(02)00271-0 DOI: https://doi.org/10.1016/S1098-2140(02)00271-0
Machamer, P., Darden, L., & Craver, C. F. (2000). Thinking about mechanisms. Philosophy of Science, 67(1), 1-25. https://doi.org/10.1086/392759 DOI: https://doi.org/10.1086/392759
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270-280. https://doi.org/10.1177/1356389012451663 DOI: https://doi.org/10.1177/1356389012451663
Nilsson, M. (2004). Effects of budget support: A discussion of early evidence (UTV Working Paper). Stockholm: SIDA.
Nkwake, A. M. (2012). Working with assumptions in international development program evaluation. New York, NY: Springer. https://doi.org/10.1007/978-1-4614-4797-9
OECD/DAC. (2012). Evaluating budget support: Methodological approach.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage.
Rogers, P. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), 29-48. https://doi.org/10.1177/1356389007084674 DOI: https://doi.org/10.1177/1356389007084674
Schmitt, J., & Beach, D. (2015). The contribution of process tracing to theory-based evaluations of complex aid instruments. Evaluation, 21(4), 429-447. https://doi.org/10.1177/1356389015607739 DOI: https://doi.org/10.1177/1356389015607739
Scriven, M. (2008). A summative evaluation of RCT methodology and an alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5(9), 11-24. https://doi.org/10.56645/jmde.v5i9.160 DOI: https://doi.org/10.56645/jmde.v5i9.160
Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). Broadening the range of designs and methods for impact evaluations (Working Paper Vol. 38). London: DFID. https://doi.org/10.22163/fteval.2012.100 DOI: https://doi.org/10.22163/fteval.2012.100
Weiss, C. H. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997(76), 41-55. https://doi.org/10.1002/ev.1086 DOI: https://doi.org/10.1002/ev.1086