Evaluation and Limitations of Social Interventions: The Case of Spain
Main Article Content
Abstract
Background: Although the number and practice of evaluation varies enormously among policy areas, there are very few studies about the unequal evaluation development in the different policy sectors.
Purpose: This article aims to (1) acknowledge the different evaluation development among different policy sectors in Spain (2) identify the factors and causes that provoke this disproportion, and (3) explore potential consequences of this unequal distribution of evaluation studies among policy areas.
Setting: Spain.
Intervention: Public polices in Spain.
Research Design: A sample of evaluations is classified by policy sector and the number of evaluations in each sector is analyzed and compared. Then, other significant variables are identified for explaining differences among sectors.
Data Collection and Analysis: The cases (evaluation studies) are drawn from two samples: (1) a data base of evaluation studies and (2) a survey to Spanish evaluators held in 2009. The comparison was done with difference in proportions, adjusted standardised residuals and crosstabs.
Findings: Analysis of Spanish evaluations shows that program evaluations are much more frequent in the social policies’ area than in the areas of security, defense or justice. A variable with a high ability to predict whether or not evaluations will be carried out is identified: the selective versus universal nature of the policies being evaluated. Selective interventions are more frequently evaluated than universal policies. This lack of balance makes selective interventions more prone to severe critical analysis. This evaluation bias, in turn, produces a series of perverse effects such as a greater probability of cutting down programs based on selective application strategies.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Ballart, X. (1992). ¿Cómo evaluar programas y servicios públicos? Madrid: MAP.
Bustelo, M. (2002). ¿Qué tiene de específico la metodología de evaluación? In Bañon, R. (Eds), La evaluación de la acción de las políticas públicas. Madrid: Díaz de Santos.
Bustelo, M. (2004). La evaluación de las políticas de género en España. Madrid: Catarata.
Bustelo, M. (2006). The potential role of standards and guidelines in the development of an evaluation culture in Spain. Evaluation, 12,: 437-453. https://doi.org/10.1177/1356389006071291 DOI: https://doi.org/10.1177/1356389006071291
Bustelo, M, Díez, M. A, Izquierdo, B. and Ligero, J. A.(2006). Bulding Capacity for Evaluation. The case of Spain. Paper for European Commission.
Cabrera, P. (1998). Huéspedes del aire. Sociología de las personas sin hogar en Madrid. Madrid: UPCO.
Chelimsky, E. (2008). A clash of cultures. Improving the "fit" between evaluative independence and the political requireiments of a Democratic Society. American Journal of Evalution, 29, 400-415. https://doi.org/10.1177/1098214008324465 DOI: https://doi.org/10.1177/1098214008324465
Christie, C. A. (2007). Reported influence of evaluation data on decision makers' actions. American Journal of Evalution, 28, 8-25. https://doi.org/10.1177/1098214006298065 DOI: https://doi.org/10.1177/1098214006298065
Comisión para el Estudio y Creación de la Agencia Estatal de Evaluación de la Calidad de los Servicios y de las Políticas Públicas (2004). Informe 4 de Octubre 2004. Madrid: Ministerio de Administraciones Públicas.
Cohen, E. y Franco, R. (1993). Evaluación de proyectos sociales. Madrid: Siglo XXI.
Datta, L.-E. and Grasso, P. G. (1998). Evaluating tax expenditures: Tools and Techniques for Assessing outcomes. New Directions for Programs Evaluations, 79. https://doi.org/10.1002/ev.1104 DOI: https://doi.org/10.1002/ev.1104
Díaz-Puente, J. M., Cazorla, A. and Dorrego, A. (2007). Crossing National, Continental, and Linguistic Boundaries. Toward a Worlwide Evaluation Research Community in Journals of Evaluation. American Journal Evaluation, 28,399-415. https://doi.org/10.1177/1098214007308413 DOI: https://doi.org/10.1177/1098214007308413
Díaz-Puente, J. M., Yaqüe, J. L. and Afonso, A. (2008). Building Evaluation Capacity in Spain. A case study of rural development and empowerment in the European Union. Evaluation Review 32, 478-506. https://doi.org/10.1177/0193841X08319015 DOI: https://doi.org/10.1177/0193841X08319015
Fernández-Ramírez, B. and Rebolloso, E. (2006). Evaluation in Spain: Concepts, Contexts, and Networks. Journal of Multidisciplinary Evaluation, 5. https://doi.org/10.56645/jmde.v3i5.57 DOI: https://doi.org/10.56645/jmde.v3i5.57
Freeman, H. E. and Solomon, M. A. (1981). The next decade in evaluation research. In Levine, R. A, Solomon, M. A., Hellstern, G.-M. and Wollaman, H. Evaluation research and practice. Comparative and International perspectives. Berverly Hills (CA) and London (England):Sage.
Furubo, J.-E., and Sandahl, R. (2002). A diffusion perspective on global developments in evaluation. In Furubo, J.-E., Rist, R.C. and Sandahl, R., (Eds). International Atlas of Evaluation. New Brunswick and London: Transaction Publishers.
García-Sánchez, E. (2005). La evaluación de programas de reforma educativa en España. Tres estudios de caso desde un enfoque de metaevalaución. Madrid: Editorial de Universidad Complutense.
Garde, J. A. (2006). La evaluación de políticas públicas y su institucionalización en España. Madrid: Ministerio de Administraciones Públicas.
Greene, J. C. (1999). The inequality of performance measurements. Evaluation 5,: 160-172. https://doi.org/10.1177/13563899922208904 DOI: https://doi.org/10.1177/13563899922208904
Halusha, J. (2006, Novembre 9). Voters in 7 states back ban on gay marriage. The International Herald Tribune, p. 7.
Henry, G. T. (2002). How modern democracies are shaping evaluation and the emerging challenges for evaluation. American Journal of Evaluation, 22, 419-429. https://doi.org/10.1177/109821400102200320 DOI: https://doi.org/10.1016/S1098-2140(01)00138-2
Henry, G. T. and Mark, M. M. (2003). Beyond use: understanding evaluations's influence on attitudes and actions. American Journal of Evaluation, 24,:293-314. https://doi.org/10.1177/109821400302400302 DOI: https://doi.org/10.1177/109821400302400302
House, E. R. (2006). Democracy and Evaluation. Evaluation. 12,119-127. https://doi.org/10.1177/1356389006064196 DOI: https://doi.org/10.1177/1356389006064196
Leeuw, F. L. and Furubo, J.-E. (2008). Evaluation Systems. What are they and why study them? Evaluation 14,157-169. https://doi.org/10.1177/1356389007087537 DOI: https://doi.org/10.1177/1356389007087537
Leeuw, F. L., Toulemonde, J. and Brouwers, A. (1999). Evaluation activities in Europe: a quick scan of the market in 1998. Evaluation, 5, 487-496. https://doi.org/10.1177/135638999400830020 DOI: https://doi.org/10.1177/135638999400830020
Lipsey, M. W. (2000). Meta-Analysis and the Learning Curve in Evaluation Practice. American Journal of Evaluation, 21, 207-212. https://doi.org/10.1177/109821400002100208 DOI: https://doi.org/10.1016/S1098-2140(00)00073-4
Lipsey, M. W. (1995). What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents? In McGuiere, J. (ed.). What works? Reducing Reoffending. New York: John Wiley.
Monnier, E. (1992). Evaluación de la acción de los poderes públicos. Madrid: Instituto de Estudios Fiscales.
Newcomer, K. E. (2004). How might we strengthen evaluation capacity to manage evaluation contracts? American Journal of Evaluation, 25, 209-218. https://doi.org/10.1016/S1098-2140(04)00027-X DOI: https://doi.org/10.1016/S1098-2140(04)00027-X
Nioche, J.P. et Poinsard, R. (1984). L'Evaluation des Politiques Publiques. Paris: Economica.
Pazos, M. and Zapico, E. (2002). Program evaluation in Spain: taking off at the edge of the twenty-first century? In Furubo, J.-E., Rist, R. C. and Sandahl, R., (Eds.). International Atlas of Evaluation. New Brunswick and London: Transaction Publishers.
Renger, R. (2006). Consequences to Federal Programs when the Logic-Modelling Process is not followed with fidelity. American Journal of Evaluation, 27, 452-463. https://doi.org/10.1177/1098214006293666 DOI: https://doi.org/10.1177/1098214006293666
Rutter, M., Giller, H., Hagell, A. (2000). La conducta antisocial de los jóvenes. Madrid, Cambridge University Press.
Shadish, W. R., Chacón-Moscoso, S. and Sánchez-Meca, J. (2005). Evidence-based decision making: Enhancing systematic reviews of program evaluation results in Europe", Evaluation 11 (1): 95-109. https://doi.org/10.1177/1356389005053196 DOI: https://doi.org/10.1177/1356389005053196
Stake, R.E. (2006). Evaluación Comprensiva y evaluación basada en estándares. Barcelona: Grao.
Stame, N. (2008). The European Project, federalism and evaluation. Evaluation, 14, 117-140. https://doi.org/10.1177/1356389007087535 DOI: https://doi.org/10.1177/1356389007087535
Stufflebeam, D. and Shinkfield, A.(1995). Evaluación sistemática. Guía teórica y práctica. Barcelona: Ed. Paidós.
Thuren, B. M.(1993). El Poder Generizado. Madrid: Instituto de Investigaciones Feministas.
Tobal, C. (1982). Guía para la formulación y evaluación nacional de proyectos de desarrollo rural integrado. Washington D.C: OEA.
Varone, F., Jacob, S. and De Winter, L. (2005). Polity, politics and policy evaluation in Belgium, Evaluation, 11, 253-273. https://doi.org/10.1177/1356389005058475 DOI: https://doi.org/10.1177/1356389005058475
Weiss, C. (1991). Evaluation research in the political context: sixteen years and four administrations later. In McLaughlin, M. W. and Phillips, D.C. Evaluation and educations: At quarter century. Chicago: The National Society for the Study of Education. https://doi.org/10.1177/016146819109200610 DOI: https://doi.org/10.1177/016146819109200610
Weiss, C. (1998). Evaluation. New Jersey: Prentice - Hall.