Participatory approaches to evaluation: A case for realist evaluation in sport for development

The benefits of these SfD approaches have been documented extensively, ranging from health-related outcomes, helping to reduce inequalities, societal benefits alongside influencing government policy (House of Lords, 2023). In some instances, the ‘success’ and popularity of interventions has resulted in their adoption and replication. As an important avenue for health and wellbeing promotion with demonstrated implications on policy and practice, Coalter (2010) argues evidence on the programme outcomes related to effectiveness and cost-effectiveness of SfD programmes is limited. Approaches to evaluation to identify indicators of ‘success’ have largely favoured four frameworks:- experimental design, constructivist evaluation, utilisation-focused evaluation and, more recently, realist evaluation (Chen, 2018). This article will highlight the benefits of using theory-based approaches to evaluation within SfD programmes – and propose participatory realist evaluation as the ideal framework.  

There is a plethora of research signifying the impact of sport for development programmes (Parnell et al., 2017; Brazier et al., 2023). However, traditional monitoring and evaluation methods are criticised as lacking in producing large-scale convincing evidence (Levermore, 2008). There is currently a need for deeper insight into factors maximising positive aspects of interventions while minimising the negative and implementing the former into practice. Funding bodies, implementing organisations and sector-wide stakeholders are increasingly focussing on this to better understand the social and economic value of SfD programmes. Development practitioners have been proposing the need to assemble proof beyond anecdotal evidence to monitor and evaluate impact (UNICEF, 2006). More recently, studies suggest evaluation of programmes might still be lacking and gaps exist vis-à-vis the ideal way to examine whether delivery of desired outcomes is being achieved through SfD programmes. Brazier et al., (2023) propose development of standardised monitoring and evaluation frameworks and outcomes. 

Secondly, more rigorous evaluation is required within these interventions to determine their effectiveness, in terms of cost and benefit to stakeholders. Generating more specific knowledge around the context of programme delivery from practitioners may also provide valuable information for the planning of monitoring and evaluation approaches (Shulha et al., 2016). This sustained push for a rethink of measuring impact is driven largely by efforts to demonstrate SfD’s contribution to society and the economy. Similarly, there is also a need for greater accountability and deeper insights into the cost-effectiveness of interventions. Ensuring SfD programmes meet the broader aims of implementing organisations and their necessity better explained to funding bodies upon conclusion are also key. To generate a higher quality of evaluation, it may be more beneficial for collaborative approaches that involve practitioners throughout the evaluation (Brazier et al., 2023). This involvement of stakeholders in the process can occur during evaluation design, data collection and analysis to the reporting stage. And as in realist evaluation, it can be informed by selective and targeted inclusion of shareholders alongside efforts to maximise effectiveness of the approach. 

Consequently, the criticisms levelled at current evaluation (experimental, constructivism and utilisation-focused) approaches necessitate re-examining aspects of delivery, monitoring and measurement of the impact generated by SfD programmes. A participatory approach such as realist evaluation could allay some of these concerns, and it is increasingly in use within sport development research (Mansfield et al., 2015, 2018). The interventions delivered through SfD tend to be complex, given the number of components in them, the expertise required to deliver them, and the flexibility inherent in them; thus, evaluations are inherently complex and incomplete. Theory-based approaches including participatory realist evaluation are increasingly being proposed as better guides to evaluating, leading to improved coherence and effectiveness in policy, and organisational and programme delivery processes (Coalter, 2007:4). As articulated in the new guidance by the Medical Research Council’s for developing and evaluating complex interventions (Skivington et al., 2021), evaluations of complex social interventions require an appropriate methodology that identifies the nuances of how the programmes work (how, for who, and in which circumstances) alongside measuring and valuing the societal costs and benefits associated with the programmes. This, in turn, will enable organisations and stakeholders to make better funding decisions long term. 

Realist evaluation aims to establish what works, for whom, in what circumstances, to what extent, in what contexts and how (Pawson & Tilley, 1997). It assumes that an intervention will not work in all circumstances, for all individuals and can be influenced by non-observable entities such as class and economic systems. Realist evaluation can lead to improved coherence and effectiveness in policy, organisational and programme delivery processes (Coalter, 2007). Through an iterative process, it tests and refines theories with accumulating knowledge to identify how activities cause outcomes in particular contexts (Dalkin et al., 2015). It also holds that mechanisms matter because they generate outcomes and context matters because it changes processes by which interventions produce outcomes. Mechanisms refer to the underlying social or psychological drivers that ‘cause’ the reasoning of actors (Better Evaluation). For example, a wellness programme delivered by a sports foundation may achieve different outcomes for females and males due to the influence of social norms. Context is also important as it affects ‘reasoning’ and mechanisms can only work if the circumstances are ideal. 

Subsequently, participatory approaches such as realist evaluation can contribute to better integration and develop ownership, build capacity and improve effectiveness and reach of interventions with positive outcomes. The ultimate aim of a realist evaluation is to create a transferrable theory about how desired outcomes are produced in different contexts (Fick & Muhajarine, 2019). Resultant theories can form the basis that facilitates the design of and implementation of future interventions. Borrowing from the desired outcomes of these ‘successful’ evaluations can benefit SfD programmes delivered by a variety of organisations. Participatory approaches to evaluation are ideal to develop better understanding amongst stakeholders, avoid missed opportunities of sharing best practise and spur improvements in measuring impact. Undoubtedly, there are a myriad of benefits in SfD interventions and adequate evaluation is necessary to safeguard their relevance long-term and maximise desired outcomes. Existing evidence-based gaps in current appraisal models could potentially be minimised by using a realist evaluation approach to facilitate learning from what has happened and inform what is to come.

However, whilst realist evaluation allows policy makers to better account for complexity, realist evaluations do not tend to explicitly capture the economic costs and consequences of interventions in order that they can allocate resources to best effect (REEM, 2022). Additionally, policymakers require economic appraisals to accompany evaluations to inform decisions in the context of cost-effective management and (re)deployment of resources. Research is underway to develop realist economic evaluation methods (REEM) to enable evaluators to establish what works, for whom, in which circumstances whilst also integrating better understanding of costs and consequences (REEM, 2022).

References

Better Evaluation. (2023, November 20). Realist Evaluation. https://www.betterevaluation.org/methods-approaches/approaches/realist-…

Brazier, J., Foster, C., Townsend, N., Murphy, J., Northcote, M., & Smith, A. (2023). Mapping the provision and evaluation practices of local community health and wellbeing programmes delivered by professional sports clubs in England: a practice-based targeted review, International Journal of Sport Policy and Politics. https://doi:10.1080/19406940.2023.2271904

Chen, S. (2018). Sport policy evaluation: what do we know and how might we move forward? International Journal of Sport Policy and Politics, 10(4), 741-759. https://doi:10.1080.19406940.2018.1488759

Coalter, Fred. (2008). A Wider Social Role for Sport: Who's Keeping the Score? Taylor & Francis Group. Routledge. 

Coalter, F. (2010) The Politics of Sport-for-Development: Limited Focus Programmes and Broad-Gauge Problems? International Review for the Sociology of Sport, 45, 295-314.

Dalkin, S., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M (2015). What's in a mechanism? Development of a key concept in realist evaluation. Implementation Science, 10(49). https://doi:10.1186.s13012-015-0237-x

Fick, F., & Muhajarine, N. (2019). First steps: creating an initial program theory for a realist evaluation of Healthy Start-Départ Santé intervention in childcare centres. International Journal of Social Research Methodology, 22:6, 545-556. https://doi:10.1080/13645579.2019.1595375

House of Lords. (2023). A national plan for sport, health and wellbeing - National Plan for Sport and Recreation Committee, HL Paper 113 Report of Session 2021–22. House of Lords: London. https://publications.parliament.uk/pa/ld5802/ldselect/ldsportrec/113/11…

Levermore, R. (2008). Sport: a new engine of development? Progress in Development Studies, 8(2), 183-190. https://doi.org/10.1177/146499340700800204

Mansfield, L., Anokye, N., & Fox-Rushby, J,. (2015). The Health and Sport Engagement (HASE) Intervention and Evaluation Project: protocol for the design, outcome, process and economic evaluation of a complex community sport intervention to increase levels of physical activity. The British Medical Journal Open, 5(e009276). https://doi:10.1136/bmjopen-2015- 009276

Parnell, D., Curran, K., & Philpott, M. (2017). Healthy Stadia: an insight from policy to practice. Sport in Society, 20(2), 181-186. https://doi:10.1080/17430437.2016.1173914

Pawson, R. & Tilley, N. (1997). Realistic Evaluation. Sage.

Realist Economic Evaluation Methods. (2022). Developing Realist Economic Evaluation Methods (REEM) and Guidance to Evaluate the Impact, Costs, and Consequences of Complex Interventions. https://realist-economic.co.uk

Shulha, L., Whitmore, E., Cousins, J. B., Gilbert, N., & al Hudib, H. (2016). Introducing Evidence-Based Principles to Guide Collaborative Approaches to Evaluation. The American Journal of Evaluation, 37(2), 193–215. https://doi:10.1177/1098214015615230

Skivington K, Matthews L, Simpson S A, Craig P, Baird J, Blazeby J M et al. (2021). A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. The British Medical Journal (Online), 374, n2061–n2061. https://doi.org/10.1136/bmj.n2061

United Nations. (2006). International Year of Sport and Physical Education 2005 – Final Report. UN.

About the author

Maureen Mwende is a postgraduate research student at Northumbria University. She is currently undertaking a doctoral research study at Northumbria University using realist economic evaluation methods (REEM) to understand the contributions of sports foundations towards addressing social determinants of health and reduction of inequalities as well as measure the cost-effectiveness of sport for development programmes in a collaborative studentship between Northumbria University and Sunderland FC’s Foundation of Light.