You are here

Policy monitoring and evaluation

Policy monitoring and evaluation

An overview of the key principles for measuring the impact of sport and development policies.

The Sustainable Development Goals (SDGs) emphasise the need to strengthen national data systems and evaluation programmes. Target 17.18, for example, refers to building the capacity of less developed countries to increase the availability of data, while 17.19 focuses on developing measurements of progress and supporting statistical capacity building in developing countries. Other sport-specific documentation, such as the 2016 UN General Assembly resolution A/71/L.38 on sport as a means to promote education, health, development and peace, emphasise the need to show evidence of impact in order to assess contributions to the SDGs.

It is essential to collect and analyse data on any policy. It demonstrates whether a policy has achieved its aims and helps guide future policy decisions. Measuring the impact of sport-based approaches may be even more important than some other areas. That is in part because some people remain skeptical that sport can be useful in development, and data is needed to show it can be effective. Conversely, it is also because of concerns that exaggerated and idealistic claims about sport’s impact are sometimes made by its proponents - there is a need for data that is reliable and goes beyond anecdotal evidence.  

However, measuring the impact of sport is very complex. 

Challenges

Challenges include the following:  

  1. It is difficult to isolate the role of sport - was it sport that led to the changes or something else? It can be difficult to demonstrate that sport contributed to any successes reported. That is particularly true given that sport-based initiatives usually take place at the same time as other initiatives working towards the same goals and given that sport programmes often involve non-sporting components which may be crucial in enabling change.
  2. Although sport can help achieve development targets, it is rarely the main policy approach to doing so
  3. Limited data exists to assess the contribution of sport to development. Levels of data availability vary from country to country, but many do not have rigorous systems related to sport. Most data focuses on specific projects or programmes, making it difficult to measure the contribution of sport to national, regional or global objectives
  4. The national-level data that does exist normally relates to the percentage of the population participating in sport, the size and contribution of the sport workforce or the measurement of the sports economy to GDP. Other data is more limited or non-existent
  5. Most sport-related data has been gathered as part of a one-off study. That makes it difficult to track changes over time
  6. Capacity to collect data is often limited. For example, when sport policies include development aims, policymakers and decision-makers in sport often lack the experience and training to collect and report on the data needed to show its impact beyond the playing field
  7. Most toolkits and resources developed focus on projects or programmes, making it difficult to build capacity for policymakers, who have different needs
  8. Much of the evidence that exists is anecdotal - there is not enough quantitative data
Recommendations

Despite the challenges, measuring the impact of sport is not impossible. There is a growing body of research and advice to help policymakers to do so. Several key themes emerge. 

Triangulate different data sources
A technique now often used in research, ‘triangulation’ involves looking at multiple sources of data and using more than one method to collect data. As it is often not possible to isolate the role of sport to a development outcome in comparison to other influences, triangulation is needed to make a judgement about sport’s direct and indirect contributions and to inform future policy formulation. 

Triangulation means looking at data from the local, regional, national and international levels, as well as from initiatives by other actors such as civil society and academic research. Bridging different types of data is key, while also including different formats, such as data from social media or sports club applications, may also be complementary. There are different types of triangulation, as described by the Commonwealth Secretariat in Enhancing the Contribution of Sport to the Sustainable Development Goals (page 39).

Work with other sectors
Developing indicators and monitoring frameworks to measure sport-related contributions to development should be done in collaboration with other sectors. That is not only to gain access to data sources (see triangulation) but also to build capacity and share knowledge. 

It helps to avoid unnecessary replication of existing efforts, while allowing sports policymakers to learn from expertise in other departments. National statistics agencies and research institutions can play a particularly important role. On the other hand, sports policymakers can also look at how their work can contribute to the efforts of others. 

Adopt a theory of change
A theory of change can be understood as a development hypothesis, which identifies a connection between a policy’s activities and the envisioned outcomes/change. It is very important in understanding the rationale and impact of a policy intervention, and is particularly useful for monitoring and evaluation. 

Theories of change present projects or policies as causal chains that show how different factors may have a positive or negative impact on a desired outcome. These factors might, for example, include information about the participants (such as their educational levels or age), making it easier to look at the extent to which it was the intervention that caused changes rather than other factors. 

However, there is no sector-wide theory of change for sport and development. Many organisations have adopted a theory of change or logframe for their own work.

Social return on investment 
Sheffield Hallam University defines social return on investment as a “framework for measuring and understanding the non-market economic, social and environmental value of an activity, intervention, policy or organisation.” It usually puts a financial value on those non-market outcomes, and provides governments, sports organisations and the private sector with data showing how, where and why to invest in sport. 

One example is a Laureus Sport for Good Foundation report, which looked at community sports projects in Germany, Italy and the United Kingdom. It argued that there is evidence to suggest every €1 invested in sports projects can reduce costs to society by an average €5.02 in crime reduction. These savings come from reducing costs to victims of crime, police and the courts. 

The technique is gaining popularity with policymakers. However, its use to measure the value of sports projects and policies is still relatively limited. 

Use SMART indicators
Originally used as a management tool to help project managers reach their objectives, the acronym ‘SMART’ is now frequently used in development for monitoring and evaluation. It refers to the notion that indicators measuring impact should be specific, measurable, achievable, relevant and timely. 

Collect disaggregated data
In relation to the SDGs, collecting data has frequently been touted to ensure that “no one is left behind”. It means breaking statistics down so that it shows, for example, income, gender, age, race, ethnicity, migratory status, disability, location and rural vs. urban population. 

Promoting equality and inclusion is central to the SDGs, as well as to sport policy frameworks such as the Kazan Action Plan. The reason disaggregated data is so important is that, if we are to track overall progress towards these objectives, we need to know that the situation of the most vulnerable is improving. When formulating policy, disaggregated data also helps to demonstrate where investment would be best placed. 

Use outputs, outcomes and impact to define different sources of value
In “Enhancing the contribution of sport to the SDGs” (page 38), the Commonwealth Secretariat urges policymakers to distinguish between three different ways to define the ‘value’ of a policy:

Outputs: Usually measure the productivity of an intervention, but not its social, economic or environmental value. Examples include the number of participants, people engaged in an educational initiative or the demographics of those in a leadership role.

Outcomes: Measure changes to people’s lives, the economy or the environment as a result of a policy or intervention.

Impact: An effect that meets wider strategic goals. This is harder to measure because interventions usually occur in the context of many other influences. 

Follow a two-tiered approach to data collection
To tackle challenges pertaining to limited data availability and collection, the Kazan Action Plan and the Commonwealth Secretariat have recommended a two-tiered approach to ensure policymaking is based on robust data. 

  • Firstly, data collection on the national level should show “basic, general indicators” (e.g. budget allocations, physical infrastructure, workforce numbers and participation) and their disaggregation in relation to key variables of the SDG framework (e.g. sex, age and persons with disabilities). 
  • Secondly, more specific indicators “should allow governments to measure the contribution of sport to identified SDGs, targets and/or indicators that are prioritized according to their specific context.” These should distinguish between the direct and indirect contributions of sport policy interventions to specific SDG targets. 

The Kazan Action Plan and model indicators
The Kazan Action Plan was the result of the Sixth International Conference of Ministers and Senior Officials Responsible for Physical Education and Sport (MINEPS VI), in 2017. It identifies ten SDGs to which sport can contribute significantly, as well as 36 specific targets under those SDGs. One of the five key areas for action agreed at the MINEPS VI Conference is to “develop common indicators for measuring the contribution of physical education, physical activity and sport to prioritized SDGs and targets.” 

This action point comes out of a need to develop sport-specific indicators that relate to the SDGs. The Kazan Action Plan mentions the risk that “policy interventions in and through sport will be neglected, ineffective and/or insufficiently recognised”. That risk is due to the fact that, although sport is mentioned as an “enabler of sustainable development and peace” in Agenda 2030, it is not specifically mentioned in any goals or targets. The document also emphasises that target 17.18 focuses on high quality data and enhanced capacity building to support its collection.  

The Commonwealth Secretariat has led international efforts on the development of the common (or “model”) indicators - click here for more information on this initiative.

Further reading

The Sustainable Development Goals

Guides and example frameworks

Examples and data sets

Research related to M&E

E-Newsletter subscribe