Measuring and communicating program impacts using evaluation

The importance of measuring program benefits

Identifying the impacts of government funded projects through program evaluation has never been more important. Fiscal constraints and efficiency savings have put a strain on many government policies and programs and associated organisations, leading to greater focus on the demonstration of impacts by the recipients of funds.

Effective communication of program benefits and impacts are essential. Programs that are unable to do so are placed at a significant disadvantage when competing for funding.

The diverse nature of impacts and program benefits

The impacts and benefits of an organisation, policy or program, are likely to be diverse, wide reaching and affect different parts of the community and the economy. They may be a result of original objectives, or occur as spill over from other work being undertaken. Impacts can be economic, environmental or social.

More than just the traditional focus on economic impacts

Many studies focus on assessing the economic impacts of an organisation, policy or program. Economic impacts may include changes to Gross Domestic Product, employment or investment output. Techniques utilised to estimate these impacts, include:

• computable general equilibrium modelling
• cost benefit analysis
• financial modelling
• multi-criteria decision analysis.

Traditionally, the majority of organisations, policies and programs have focused on achieving economic impacts as part of their program benefits, however this has begun to shift. Increasingly the role of Government and the focus of organisations has begun to consider social and environmental outcomes, as well as economic outcomes.

The importance of considering the full range of impacts and program benefits

Measuring and communicating social and environmental impacts, as well as economic impacts, is vital. For example, ACIL Allen recently undertook a study of the economic, social and environmental impacts of the Cooperative Research Centres (CRC) Program. In addition to assessing the economic impacts, the study identified the breadth and scope of environmental and social impacts the CRC Program has produced. Impacts identified included:

• reduced greenhouse gas emissions
• protection of endangered species
• improved health and well-being
• improved safety
• increased business diversity.

The quantity and extent of environmental and social impacts demonstrated that purely focusing on economic impacts would have underestimated the true value of the CRC Program to Australia.

Measuring environmental and social impacts

The concentration on the economic impacts of an organisation, policy or program is, in part, because most economic impacts can be readily measured, valued and communicated in a way that is understood by key stakeholders, particularly through expression as a dollar figure.

While some environmental and social impacts can be valued in dollar terms, not all of these impacts naturally lend themselves to this type of quantification. Assigning values to such impacts is highly sensitive with, in many industries, very little supporting data and information.

The approaches used to measure and value social and environmental impacts are not as well recognised as economic approaches, particularly for social impacts. Environmental economists have developed a number of techniques to value the environment, including methods such as the market price, travel cost, benefit transfer and the hedonic pricing approach.

Importantly, social and environmental benefits tend to be characterised by increasing marginal returns, whereby the value gained can be amplified through sharing amongst stakeholders, in contrast to fiscal benefits that are fixed and can only be distributed between stakeholders in a zero-sum way. Holistic value measurement is one approach that ACIL Allen has used to compare the effective value of policy options, such as in measuring the outcomes of investment in education and other knowledge economies.

The importance of communicating impacts

Although the commodification of impacts may not always seem suitable, it is the direction much public policy is taking. The impetus falls on the organisation, policy or program to ensure that impacts are articulated in language understood by funding bodies. This approach is consistent with principles of accountability, performance monitoring and efficiency.

For many programs, policies and organisations with large economic impacts, a focus on these impacts provides sufficient evidence that they are performing well, but this may understate the benefits to the wider community. With continued fiscal pressures, the ability to identify and express diverse impacts will remain at the forefront of demonstrating success and benefit to the community.

Those organisations best able to gather, measure, value and articulate their impacts are at an advantage when talking to funding bodies and policy makers. It is important that organisations consider what their impacts are and how these can be best measured and communicated.