How a pathway to impact approach measures progress

Evaluating a new organisation, strategy, service or program presents a practical challenge: how do you assess whether it is succeeding when the change it was created to achieve may take years to emerge? For mature organisations delivering well-understood services, conventional evaluation models (assessing results against objectives at the end of a funding cycle) serve well. The challenge arises with newly established organisations, strategies or programs seeking transformational or systems-level change, where the outcomes that matter most sit well beyond the horizon of the first evaluation.

Early on, much of the important work happens well before those longer-term outcomes are visible. Systems are being established, governance is taking shape, relationships are being built and new ways of working are being tested. Assessing performance primarily against ultimate outcomes at this stage can miss this progress, or make an organisation appear to be underperforming against results it could not reasonably be expected to have achieved yet.

It is a little like judging a building by its footings. The question is not yet whether the building is finished, but whether the right foundations are being laid. That is where a pathway to impact approach can help. This isn’t a radical new model or theory, rather a return to first principles. Over the years we have found this to be particularly effective in dealing with complexity and can leave our clients with more practical tools and metrics that they can continue to use when our engagement ceases.

Pathway to impact approach

A pathway to impact approach builds on established tools of evaluation practice. A program or outcome logic describes how a program’s inputs and activities are expected to produce outputs, outcomes and, ultimately, longer-term impact¹. A theory of change goes a step further, making explicit the assumptions and causal mechanisms that connect each link in that chain (why we believe doing this will lead to that). The two are complementary, the logic model maps the route, and the theory of change explains why the route should work.

The program or outcome logic provides a shared account of what “on track” looks like and a basis for assessing early progress. Each link in the causal logic can be explicit and tested as activity progresses, otherwise known as contribution analysis². A program or outcome logic should ideally developed with the people who will be held to it and impacted by it, such as boards, leadership, staff responsible for delivery and implementation, funders, delivery partners and the communities affected. Co-production workshops and external consultation stress-test the logic, surface blind spots and gain agreement on what progress should look like at each stage. In our experience, balancing ambition with feasibility can be a key challenge but is well teased out with independent facilitation.

A pathway to impact approach structures the evaluation framework around this journey from design objectives, inputs and activities, to outputs, and how these translate into short- and longer-term outcomes. It aligns directly with program or outcome logic elements while still attending to the core elements of evaluation commonly assessed in relation to government funding — appropriateness, efficiency, effectiveness, impact³.

Taking a pathway to impact approach also facilitates undertaking a phased approach to evaluation that considers the various stages at which enablers are in place, change begins to show and impact starts. For example:

1. In the short term, a formative evaluation can assess both process and early outcomes, providing the opportunity to address barriers to, or support required for, the realisation of longer-term outcomes
2. In the longer term, a summative evaluation with more time for outcomes to be realised can undertake a more fulsome assessment of the impacts achieved, with sufficient time for benefits to be fully realised.

Assessing contribution analysis maturity

Assessment against the pathway needs a scale that recognises maturity. One way to do this is to establish a scale articulating what outcomes look like at each point, for example across stages such as Establishing, Delivering, Influencing and Transforming as outlined in Figure.1. An organisation one-year into a five-year establishment phase can be assessed as performing strongly at “Delivering” without any pretence that it has yet reached “Transforming”. This approach also reveals where one outcome area is moving quickly because it builds on existing relationships, another moving slowly because it depends on infrastructure that may take longer to stand up.

Figure.1 Four-stage maturity scale for assessing progress towards outcomes

^{Source: ACIL Allen}

Where the approach fits best

In our experience, a pathway to impact approach fits best wherever the gap between activity and impact is long, indirect or heavily mediated by others. That includes:

• newly established organisations, where the early story is about capability, governance and relationships rather than results
• programs where impact is realised through the work of others such as grant programs and capacity building which may take time to become visible at scale or where success appears first as changes in other people’s behaviours, practices and decisions
• strategies where delivery is dispersed across multiple agencies or partners
• services built on trust and relationships, such as partnerships with communities, where the strength of the relationship is itself an early outcome and a precondition for everything that follows.

Where we have applied it

We recently developed a program logic, evaluation framework and data collection tools for a newly established national body created to change the way research in a complex social policy area is funded and conducted, with a mandate for genuine partnership between government, researchers and the communities most affected. Two years into its establishment, the core evaluation question was not whether it had transformed its sector, it was whether it was building the right foundations to do so.

That framing allowed the evaluation to take seriously the things a conventional outcomes evaluation would have struggled to count: the quality of governance and process, the depth of co-design, the growth of a network, and the establishment of trust with a community that had good reason to be sceptical of a new body.

The value of a pathway to impact approach

The value of a pathway to impact approach lies in matching the evidence to the decision at hand. It provides a disciplined way to judge progress now, while building the evidence needed to assess longer-term outcomes later. Impact may take years to emerge, evidence that you are on the right path should not.

¹ W.K. Kellogg Foundation (2004) Logic Model Development Guide, W.K. Kellogg Foundation, Battle Creek, Michigan. Retrieved from: https://wkkf.issuelab.org/resource/logic-model-development-guide.html

² Mayne, J. (2008) Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16, Institutional Learning and Change Initiative. Retrieved from: https://www.betterevaluation.org/sites/default/files/ILAC_Brief16_Contribution_Analysis.pdf

³ Australian Centre for Evaluation, Evaluation guidance and resources, Department of the Treasury, Australian Government, Retrieved from: https://evaluation.treasury.gov.au/index.php/about/about-australian-centre-evaluation