WWF-Australia science and evaluation case study

Dr Squirrel Main, Romola Stewart and Lachlan Florez

Why a new approach was needed

WWF-Australia runs one of the most active science and research programs of any conservation organisation in the country. Over the past 5 years, its annual research activity has roughly doubled. While WWF-Australia has been producing a high volume of science, the challenge has been ensuring that the research has been changing what’s done in the field, and in the offices of policymakers.

Conservation programs operate across long time horizons. A research finding that shapes a policy decision today may have been planted in a relationship formed 3 years ago, via a dataset collected 5 years before. Tracing that chain of cause and effect is hard enough. Doing it across 88 separate science and research projects, spanning ecological fieldwork, Indigenous knowledge systems, social science, and policy advocacy, makes it harder still.

When WWF-Australia asked ACIL Allen to evaluate the program, the question on the table was deceptively simple: does science lead to influence, and if so, how? In practice, answering it turned out to be a journey into Kumu and systems mapping.

Traditional evaluation approaches were not well suited to the task. A standard survey or document review could tell us what projects produced, datasets, articles, reports, but not how those outputs travelled through the organisation or out into the world. A simple logic model could map inputs to outputs, but it could not show the conditions under which science becomes influential, or why some projects punch well above their weight while others produce excellent evidence that sits on a shelf.

WWF-Australia also had a concern that, in some instances, science happening inside the organisation may be invisible, flying under the radar. The team knew of projects not connected to the broader strategy or ‘captured’ as research. Before you can evaluate the impact of science, you must know what is occurring.

Methodology

The conceptual foundation: starting with the system, not the outputs

The core of our approach was a causal pathways analysis anchored to WWF's Theory of Change. What made it novel was connecting evidence about enablers and mechanisms directly to the levers and outcomes in the Theory of Change.

The Theory of Change describes a chain of intent: if WWF strengthens collaborative networks, enables high-quality research, and promotes epistemic justice, then researchers will communicate their findings, program managers will use evidence-based approaches, and policymakers will incorporate research into policy. The long-term aspiration is for WWF to become a genuinely evidence-based organisation whose science drives conservation outcomes at scale.

The problem with most Theory of Change evaluations is that they treat the theory as a series of boxes connected by arrows and then ask whether the arrows fired. That approach can tell you if outcomes were achieved. It cannot tell you why, or which conditions made the difference.

We flipped the question. Instead of asking "did the outcomes occur?", we asked survey respondents to describe what worked well in their projects and what caused it to work. Those responses were coded thematically using our in-house AI and mapped as nodes and relationships in Kumu. The resulting systems map shows not just which mechanisms were activated, but which enabling conditions gave rise to them and how those conditions connect back to the levers in the Theory of Change.

This builds directly on work developed by Matt Healey and colleagues,^[1] published in the Evaluation Journal of Australasia, which demonstrated how systems mapping in Kumu can be used to track and visualise change in complex programs. Our contribution was to link that approach explicitly to a Theory of Change, so that the systems map became a tool for evaluative judgement, not just visualisation.

Design principles: what the approach had to do

WWF had been running an annual survey for 5 years to strengthen science as an enabler of impact we wanted to test contribution pathways. With 88 projects spanning multiple years, stakeholders, and outcome areas, we could not attribute specific conservation outcomes to specific research activities. What we could do was trace the conditions and mechanisms that distinguished more influential projects from less influential ones and use that evidence to make evaluative judgements about the levers in the Theory of Change.

The analysis worked in three layers.

• Descriptive. What was WWF's science and research portfolio actually doing? We used the annual survey data, supplemented by a second survey item focused on contribution pathways, to map activity across outcome areas, types of output, and types of utilisation. This established the baseline and confirmed the hunch that some of WWF's science was invisible and not recognised, tracked, or connected to the current organisational strategy.
• Comparative.What distinguished more influential projects from less influential ones? All 88 projects were mapped against the rubric, and each was scored as emerging, influential, or transformative. We then looked at what the influential and transformative projects had in common. Four themes emerged:
  1. durable evidence products such as datasets and peer-reviewed articles;
  2. clear pathways to government decision-makers;
  3. translation into tools, management practices, or investment decisions; and
  4. Indigenous-led or Country-based work. Influential and transformative projects were three to seven times more likely to exhibit these features than emerging projects.
• Systemic. What conditions give rise to influence? This is where the Kumu systems map came in. We asked researchers to describe what worked well in their projects, and what caused it to work well. Those responses were coded and mapped as a network of around 40 conditions and mechanisms. We worked with team at Kumu to create a map that showed which conditions tend to activate which mechanisms, and which mechanisms connect to which outcomes in the Theory of Change. The conditions that researchers most reported as present in their work were long-term funding, trusted partnerships, and science being built into projects from the start. These conditions also had strongest pathways to policy influence and high-quality research. In other words, what WWF was doing well lined up with what the evidence says matters. The conditions that were least present (pausing to ask whose knowledge counts and why and actively making space for Indigenous and other knowledge systems) were also visible. The systems map made the connections between these gaps visible, moreso than a standard survey result graph.

The Kumu map is interactive and publicly accessible, allowing WWF's team to interrogate the system themselves: tracing which conditions strengthen which mechanisms, and which mechanisms connect to which outcomes. This transforms the systems map from a snapshot into an ongoing learning tool.

What the evaluation found

Our Kumu-rubric evaluation approach revealed several findings.

First, the basic survey analysis showed science activity at WWF has been growing strongly but translation into influence was inconsistent. The challenge is not how much science we produce – it is whether that science is changing what happens in the field and in the office of decision-makers.

Second, the rubric analysis indicated this was not primarily a resourcing problem. Influential projects were not always better funded. Instead their design started with a clear pathway to a decision or a decision-maker. They invested in translation (turning evidence into briefings, tools, or narratives). And they were significantly more likely to involve Indigenous knowledge systems and leadership, with Indigenous-led or Country-based projects around seven times more likely to be classified as influential or transformative.

Third, the systems map showed that the conditions most strongly associated with influence (long-term relationships, values-aligned partnerships, and early design integration) were also the conditions most difficult to create under short-term, project-based funding cycles. The map made visible the mismatch between the time horizons that influential science requires and funding timelines.

Lastly, the evaluation found that traditional publishing pathways of peer-reviewed journals with timelines of 18–24 months were providing diminishing return on investment for an advocacy-facing organisation that needs evidence to be responsive, accessible, and useful at the point of decision. Peer-reviewed science may have some value, but the pathway from research to influence appeared to occur through ongoing translation.

What this means beyond WWF

Any organisation that generates evidence to influence outcomes faces the challenge of connecting what it knows to what it does and connecting what it does to what changes. A systems map anchored to a Theory of Change gives organisations a practical way to diagnose where that connection breaks down and what conditions would strengthen it.

For evaluation practitioners, the approach demonstrates that systems mapping tools like Kumu can be linked to an evaluative framework and populated with data from real program participants. The tools become analytical instruments capable of showing which enabling conditions have the strongest pathways to the outcomes an organisation is trying to achieve, and which are being systematically under-invested in.

For ACIL Allen, this project is an example of what happens when you treat evaluation as a design problem rather than a reporting exercise. We are drawn to evaluation problems that do not have an obvious method sitting on the shelf; where the answer requires building something new, testing it against real evidence, and being curious about what it reveals.