Evaluating the outcome of a research project in an annual report is difficult because many projects run over multiple years and there is a period of time between when R&D is undertaken, completed and then adopted by end users as to when the total value of the investment is realised.
The time scale can also vary depending on the activity undertaken. While there can be an instant impact from a project — resulting in change of practices or management arrangements for example — the total outcome may take time to accrue and that can only be measured when looking back.
The FRDC has put in place metrics to anticipate potential value (ex ante) and a formal measurement process to evaluate benefit cost (poste ante), which aligns with the Council of Rural Research and Development Corporations (CRRDC) evaluation framework.
The first population of projects was defined in January 2011 and projects were placed into clusters. Where there were high numbers of projects, these divided more specific groupings and this resulted in a total of 25 clusters across the 14 themes being assessed.
Evaluation of the first eight clusters was completed in October 2012 and the evaluation report provided to FRDC. The second round of evaluation was undertaken between July 2013 and December 2014 with nine clusters (2015 assessments outlined below) subjected to impact assessment. The assessment used benefit cost analysis to estimate investment criteria for each cluster of projects.
The evaluation program being undertaken by the FRDC is part of the CRRDC work to collaboratively implement a framework of benefit cost analysis to evaluate R&D activities.
The FRDC assessment uses the methodology developed by the rural RDCs benefit cost framework which is based on the work of the Department of Finance in Introduction to Cost-Benefit Analysis and Alternative Evaluation Methodologies, and subsequent discussions with the department to refine the methodology.
Generating and documenting evidence of impact and demonstrating performance of the RDCs as a collective is also a key objective for the CRRDC.
Over 2015-16 the FRDC, along with the other RDCs in the CRRDC, has funded a consortium of evaluation consultants to update the evaluation analysis to cover the period 2010–15 and develop an updated framework for ongoing monitoring and reporting across the RDCs. This builds on the work done by each RDC, but it is the first major cross-portfolio review in the past five years.
A stocktake of current evaluation activities of rural RDCs for the CRRDC was conducted by Lange Consulting & Software in the second half of 2015. The stocktake aimed to document existing evaluation processes and practices and inform the development of a high-level joint RDC performance framework.
From the recommendations, an opportunity was identified to standardise practices, terminology, measurement of intangibles, metrics and performance measures against Australian Government strategic research priorities for benefit cost analysis. This will better enable aggregation of performance data and provide improved visibility of the cost effectiveness of research programs across each RDC’s portfolio.
Developing a Benefit-Cost Evaluation Framework
Individually, Research and Development Corporations (RDCs) use a range of evaluation approaches to report value to stakeholders and to provide recommendations and guidelines for ongoing and future investment. In 2007 the RDCs agreed to work together to measure and report on the overall return on R&D investment.
The Council of Rural Research and Development Corporations’ Chairs (CRRDCC) prepared the evaluation framework, methodology and approach for this work. These were reviewed by key economic agencies of the Australian Government including the:
Following agreement being reached on the framework the 15 RDCs engaged independent economic consultants to undertake the first phase of evaluations. The results for the first round were released in November 2008. For more information, please see the CRRDCC [PDF].