An economic evaluation of a statistical sample of 18 randomly selected clusters of FRDC investment across three programs has found that the average return to FRDC investment is 5.6 to 1.
All research projects that were completed or substantially completed between July 2002 and June 2007 were included in a population of projects, and this population was divided into 32 clusters of investment by grouping projects into homogenous subject areas. It was from these 32 clusters that the 18 clusters analysed were randomly selected. The total value of FRDC funding for this population was $92.76 million (nominal). The FRDC investment in the eighteen clusters analysed totalled $51.63 million therefore represented 56% of FRDC’s total investment (in nominal terms) in the population.
The majority of the number of benefits qualitatively identified from the 18 clusters of research was economic in nature, although significant numbers of environmental and social benefits were also identified. The major beneficiary of the impacts of the 18 clusters of research has been the fishing industry (61% of benefits identified), with 38% of the identified benefits being public in nature. The results demonstrate the significant spillovers of benefits to the public sector from research targeted at the fishing industry. There were limited benefits to other industries and to overseas.
A number of the identified benefits were quantified, and investment criteria for each of the clusters of investment calculated. Benefits were estimated over 30 years from the final year of investment in the research. Benefits and costs were expressed in 2008/09 dollar terms, and discounted to 2008/09 using a discount rate of 5%.
The net present values (NPVs) for total investment for the individual clusters ranged from $1.4 million to $160.2 million and the benefit:cost ratios (B/C Ratios) ranged from 1.2 to 33.8. On average, FRDC investment made up 39% of the total investment, and the NPVs for FRDC investment ranged from $0.6 million to $54.6 million.
When all 18 clusters are aggregated, the B/C Ratio for the $214 million investment in the 18 clusters (present value terms) is 5.6, with benefits (PVB) of $1,200 million, an NPV of $986 million, and an IRR of 24.3%. For the FRDC investment of $83 million, the NPV is $398 million.
The stratified sample included eight clusters from Program 1, eight from Program 2, and two clusters from Program 3. The NPVs for Programs 1 and 2 were $435 million and $538 million respectively, with the NPV for Program 3 being much lower at $13.1 million (reflecting the smaller number of clusters sampled). The B/C Ratios for Programs 1 and 2 were 5.3 and 6.1, while the IRRs were also similar with 25% and 23% respectively. For Program 3 however the B/C Ratio was significantly lower (2.7 to 1) and the IRR was significantly higher (61%).
In 2007 the Council of Rural Research & Development Corporation Chairs (now the Council of Rural Research & Development Corporations (CRRDC)) employed ACIL Tasman as a secretariat. One of the tasks undertaken by ACIL Tasman was to coordinate the Research and Development Corporations (RDCs) to report on the returns to the R&D investment from all RDCs as a single entity. This is being achieved through a process of asking each RDC to submit a number of cost-benefit analyses each year, which are then combined into a single report by ACIL Tasman that summarises the impact of the RDCs as a whole. The methodology for the CRRDCC process can be accessed at http://www.ruralrdc.com.au/Page/Evaluation+/Methodology.aspx
Two summary reports have been produced by the CRRDC so far (December 2008 and January 2010). The 2008 report included ‘hero’ projects and ‘random’ projects. Hero projects were selected as being able to demonstrate very high impact. FRDC contributed one hero analysis on the investment in one project associated with Marine Protected Areas (MPAs).
The random projects involved each RDC grouping their portfolio into clusters of projects with common outcomes, and then ACIL Tasman randomly selecting a number of clusters for the RDC to analyse. The completed analyses were then submitted to ACIL Tasman and the summary report produced to give an indication of the range and types of benefits across the RDCs. Not all RDCs submitted random analyses in the first year, including FRDC. Those who did submit provided a limited number. FRDC submitted evaluations for three random clusters for the 2009 report, and will submit an additional 15 clusters for the 2010 report. This report summarises the findings of the total of 18 randomly selected clusters for FRDC (the three already submitted to the CRRDC, and the additional 15).
Cluster Definition and Selection
FRDC and Agtrans clustered FRDC’s population of projects into 32 clusters, with each cluster ranging in size from 3 projects to 29 projects, and in cluster value from $0.27 to $8.20 million. Australian Fisheries Management Authority (AFMA) projects were excluded as they were simply managed on behalf of AFMA, and did not involve FRDC funds or decision making. The population included all projects completed or substantially completed between July 2002 and June 2007. The total value of FRDC funding for this population was $92.76 million (nominal dollar terms).
The final cluster definitions were approved by the FRDC Board and then submitted to ACIL Tasman. The clusters were stratified by Program, and within each Program by size (number of projects). It was noted that some clusters did include projects from more than one Program, however all clusters were able to be classified by Program based on the major focus of the cluster.
ACIL Tasman, in their economic evaluation guidelines, provides a recommendation of statistically how many clusters of investment should be contributed over three years, given a specific number of clusters.
ACIL Tasman then randomly chose the 18 clusters for analysis. Three of these (the Salmon Aquaculture Subprogram, the Southern Bluefin Tuna Aquaculture Subprogram, and the spatial analysis cluster) were forced into the stratified sample as Agtrans had previously undertaken some analyses for the Aquafin CRC and FRDC that included some of these projects. These three analyses were undertaken first, as being familiar with the clusters allowed the clusters to be completed in time to submit to ACIL Tasman for the January 2010 summary report (originally planned to be released in October 2009).
The eighteen clusters of investment randomly selected for analysis are listed here
Lessons Learnt for Future Investment
A number of lessons learnt for future investment, and for future evaluations, were identified from each cluster. These lessons are summarised below:
Lessons relating to evaluation
•Strategic research can be successful in developing knowledge and understanding but outcomes can be limited in terms of quickly producing products or practices that can be utilised by seafood producers in the shorter term. The outputs from these projects are being used in other projects currently being funded by FRDC. Any future analysis of such projects should be sure to acknowledge the contributions of these earlier strategic projects, and double counting of quantified benefits should be avoided.
•Innovations that apply to new industries that are growing (whether stimulated by the investment or not) have a greater capacity to provide higher return to R&D than small and /or slow growing industries.
•The size of the return to investment is significantly influenced by the size of the industry being targeted.
•There were sometimes difficulties contacting and obtaining responses from appropriate personnel, particularly for older projects where some principal investigators had retired or moved overseas, and other people in the research organisations were not knowledgeable enough of the industry or the specific project to be of assistance. FRDC could attempt to ensure some form of corporate responsibility for project knowledge is addressed in research organisations, but this may be difficult to enforce. This issue is part of a larger challenge of improving the tracking of outcomes for all investments.
•The inability to value with any confidence changes in biodiversity features of fishing areas should be noted. While it may be possible to value extinction of a marine species through the community’s willingness to pay, it would be far more difficult to make credible assumptions about how the improvements to habitat features (e.g. more protection from fishing given to seamounts and shelfs) contribute to biodiversity and reduce the risk of declining biodiversity or extinction.
•For projects targeted at prevention of disease and pest problems, benefits can be difficult to demonstrate in the absence of a disease outbreak.
•The government’s rural R&D priorities do not accommodate a number of social benefits such as improved recreational experiences for fishers and improved health and safety. The implication of this is that these areas of research investment should be given a low priority by FRDC or that national RD&E priorities should be changed to provide some focus on social benefits.
•FRDC’s past KPIs did not encompass some of the areas of research funded such as improvements in recreational fishing experiences and improved health and safety.
•Authoritative information on the costs along the value chains for wild catch destined for domestic and/or export markets are not readily available; this hinders effective evaluation. It is understood some work that might contribute to this is currently being undertaken as part of the Seafood CRC.
•There was little integrated information available on the current processes, accuracies and costs of existing stock assessments to provide a baseline for measuring improvements. A standardised process for assembling such information could be helpful to FRDC in assessing proposals for improving stock assessments in future.
•The approach developed for valuing the benefits from the population dynamics and stock assessment clusters could be used by FRDC for ex ante assessments of proposals aiming to improve stock assessments. An additional variable on the probability of success of the proposed project would need to be added if the approach was to be used for that purpose.
•It would be helpful to future evaluations if the FRDC project management system were able to more easily extract funding information by financial year across a range of individual R&D areas. While some capacity to do this does exist, the system could be made more friendly by requiring project managers to provide a wider range of category inputs at the time of contracting or final report assessment. Such an initiative may prove valuable not only in evaluation but also in planning and reporting.
•In the interests of addressing the marginality issue, it could be of interest for FRDC to document the workshops and conferences not supported and the criteria on which funding decisions were made. Such a principle could apply also to project selection/rejection in other R&D areas.
Lessons relating to future investment and management
•For a number of projects there were large gaps between the first few years of the investment and the final year of investment. The timing of arrival and acceptance of final reports, and the final payment for each project should be investigated. Possible reasons for the late timing were that draft reports were very late, it took significant time between the draft and final reports, or final reports were finalised but then it took considerable time to make final payments.
•FRDC could consider developing cost of production models for aquaculture enterprises that include unit feed costs, feed conversion ratios and other production parameters in order to assess research priorities and individual research proposals, as well as assist in the communication of research results to the relevant industries.
•Significant gains from win-win situations for addressing environment and productivity issues in the fishing industry have been demonstrated, a similar situation to that observed in research and development success for agriculture.
•The salmon aquaculture cluster demonstrated the importance of identifying and addressing high priority issues for the industry and utilising existing research from overseas that can be adapted to Australian situations.
•Backgrounding emerging issues and preparing for change are important for FRDC in order to respond effectively and quickly to sudden external policy changes that affect the fishing industry.
•Subprogram structures can be of value in assisting with priority setting across the research – development spectrum, and across different sectors and issues.
•The introduction of a marketing levy could be reconsidered by the Australian Government and industry due to the apparent need for market and marketing investment by the seafood industries. This need has been met to a large extent by FRDC but this diverts resources away from R&D investment.
•Investments that result in improvements to the quality of life of individuals, through improved health and safety or reduced stress, can yield significant benefits due to the value placed on quality of life through willingness to pay studies.
•If leverage factors for projects in different R&D areas were available at an R&D area level (for both FRDC project funding and for the national context), they may be important in assessing the FRDC current and prospective roles in different areas and where public benefits are manifest but where funding is difficult to attract.
The evaluations carried out were ex-post evaluations, as they were undertaken after the completion of each project. Economic evaluations can also be ex-ante in nature, in that they take place prior to the project being undertaken, and they seek to predict the likely benefit from the research. There are a number of ways in which ex-ante evaluation can be used, and stages at which it can be implemented. As for ex-post evaluation, it may not be cost-effective to undertake a cost-benefit analysis on every single investment funded by the Corporation, and therefore selection of those investments on which ex-ante analyses should be undertaken is important. This should relate to the purpose of undertaking the ex-ante evaluation. Three potential uses of ex-ante evaluation are described here:
1.As part of the proposal process: Those submitting project proposals could be asked to complete a basic cost-benefit analysis and actually produce expected investment criteria. Experience has shown however that this is not often an effective tool in decision making regarding resource allocation. When the analyses are carried out by the proposers they are often inconsistent and incomparable as a range of different methods, data, and assumptions are used. The researchers will often not have the skills to complete the analysis and it can become very time consuming or costly for the proponent. The quality of what is submitted is questionable and therefore negates its purpose. However, the logic involved in the researcher developing such an analysis is of great value in thinking through the eventual impact of the research, and options for including this thinking (without taking it through to investment criteria) are discussed later. An alternative would be to develop an industry model into which key variables could be entered by the proponents, however there is possibly too much variety in fisheries industries to have a standard model that could be completed and be ‘user-friendly’.
2.As part of the funding decision process: Cost-benefit analysis is a tool that can be used to aid in R&D funding decisions when choosing between projects at the margin of being funded. In the past, one RDC we are aware of had a funding process whereby each Program had a limited budget, and projects were prioritised by Program committees and funded within that budget. Often, there was a situation where there was only enough funding remaining to fund one more project, but there were three projects that it was difficult to decide between. These three projects at the margin would be subject to ex-ante cost-benefit analyses to aid with the funding decision. It was noted by the decision makers using these analyses that it was not the actual investment criteria that were used in the decision, but rather the enhanced project logic and assumptions that were of value in comparing the projects. This approach could also be useful in assessing marginality (over or under investment in research) by completing (after project completion) cost benefit analyses for marginal projects that were funded.
3.As an M&E tool for large investments: Ex-ante cost-benefit analyses can be a valuable tool for developing an M&E framework for large investments, or subprograms of investment. Following the decision to fund the investment, an ex-ante cost-benefit analysis can be undertaken. The process of defining the expected outputs, outcomes and benefits is useful for confirming the intentions of the investment and identifying any potential risks or flaws in the investment. It can also aid with identifying areas for appropriate industry involvement and for exploiting potential synergies between projects. The process of undertaking the quantitative part of the analysis helps to identify the key data that should be collected throughout the life of the investments to assist in evaluating its impact at the completion of the investment. It can also be valuable for ongoing project management and ongoing assessment of the progress of an investment with respect to its planned objectives, particularly with respect to target audiences, extension and adoption.
The key conclusions from this report are:
•The majority of the benefits identified from these 18 clusters of research have been economic in nature, although significant environmental and social benefits have also been identified.
•The major beneficiary of the impacts of the 18 clusters of research has been the Australian fishing industry, but with still approximately one third of the benefits identified being public benefits. There were limited benefits to other industries and to overseas interests.
•The NPVs for total investment for the individual clusters ranged from $1.4 million to $160.2 million and the B/C Ratios range from 1.2 to 33.8. On average, FRDC investment made up 39% of the total investment, and the NPVs for FRDC investment ranged from $0.6 million to $54.6 million across the clusters.
•When all 18 clusters are aggregated, the return to the $214 million investment in the 18 clusters (present value terms) is 5.6 to 1, with benefits (PVB) of $1,200 million, an NPV of $986.1, and an IRR of 24.3%. For the FRDC investment of $83.2 million, the NPV is $398 million.
•For the aggregate investment criteria, Programs 1 and 2 had similar returns, while Program 3 (with only 2 clusters) had a low B/C Ratio but a high IRR due to the early capture of the benefits valued.
•It was estimated that the NPV for the investment in the entire portfolio of 32 clusters (including the 14 clusters not quantified), would approximate $1,764 million (over 30 years from last year of investment, discount rate of 5%).
•The confidence in the coverage of benefits was medium to high, while the confidence in the assumptions used was low to medium.
•Of the 215 projects included in the 18 clusters, 46 of these did not contribute to the benefits quantified (21%). The categories of reasons for a project not contributing to the benefits valued were that the project largely produced strategic knowledge (10%); that there was limited evidence of adoption/impact or only a minor benefit compared to other benefits valued (70%); or that there were no benefits identified (20%).
•It was difficult to draw conclusions regarding the additionality and marginality of the investments analysed. It was concluded that if public funding to FRDC was reduced, that spillover benefits to the public would also be reduced. However, the degree to which this reduction would occur would depend on the relative priorities for funding individual research areas between FRDC, the industry itself and other funders.
•The majority of the benefits valued accrue to the first rural research priority (productivity and adding value). There was also a significant contribution from the clusters to the third rural research priority (natural resource management).
•The majority of the clusters evaluated contributed to national research priority 1 (an environmentally sustainable Australia) and national research priority 2 (frontier technologies for building and transforming Australian industries).
•The clusters contributed to the five Strategic Challenges fairly evenly. However, there were a number of clusters that did not contribute to any specific KPIs, and a number of KPIs that no clusters addressed. It is noted that these KPIs were only defined for the period 2005-2010, and that the funding for much of the research in these clusters commenced before this time.
•A number of issues were identified with implications for FRDC priority setting, management and future evaluation.
Full report [DOC] available for download