Measuring Research, Making a Difference
One of the things that has become clear in discussions with people leading up to the workshop is that the language we use to discuss research impact is a barrier in its own right. Confusion over terminology and interpretations of the agenda behind terminology are a significant source of problems. To help deal with this I am suggesting the following set of terms and some of the distinctions they imply as a useful way to talk about these issues.
Output: Any form of object generated directly by a research activity. This may include research papers, articles, posters, abstracts, data, samples, software, methodology, online writing, reports, and other grey literature. Outputs may be open (as defined by the Budapest declaration including the right to re-use), publicly accessible (freely viewable or obtainable), or partly accessible (if for instance they are available to subscribers, or to professional researchers, or for specific fields of use such as non-commercial use). Prior to release or publication an output is private. Outputs are the direct result of research and are therefore distinguished from outcomes.
Outcomes: Outcomes are events or changes that can be traced more or less directly to the influence of a research activity. These include; wider adoption of developed methodology, adoption of a new conceptual framework, changes to clinical guidelines, development of specific products (including medical products), changes to policy or procedures, development of a commercial enterprise, the training of research personnel, increased research efficiency. Outcomes are therefore tightly coupled to the re-use of research outputs. Outcomes can be indicator events if they are single events but may also be broader e.g. a single citation is an indicator event, an outcome would be the quantitation of all citations to a given output. Not all outcomes are quantifiable or measurable.
Indicator events: Indicator events (in the CASRAI framework) are specific occurrences that can be traced to one or more specific research activity. The generation of outputs and specific outcomes can both be indicator events. David Baker defines an indicator event as “a single recordable occurrence that changes the count of one or multiple impact indicators and that can be related to one or more research personnel or activity”.
Impact: Impact is a measure of the influence of outputs and outcomes of research. The measure may be qualitative or quantitative and may be absolute or comparative. To determine impact is to ask the question: how different would the world be if this research had not been carried out? Outcomes can provide evidence to support an assessment of impact. Outputs by contrast can not, in and of themselves, be evidence of impact. Impact may be felt in economic, social, policy, environmental, educational, or research arenas. Qualitative assessment of impact may be based on a narrative describing outcomes and indicator events. Quantitative assessment of impact may be based on one or more measures of outcomes and/or indicator events. Quantifiable outcomes and indicator events are used as proxies for impact or may be fed into a model which attempts to directly estimate or provide a comparative measure of impact.
Research Activity: Broadly any activity or endeavour that seeks to advance knowledge, understanding, or capability. Research may be directly funded or indirectly funded (often referred to incorrectly as “unfunded”). It may be undertaken by professional or amateur researchers and may or may not occur in the context of a research institution, including commercial and industrial institutions. The majority of current research impact derives from research activities carried out by professional researchers in academic and commercial research institutions.
What have I missed? What other terminology do we need to sort out?