Defining Success for Translational Research Organizations

See allHide authors and affiliations

Science Translational Medicine  03 Aug 2011:
Vol. 3, Issue 94, pp. 94cm20
DOI: 10.1126/scitranslmed.3002085


Translational research organizations (TROs) seek to enhance the clinical impact of scientific discoveries and play a multifaceted role characterized by multidisciplinary collaboration, outreach initiatives, and the provision of shared resources and facilities. Given this complexity, TROs require a flexible framework for performance assessment that tracks their progress, incentivizes fruitful activities, and aligns individuals throughout the organization. We suggest a framework that assesses TRO performance along seven main dimensions—funding, talent, creation, validation, dissemination, external uptake, and collaboration—and we encourage individual organizations to develop additional metrics as needed.

Despite heavy investments of time and money, only 14% of new health-related scientific discoveries are applied to day-to-day clinical practice, and translation takes an average of 17 years (1, 2). Translational research organizations (TROs) have been established to enhance the impact of scientific discoveries on disease prevention, diagnosis, and treatment. TROs can engage directly in the research process by employing a dedicated set of member scientists (for example, the Harvard NeuroDiscovery Center and Translational Genomics Research Institute) or indirectly by funding, monitoring, guiding, and providing shared resources for translational research (for example, the Michael J. Fox Foundation and Project ALS). By promoting cross-functional collaboration, outreach initiatives, and sharing of resources, TROs increase productivity and hasten the rate at which discoveries progress from basic research to clinically viable technologies and therapies (3).

In carrying out these functions, a TRO needs a flexible framework for performance assessment that tracks the organization’s progress, incentivizes activities that breed success, and aligns the organization with its goals. This Commentary seeks to fill this gap by proposing a general framework to serve as a resource for TROs as they tackle the issue of performance assessment.

A disciplined process of articulating goals and monitoring metrics for achievement will increase a TRO’s probability of success and speed of progression against key milestones. The need for an effective system of TRO performance assessment is particularly urgent as translational research is gaining prominence domestically and abroad: The U.S. National Institutes of Health (NIH) announced plans to found its own TRO, the National Center for Advancing Translational Sciences (4, 5); the UK just boosted public funding for translational research grants to £775 million, representing a 30% increase since 2007 (6). However, interviews with multiple TROs have revealed that there is no consensus on how best to assess a TRO’s performance, and many such organizations have expressed dissatisfaction with the adequacy and breadth of their existing assessment methodologies and frameworks (7).

We concentrate on TROs in the so-called T1 space—research efforts that focus on the bench-to-bedside translation of basic research findings into clinically relevant therapies and technologies (8). We also concentrate on nonprofit TROs, although some elements of the proposed framework may apply to for-profit TROs as well. The issue of performance assessment is particularly challenging when profit is not the guiding light; nonprofit TROs must generate their own goals and develop appropriate criteria for measuring success relative to these goals. The audience for our proposed framework includes the following TRO stakeholders: a TRO’s scientific leadership and governing body (board of trustees or oversight council), patients and their families, community representatives, governmental institutions, and private donors.


Many TROs were created to help cure a specific disease. Although this may be their ultimate objective, it cannot be the sole goal of the TRO. Finding a cure usually takes many years of basic disease research, multiple approaches to therapeutic discovery and development, and a healthy amount of luck. On the other hand, few TRO funders are satisfied with the statement “We have smart scientists who are working hard on discovering a cure for the disease” (9, 10). During the lengthy and complex process of searching for cures, a TRO needs a performance assessment system with specific intermediate goals against which to measure progress.

Because of differences across TROs in scope and breadth of mission, it is neither practical nor well advised to promote the application of a single identical performance assessment system across all TROs. Our goal instead is to provide a high-level framework with seven key dimensions for performance assessment to be used by TROs as part of their strategic planning process. Applying this framework, stakeholders within the TRO should engage in a process of collective goal setting and prioritization of these goals in light of the TRO’s financial resources and scientific expertise. Then, they should develop key performance indicators (KPIs) and metrics by which to track progress toward goals.


Performance assessment of TROs is important for many reasons. First, the scientists who head most TROs must be able to clearly communicate progress toward goals to their employees and interested outside parties, such as patient communities. Second, performance assessment helps TROs highlight areas of concrete achievement short of curing a disease—the return on investment to donors. Third, performance assessment helps TROs identify areas for improvement and provides an opportunity to capture lessons learned. Lastly, performance assessment system design stimulates discussion and clear articulation of the TRO’s mission among multiple stakeholders. This process helps TROs refine their mission and obtain buy-in for the performance assessment system throughout the organization.

When implementing a performance assessment system, a TRO should consider three different levels: performance dimensions, KPIs, and metrics. Performance dimensions are the highest, most generalized level (11). KPIs represent the next level of performance assessment and are the criteria used to evaluate each performance dimension. Metrics comprise the most granular level of performance assessment and are used to gauge the magnitude of performance against a particular KPI.

The seven key dimensions that TROs should use to assess performance are funding, talent, creation, validation, dissemination, external uptake, and collaboration (Fig. 1) (12). The relative importance of each dimension will vary among TROs, which may decide to de-emphasize one or more dimensions in light of their goals. However, most TROs should consider all seven dimensions because many are interrelated and mutually reinforcing. Our discussion also suggests several KPIs and commonly used metrics that may be applied to each performance dimension (Fig. 2). However, this is not meant as an exhaustive list, and TROs are encouraged to develop their own KPIs and metrics when appropriate.

Fig. 1. Multidimensional assessment.

TROs should assess performance along seven dimensions: funding, talent, creation, validation, dissemination, external uptake, and collaboration. Questions to ask regarding each metric include: Is our work financially sustainable? (Funding); Are we recruiting and retaining the best people? (Talent); Are we generating enough important output (pipeline)? (Creation); Do others believe our output is well founded and innovative? (Validation); Are we sharing the results of our work? (Dissemination); Are commercial or other external entities investing in our output? (Uptake); Are we bringing people together and providing useful tools and facilities? (Collaboration)

Fig. 2. Illustrative measures.

Shown are sample performance assessment KPIs and metrics. We suggest that TROs consider these parameters in order to assess performance along each dimension (23).


Funding. Funding indicates a TRO’s financial sustainability and can be both a leading and lagging indicator of performance. As a leading indicator, a TRO that garners large amounts of donations and grant funding is in a better position to maximize support for researchers and promising projects. Strong funding improves the chances of discovering a breakthrough treatment and external uptake such as licensing or other financial investments by a third party. As a lagging indicator, follow-on funding can be a sign of an organization’s satisfactory progress toward key milestones to date.

On the other hand, organizations that focus too heavily on funding may select projects on the basis of the availability of funding rather than on the basis of its fit with the TRO’s mission. For example, organizations may prioritize lower-risk, proven projects if these short-term wins are more likely to result in more follow-on funding. To avoid such gaming of the performance assessment system, TROs should assign a risk rating to every project—reflecting its chances of success relative to its goals—and develop an overall portfolio with diversified risk levels.

KPIs for funding can include grant, foundation, and private donation dollars as well as in-kind donations, such as equipment, reagents, research tools, or lab space. Another important KPI is the degree of funding diversification, which lowers the risk of project delays or cancellations upon loss of a key funding source. TROs should also consider creating subgroups of grants and donations to enhance fund-raising performance tracking. Similarly, TROs should specify the portion of their revenue allocated to general administration and overhead because this statistic is important to many funders.

Talent. Talent addresses the ability of a TRO to recruit and retain researchers. Like funding, talent is both a leading and lagging indicator of an organization’s success in its translational activities. As a leading indicator, outstanding performance along the talent dimension indicates that a TRO is positioned for future success. By recruiting the most qualified, respected, and productive researchers, a TRO is much more likely to generate breakthroughs that result in external uptake.

As a lagging indicator, performance on the talent dimension may reflect the efficacy of the TRO’s past translational research initiatives or the perception of these initiatives throughout the wider scientific community. Scientists who have proven research and funding track records are more likely to be attracted to a successful TRO that has a reputation for moving a field forward.

TROs could consider quantitative talent KPIs that track the number of current full- and part-time researchers, the organization’s recruitment yield (percent of researchers who join the TRO in response to an offer), and staff retention rate. In addition, TROs could include qualitative KPIs, such as level of commitment from senior faculty members and willingness to work together with clinicians. Furthermore, TROs should track the rate of employment termination for researchers and other staff, as well as the reasons for termination. TROs should be prepared to terminate researchers who do not produce results as measured by other relevant performance dimensions.

Creation. Creation addresses the quantitative and qualitative characteristics of a TRO’s research pipeline. Quantitative KPIs should describe the size and diversity of the pipeline and assess, for individual projects, the extent and speed of progress toward established milestones (on an absolute basis and in comparison with similar projects outside the TRO). Qualitative KPIs should attempt to capture the TRO’s contribution to innovation through questions such as: Are clinical trials being funded by TROs for scientific discoveries that were made several years earlier or for diseases that are neglected by the pharmaceutical industry? Is there evidence that the TRO’s research advances our understanding of human disease mechanisms? (13).

The creation dimension also can encompass any tools a TRO develops in the course of research, including assays, databases, biomarkers, and any other product that can be leveraged to aid research across other projects or institutions. Tools are different from other types of TRO outputs because they are not expected to directly affect patient care. Instead, tools facilitate other projects or aid other organizations in developing therapies, diagnostics, or other medical technologies.

One pitfall of the creation dimension is the danger of rewarding organizations solely on the basis of the size of the pipeline. An organization can game the system by keeping projects in the pipeline even if they do not appear to be destined for a successful outcome. It is critical that a TRO monitor whether projects are meeting specific milestones in a timely manner and whether underperforming projects are being terminated appropriately.

Validation. Validation indicates overall impact of a TRO’s output and confirms the quality of its research program. Validation can take many forms, such as publication of articles in peer-reviewed journals. Another form of validation is citations of published articles, which enable a TRO to ascertain its impact on the research community. TROs can combine the insights gained from publications and citations by tracking the h-index or m-index at an organizational level. The h-index is a popular measure of the impact of publications on the broader scientific community and is based on the number of a scientist’s cited papers and the number of citations these papers have received in other scientists’ publications (14). The m-index is similar to the h-index but is adjusted to avoid a time-related bias, leading to higher scores for investigators with longer tenure and more time to publish (15).

Patents are a third form of validation. Issuance of a patent for a given TRO discovery indicates that after an extensive review, governmental examiners have determined that the discovery is new, nonobvious, and useful. Patent protection enhances the value of a TRO’s output and attracts future commercial investment.

A fourth type of TRO validation is an independent peer review, which avoids cronyism by subjecting a TRO to scrutiny by experts with no affiliation to the TRO, its donors, or its grant-giving organizations. Such reviews should broadly evaluate the viability of a TRO’s mission, process for executing on that mission, and organizational structure. Independent peer reviews should provide TRO boards with actionable suggestions for improvement.

Lastly, the validity of a TRO’s work may be confirmed via follow-on funding and regulatory approvals.

Dissemination. Dissemination reflects the extent to which a TRO shares its methods and findings with the scientific community and the general public. Dissemination of discoveries by a TRO allows other scientists to build on the breakthroughs and tools achieved by the TRO. Beyond publishing of scholarly articles, which is also part of the validation process, a TRO should—to the extent feasible—make publicly available data that support its published research. Such sharing of information allows collective analysis of the data by the broader scientific community (crowdsourcing), which can speed the development of insights into disease mechanisms and treatment strategies (16).

On the other hand, it may be necessary for TROs to keep certain findings confidential until they can obtain patents for drugs, devices, or tools. However, while patent protection is essential in many cases, patents can also inhibit dissemination of key findings to the broader scientific community. Therefore, TROs should be willing to license their patents to academic and commercial researchers on reasonable terms and allow these researchers to access the technology without burdensome restrictions.

Data sharing is particularly suitable for precompetitive arrangements among academic, industry, and government organizations. Such arrangements have been used to generate, for example, shared databases for new biomarkers and improved disease models (17).

TROs should promulgate compelling medical findings to the public because heightened public awareness can be a precursor to media attention and governmental support. TROs that are well known throughout the scientific and donor communities are more likely to attract funding and recruit productive, talented researchers.

Like creation, dissemination KPIs will vary widely from organization to organization. Potential KPIs include, but are not limited to, the quality of a TRO’s Web site; sponsorship of scientific and policy-related conferences; presentations at symposiums; and impact on public policies. One particularly useful KPI is the extent to which a TRO offers training courses in translational research (for example, how to design clinical trials or use tools developed by the TRO).

External uptake. The external uptake dimension measures financial investments in a TRO’s output by independent third parties. The third party can be a for-profit company, a venture philanthropy, or a public-private partnership (PPP). Third-party investment enables a TRO to leverage the dollars it has spent in pursuit of its mission many times over. External uptake is a performance dimension that is especially relevant for translational research. Although some basic research may result in financial investment, uptake is not typically a primary goal. In contrast, for TROs, facilitating external uptake is almost always the ultimate objective.

TROs may lack the expertise and resources needed to secure regulatory approval and commercialize translational breakthroughs on their own. Therefore, a realistic measure of TRO efficacy is a calculation of the number of technologies and treatments that a TRO transfers to or develops jointly with third parties. Technology transfer or joint development enables TROs to advance technologies beyond early-stage proof of concept while allowing them to avoid the high costs associated with sole sponsorship of large-scale clinical trials and regulatory approval.

Uptake KPIs include commercial investment through licensing deals or outright sale of intellectual property and the establishment of companies or PPPs based on the TRO’s intellectual property.

Collaboration. Many characteristics of academia inhibit collaboration: lack of multidisciplinary training, promotion and recognition criteria based around individual contribution, physical separation of scientists, and weak relationships with primary care practices (18). TROs play a major role in breaking down these barriers to facilitate the interdisciplinary collaboration of basic, translational, clinical, and regulatory scientists (from academics and industry) (19). For example, many TROs actively recruit interdisciplinary teams of researchers with diverse training and areas of expertise (20) and promote interaction among these scientists by providing access to shared resources, such as specialized assays and animal disease models. TROs may encourage members to co-author papers with scientists within the TRO at other academic institutions.

One important area for TRO collaboration is in the precompetitive space. TROs may engage in data or facility sharing across organizations in order to establish the scientific foundation for developing clinical therapies for high-risk, high-return disease targets (15, 21). Such precompetitive arrangements may leverage a broad network that comprises scientists from academia, industry, and private research institutions (22).

Common KPIs for collaboration include co-authorships, academic partnerships, data sharing, and provision and use of shared equipment or facilities. In addition, TROs should assess the ancillary impact of their work on partner organizations as part of the collaboration performance dimension—for example, if a paper co-authored by a TRO member helped a partner organization secure grant funding.


TROs can benefit from adopting a disciplined approach to assessing performance along the seven dimensions discussed above. Such an approach reflects the multifaceted nature of the activities essential to the translational mission but respects the individuality of each TRO by allowing for the incorporation of tailored KPIs and metrics goals.

This approach has three main objectives. First, it encourages a TRO’s scientific leaders and other key stakeholders to clearly define its goals along each performance dimension and prioritize these goals in light of total TRO funding. Second, it helps to align organization-wide support and resources with these goals. Third, it serves as a useful tool to facilitate external communication on past achievements and future efforts to nonscientist stakeholders.

We believe that most TROs experience a life cycle in terms of performance assessment. At the start, it is difficult for the governing body of any TRO to establish a definitive system for performance assessment. Without much practical experience, the board members are likely to be unclear about which aspects of translational research to emphasize and how success should be gauged for each aspect. Thus, our framework should be especially helpful to board members in structuring their thinking about possible measures of success.

After a few years of experience, a TRO should reexamine its KPIs and metrics as it refines its goals and clarifies its priorities. Board members should try to agree on specific goals for the TRO on the basis of what they have learned about its effectiveness: its comparative advantage in research, its capabilities in raising funds, and its skill in implementing its scientific programs. The leadership should then define detailed KPIs and metrics in order to assess progress against these goals over the next 3 to 5 years.

During this period, the TRO’s management should collect data and calculate metrics in order to supply the governing board with an annual performance review. However, the board members should not amend its goals each year. Given the long-term, high-risk nature of translational medical research, a TRO needs a period of goal stability that is longer than 1 or 2 years.

After 3 to 5 years of annual reviews, the board should conduct another broad-based assessment of the TRO’s progress—including possible changes to the organization’s goals and implementation strategies. As part of this assessment, the board members should seek assistance from an independent panel of peer reviewers that includes bench scientists, translational researchers, clinicians, and industry experts. Such peer reviews can help a TRO’s leadership define relative strengths and develop specific solutions to address deficiencies.

Guided by revised goals, KPIs, and metrics that flow from this broad assessment, the TRO should then embark on another 3 to 5 years of translational research with annual progress reviews. At the end of each such period, the TRO can again conduct a broad reappraisal with the assistance of an independent peer review. In this manner, a TRO can use our proposed framework to learn from its experience and reenergize its stakeholders on a regular basis.


    References and Notes

    1. Acknowledgments: R. P. is on the governing council of the Harvard NeuroDiscovery Center and is an outside director of Medtronic. Special thanks to D. Levy (7), Harvard Business School, who contributed essential insights through his interviews of leading TROs, reviews of their performance assessment systems, and literature reviews on performance assessment and the drug development process.
    • Citation: R. Pozen, H. Kline, Defining Success for Translational Research Organizations. Sci. Transl. Med. 3, 94cm20 (2011).

    View Abstract

    Navigate This Article