CommentaryRisk Assessment

Traversing the Valley of Death: A Guide to Assessing Prospects for Translational Success

See allHide authors and affiliations

Science Translational Medicine  09 Dec 2009:
Vol. 1, Issue 10, pp. 10cm9
DOI: 10.1126/scitranslmed.3000265

Abstract

On the basis of our experience in drug and device development, we have identified six broad categories of issues that profoundly affect the probability for successful translation of basic scientific discoveries into effective therapies. Within each category, we propose a series of questions that form the beginnings of a self-guided global risk assessment tool. Although this preliminary tool will require validation and adjustment, we offer it to stimulate a discussion about more rigorous methods for selecting and conducting translational projects.

INTRODUCTION

The application of science to the development of new medical technologies holds promise for eradicating diseases and personalizing health care. Effective translation from bench to bedside, however, requires that our major engines of biomedical research—the academic medical centers funded by the U.S. National Institutes of Health (NIH)—work effectively with industry (1).

Translational research has recently become a focus of concern among policy-makers because of growing awareness of a widening gap between advances in basic science and the practical application of that knowledge (2). One manifestation of this gap is the lack of growth of new drug approvals despite a more than 10-fold increase in inflation-adjusted spending by the pharmaceutical and biotechnology industries over the past 40 years (35). These concerns have rightfully prompted examination of factors contributing to this disappointing performance. Special attention has been paid to the difficulty of crossing the proverbial “valley of death” (6) (Fig. 1): the gulf between finding a promising new agent and demonstrating its safety and efficacy in humans. Venturing into these badlands can be challenging for academic investigators who lack experience in drug development and regulatory processes.

Fig. 1. The “valley of death.”

Many research projects perish as researchers try to cross the rough terrain between basic discovery and useful therapy.

CREDIT: JUPITERIMAGES.COM

Biological science has moved “downstream” as the ongoing molecular biology and genomic revolutions provide abundant insights into potential disease targets that lend themselves to clinical translation. Thus, basic scientists increasingly seek to move their efforts from the discovery phase to the development phase (7). Yet their institutions typically lack systems to support such research and often face difficult decisions regarding which translational projects to support (8).

A major goal of NIH’s Clinical and Translational Science Award program (9, 10) is to develop translational research systems and provide trainees with mentored experiences and didactic programs. As part of this effort, establishing methods for judging the likelihood of translational success before making substantial commitments to a developmental project would be valuable. Unfortunately, scientists lack validated methods for making this judgment, a shortcoming attested to by the high attrition rates for lead compounds and even for drugs that have progressed beyond early-phase human studies (4).

Drawing generalizable conclusions is further hampered by the complex interplay among inventors, academic institutions, venture capitalists, and small and large companies. On the basis of our personal experience in drug and device development, which includes bench discovery, design and execution of early-phase and pivotal–phase III trials, regulatory approval, reimbursement, and the use of comparative effectiveness and systems quality in implementation (1, 11), we have identified six broad categories of issues that profoundly affect the probability of translational success across this spectrum of interests. Within each category, we propose a series of questions that form the beginnings of a self-guided global risk assessment tool (Table 1). This tool will, of course, require future validation and adjustment with empirical data (7).

Table 1. Questions for self-guided assessment of issues affecting the success of translational research projects.
View this table:

When judged by the primary goal of translational research—namely the development of useful technologies to improve human health—the vast majority of translational projects fail (12, 13). However, many projects that fail to produce medically useful results succeed in generating new knowledge that may prove important. Further, the history of medical technology is replete with successful innovations that were initially considered insignificant, and the record of our best scientific journals in accurately judging future benefits to human health is weak at best (12, 13). Many transformative technologies have been dropped by research teams and companies, only to be picked up later with eventual success (14). However, the resources needed are substantial, the costs high, and the failure rates daunting. Given these potential obstacles, we offer the following in hope of stimulating a more rigorous discipline of project selection and conduct, and we welcome dialogue about better ideas for choosing investments in translational projects.

CATEGORY 1: IS IT WORTH THE EFFORT?

Does the new technology’s intended use address a compelling health need?

A firm sense of the limitations of current medical practice, supported with rigorous epidemiological and clinical practice data, is needed to answer this question. If the answer is “no” (or is even equivocal), it is unlikely that other partners will be willing to provide the necessary resources.

Is the scientific rationale strong, and does it suggest a medical benefit when compared with existing therapies?

A strong underlying scientific rationale is necessary but not sufficient for translational success. Criteria for consideration include whether data from multiple disciplines all lead to the same target, as well as whether the assays used to interrogate the target and develop the technology have incorporated biologically and medically plausible conditions. If multiple laboratories have reached similar conclusions, confidence grows. Another critical step is to evaluate the potential for off-target effects based on known mechanisms. Although this cannot be definitively assessed at a project’s inception, it should be considered at an early stage, because it may indicate the need to redesign the agent to achieve greater specificity.

The complexity of making a “rational” choice regarding specificity is perhaps best illustrated by the astounding success of aspirin, which by conventional criteria of specificity would be considered enormously risky. Aspirin not only affects a fundamental process common to multiple homeostatic systems (arachidonic acid metabolism), but does so by acetylating many different proteins in order to acetylate one crucial serine residue in the two isoforms of cyclooxygenase (prostaglandin H-2 synthase), which are involved in mediating platelet aggregation, pain, fever, and inflammation (15). Gauging the relevance of therapeutic targets must therefore accommodate a degree of serendipity, which itself can be systematized (16), as well as individual investigators’ insights.

CATEGORY 2: IS THERE AN ADEQUATE POTENTIAL MARKET?

Does the size and type of market indicate a high likelihood of economic viability?

A strong possibility of commercial success will dramatically increase the chances of obtaining funding for a project. In medical technology development as elsewhere, market size and the customer’s ability to pay are key determinants of the ability to raise capital. Oncology is presently a favored area of development because of increasing disease prevalence and an extraordinary willingness among insurers and patients to pay for new therapies. However, the favorable market exclusivity provisions of the Orphan Drug Act of 1983 provide powerful economic incentives to develop agents that address the health needs of patients with relatively rare diseases. The wise institution will help its investigators understand how estimates are made by the venture capital community, thereby sidestepping mistakes in estimating market potential (in both directions) and avoiding delay. Appreciating the likely need for multiple rounds of funding is particularly important.

Researchers seeking to develop treatments for diseases that disproportionately affect patients who are economically disadvantaged face particular difficulties in securing funding. However, substantial progress is being made through the efforts of industry, creative consortia of industry and academic institutions in the developed world (17, 18), philanthropic support (such as that from the Gates and Clinton foundations) (19, 20), provision of the equivalent of orphan drug status for promising treatments (21), and NIH programs (22, 23). Thus, even if a projected market is not large or wealthy, it may still prove commercially viable.

Although we seek to hasten the day when the sole determinants of therapeutic development will be scientific opportunity and likelihood of improving human health, until that day arrives, commercial considerations will continue to affect the success of translational projects, and investigators and institutions alike must understand how markets influence translational research decision-making.

Is the intellectual property protection solid?

Adequate protection for intellectual property is inextricably tied to the potential for commercial success and thus is a key consideration for the private sector when deciding whether to invest in a project. Some academic investigators strongly disapprove of patenting inventions, viewing it as antithetical to the free exchange of knowledge. In fact, the U.S. patent system, conceived by the country’s founders and enshrined in the Constitution, was designed to advance the public good by encouraging the free flow of technological knowledge. This encouragement takes the form of a period of commercial exclusivity offered in exchange for public disclosure of a complete description of the invention (24). The founders considered a trade secret that is never disclosed the real enemy of technological advance, because it made it impossible to use the information to make new discoveries and inventions. Moreover, the period of commercial exclusivity was seen as a necessary precondition for raising the capital needed to convert inventions into products, because there would be no incentive to invest in expensive development programs if a competitor could wait for a product to show its commercial success and then offer it at a lower price (made possible by not having to recoup the development costs). Although there are serious questions about whether current U.S. patent law optimizes public benefit (25, 26), it is clear that failing to protect intellectual property makes it more difficult or impossible to translate important new discoveries and inventions into products that improve human health (27, 28).

Is the technology likely to be cost-effective?

Demonstrating the medical value of an intervention independent of its cost will not of itself ensure success, particularly in the face of increasing efforts at medical cost containment. Investigators must therefore build into their development plans the capacity to assess the cost-effectiveness of their intervention. Such analyses are expressed in terms of the cost in dollars required to save a year of life [life-year (LY)] or to produce an extra year’s equivalent of higher-quality life [quality-adjusted life-year (QALY)]. Many inventors do not realize that almost all effective medical interventions increase lifetime costs; typically, the issue is whether the incremental health benefit is worth the incremental cost (again, aspirin is the exception that proves the rule: It saves billions of dollars each year in direct health care costs and adds billions more through increased worker productivity beyond its cost).

For example, if it costs the same amount of money to save the life of a child or an elderly person, the cost per LY will be less for the child because the cost will be divided by a larger number of years of expected remaining life. The LY principle can be refined by adjusting years of life based on one or more measures of the quality of the life saved, or even by evaluating interventions that improve health but do not confer survival advantage according to LY equivalents. The cost-effectiveness of renal dialysis (~$50,000 to $75,000/LY) is often taken as the standard for cost-effectiveness, because dialysis is covered under Medicare regardless of a patient’s age (29).

Cost-effectiveness it is likely to become an increasingly important factor in the adoption of new interventions even though the methods for quantifying it remain contentious (30). In Europe, the criteria used to decide on approval for payment are as rigorous as for regulatory approval; given the focus on costs in recent U.S. health care reform efforts, the United States is poised to move in a similar direction. Thus, those who fund R&D are likely to increasingly scrutinize whether payors will pay for a new technology, in addition to whether it will achieve regulatory approval.

CATEGORY 3: WHAT CAN BE INFERRED FROM HUMAN AND ANIMAL DATA ABOUT LIKELY SAFETY AND EFFICACY?

Is there a human genetic disorder that affects the therapeutic target? If so, does the phenotype support the efficacy and/or safety of the agent?

Detailed and rigorous analysis of the phenotype of patients with genetic disorders in which the proposed target protein is either deficient or defective can provide insights into the likely effectiveness of a technology designed to inhibit the target (31). Thus, one would expect patients who lack the protein being considered as a therapeutic target to be protected from the disease that the inhibitor is being developed to treat. There always are caveats to consider when interpreting such genetic data, but failing to identify the expected impact of the defect on the disease process should give one serious pause about proceeding (32).

Human genetic disorders can also provide insights into the likely toxicity of an agent that inhibits a novel target. Thus, in the case of disorders in which the proposed target protein is completely lacking, careful analysis of the patients will provide information on the effect of complete inhibition of the target by the new agent (11). These data are also limited by the possibility of coincidental interacting genetic alterations that are not easily discovered. In addition, the nature of the illness, or coincidental medications used to treat the illness, may affect the agent’s ultimate toxicity in the target population unpredictably. Nonetheless, if observations on patients with the genetic disorder are reassuring with regard to toxicity, this surely augurs well, and vice versa.

Are the animal models used to assess efficacy and safety convincingly representative of the human disease?

The integrity of the animal model as a plausible simulacrum of the human disorder is a key factor in assessing the likelihood of success. Some human disorders are easily simulated in animal models, but many are not. Transgenic animals have expanded our ability to model human disorders but nonetheless have many limitations, including differences between human and mouse physiology and differences between mouse background strains. When, despite one’s best efforts, the animal model diverges substantially from the human condition, considerable uncertainty will remain regarding the likelihood of the project’s success.

CATEGORY 4: CAN THE AGENT BE DELIVERED TO ITS TARGET AT AN ADEQUATE CONCENTRATION?

Are the pharmacokinetics and pharmacodynamics acceptable for the intended use, based on direct assessment of the effects on the target molecule or meaningful functional assays?

Delivering an agent to its target in sufficient concentration and for an appropriate duration to achieve the desired effect is an essential part of drug development. Even the most promising agent can falter if problems of formulation, absorption, distribution, metabolism, and/or elimination interfere with achieving therapeutic levels at the target. Because a systematic analysis of formulation is rarely a component of academic investigation, substantial uncertainty during early developmental phases might exist. Unfortunately, academic clinical pharmacology has been in eclipse for some time, and relatively few academic institutions have individuals with expertise in pharmacokinetics and pharmacodynamics.

The most desirable pharmacodynamics assays are those that test the agent’s effect on the target molecule itself, coupled with an analysis of the functional impact.

CATEGORY 5: IS THERE AN INDUSTRY PARTNER THAT CAN DEVELOP THE AGENT EFFECTIVELY AND EFFICIENTLY?

Is there an industry partner willing to make the development program a high priority?

Undertaking preclinical studies to support an investigational new drug or device exemption application is a daunting task requiring multiple teams of investigators with widely varied expertise. Successful translation thus requires sustained effort over a period of years. It is vital, therefore, that an industry partner have the resources and commitment to see the process through, especially because historical evidence indicates that serious problems are likely to arise during development. Biotechnology and pharmaceutical companies are under intense pressure to control costs and demonstrate short-term profitability, both of which can jeopardize long-term development programs.

Smaller companies with fewer products under development may be prepared to make a stronger commitment; however, larger companies are generally better capitalized, more experienced, and more likely to have the requisite expertise. One method for encouraging a company to maintain its focus is to include expected resource commitments and developmental milestones in the licensing agreement. If the company fails to meet these benchmarks, the technology reverts to the licensing institution, which can then seek another partner.

Will the industry partner ensure that the preclinical and clinical development groups exchange ideas throughout the development process?

Understanding the interconnectedness of all steps in the development process is central to success. Thus, the intended use of a therapy and the design of future clinical studies can profoundly affect the optimal preclinical development program and even the manufacturing process. The animal models selected must adhere as closely as possible to the eventual target patient population, and the proposed dose for human patients may make one form of manufacturing preferable. Even during the preclinical phase, there may be changes in the standard of care for the intended clinical condition, radically affecting the development plan. Thus, continual communication must be fostered between the preclinical and clinical groups. In small companies, this is usually easier to achieve because there are fewer people, but larger companies may offer compensating advantages in expertise and experience.

Is there an industry partner that will refrain from excessive secrecy?

Nowhere is the difference in perspectives between corporate and academic cultures more evident than with regard to secrecy. Nondisclosure agreements and trade secrets are typically viewed by industry as important to sustaining a competitive advantage. In sharp contrast, rapid publication and the sharing of resources and knowledge are prized values in academia. Striking an optimal balance between these perspectives is crucial for both the scientific and commercial success of a translational project. Some foreseeable issues can be addressed in the initial licensing agreement. For example, one author (B.S.C.) inserted a clause requiring that the monoclonal antibodies being licensed be supplied to academic investigators free of charge. This not only kept faith with academic principles but also provided crucial information for subsequent drug development, because other investigators confirmed previous observations and went on to make important discoveries with the antibodies that had implications for the drug development plan.

Can the technology be manufactured easily and at a reasonable price?

Academic investigators often lack knowledge of the scientific complexities and regulatory requirements of industrial-scale production. Some compounds and devices are much more expensive to manufacture than others, affecting the project’s economic viability. These issues need to be considered even at the beginning of a project, because the size of the lots of the agent for the preclinical and initial human studies must balance the benefit of having a large lot (so as to avoid the extra costs incurred by having to perform multiple quality-control assessments on small lots) versus the potential waste involved in producing more material than needed if the project is halted prematurely.

CATEGORY 6: CAN A PIVOTAL TRIAL BE DESIGNED AND COMPLETED?

Can a study be designed with a medically meaningful endpoint?

Selecting a medically meaningful, quantifiable endpoint that can support regulatory approval and gain acceptance by payors is one of the most important and difficult challenges in translational research (33). Because the appropriate roles for biomarkers and putative surrogate endpoints in supporting regulatory approval remain contentious (34), the ability to assess a new therapy directly by its effect on a medically meaningful endpoint confers a clear advantage.

Can a study be designed to reflect clinical equipoise and be attractive to both participants and their clinicians?

When there is broad consensus among the academic community that a trial’s design is based on equipoise, there will be an equally broad consensus about the ethical basis of the study. On the contrary, when the clinical practice or academic communities are sharply divided in their views regarding the efficacy of different therapies, it will probably be difficult to design a study that will be free from criticism. Because clinical trials require the enthusiastic support of both patients and their physicians, protocols must balance the risks and benefits of the therapy within the comfort zone of the clinical investigators.

Can a study be designed with sufficient statistical power to detect the endpoint?

Biostatistical issues must be addressed at the earliest stage of trial preparation because they will determine many key design elements. Practical issues, such as the number of potential participants with the condition of interest and their geographic distribution, should be weighed against optimal biostatistical models. Similarly, although a large sample size is almost always desirable, costs and time needed to complete the study are also important considerations.

CONCLUSIONS

Translating scientific discoveries into improved therapies is one of the most exciting, satisfying, and complex challenges academic investigators can take on. The inherent difficulties and uncertainties, however, make the odds of success quite small. We believe that many key issues in translational research projects (Table 1) are amenable to upfront analysis. Although the value of the questions proposed will need to be assessed and independently validated with empirical data, we strongly suspect that researchers who can answer “yes” to most of them will be more likely to make it through the valley of death, even if it does not quite feel like a Garden of Eden.

Footnotes

  • Citation: B. S. Coller, R. M. Califf, Traversing the valley of death: A guide to assessing prospects for translational success. Sci. Transl. Med. 1, 10cm9 (2009).

References

  1. We thank S. Rivera and J. McCall for administrative and editorial assistance, and D. Rosenblum of the National Center for Research Resources of NIH, whose probing questions stimulated us to undertake this project. This manuscript was funded in part by grants 1UL1RR024143-01 (B.S.C.) and 1UL1RR024153-01 (R.M.C.) from the National Center for Research Resources (NCRR), a component of NIH, and the NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. Competing interests. B. Coller is on the scientific advisory board and is a consultant to Accumetrics. He is listed as an inventor on patents related to abciximab (assignee, Research Foundation of SUNY) and the precursor of the Accumetrics VerifyNow assay (assignee, Mount Sinai School of Medicine).
View Abstract

Navigate This Article