CommentaryEmergency Situations

Use of Forensic Methods Under Exigent Circumstances Without Full Validation

See allHide authors and affiliations

Science Translational Medicine  25 Nov 2009:
Vol. 1, Issue 8, pp. 8cm7
DOI: 10.1126/scitranslmed.3000372

Abstract

Forensic science can be instrumental in providing investigative leads or clues to help identify the perpetrators, and those who are innocent, of a chemical, biological, radiological, or nuclear terrorist attack, as well as identify leads during infectious outbreaks or other public health threats. Because of a need to react quickly in exigent circumstances during which the threat of continued attack persists, methods may be used that are not fully validated. A preliminary validation should be performed to evaluate the acquisition of limited test data to ensure that the interpretation of results remains within the limitations of known performance of the method. If results from a preliminarily validated method are used beyond developing an investigative lead, further validation should be considered to support its reliability for adjudication purposes.

INTRODUCTION

When life is threatened, forensic scientists often need to gather, analyze, and interpret data rapidly (Fig. 1). In the event of a chemical, biological, radiological, or nuclear (CBRN) terrorist attack, a rapid and effective forensic response will be needed. Forensic science can play an invaluable role in providing investigative leads or clues to help identify those who committed the attack, as well as those who are innocent, and provide supportive evidence for legal proceedings or decisions regarding an effective response. Identification and interdiction of those responsible can help prevent subsequent attacks and allay public fears (1). Because of unique circumstances, including the use of novel biological agents, and time constraints imposed by the need to respond to terrorist attacks, forensic science methods might need to be used before they have been fully validated to the standards recommended by the specific discipline(s) and without having standard operating protocols (28). As with many advances in medicine, such as the computed tomography scan (9), these methods may arise from basic science studies and have yet to be fully vetted for creative applications to the situation at hand. When the safety and security of people, agriculture, and resources are at risk, it would be unacceptable to delay forensic analyses until a method is fully validated. Although less-than-fully validated methods might be employed, they still need some degree of evaluation and might require further evaluation and validation if the results are to be used to support legal prosecution directly or be used to effect policy and other governmental actions (10). Full validation is not easily prescribed before an actual event, because methods are different and requirements can vary. For microbial forensics, a list of validation criteria has been generated for assay developers to consider; these provide starting points from which the developer can select, demonstrate why those selected were applied, and demonstrate why others were not selected (2). Although this topic is not within the scope of this paper, it is incumbent that suitable validation criteria be applied. Other manuscripts on validation criteria and quality assurance provide more in-depth discussion (2, 11). This document provides the rationale for and guidance on the use of forensic methods in exigent circumstances without a comprehensive validation, which may also be extended to other fields in medicine.

Fig. 1. Evidence collection.

Federal Bureau of Investigation (FBI) Hazardous Materials Response Team members disassemble a weapons of mass destruction laboratory for evidence collection. (The photograph shows a training exercise in progress.)

Photo courtesy of FBI Laboratory, Hazardous Materials Response Team Unit

PRELIMINARY VALIDATION OF METHODS AND ASSAYS

The use of methods for the purpose of generating investigative leads and intelligence does not obviate the need for evaluation before use. A process known as preliminary validation should be performed. Preliminary validation can be considered as the acquisition of limited test data to evaluate performance characteristics of the method (12) before its use in exigent circumstances. This procedure will also help reduce the risks of overinterpreting results.

The intent of preliminary validation is to evaluate key parameters, operating conditions, and limitations of the methods based on available knowledge or data. Such assessment might be important to appreciate (i) the degree of confidence or uncertainty; (ii) the possibility of false-positive or false-negative results; and (iii) the sensitivity, specificity, accuracy, precision, and other relevant validation criteria for the test in question (2, 3). This evaluation should be done before the examination of evidence to instill impartiality or reduce the possibility of bias and to help determine whether the methods can perform properly within the conditions required (12). The need to reduce technical bias is common to all fields of medicine and science. It is also important to reduce the effect of external factors, such as political actions and public and media attention, that are common in high-profile public health or criminal cases and can influence the direction of an investigation.

Table 1 outlines some elements to be considered when non–fully validated assays are employed under exigent circumstances. The intent is to ensure that reasonable efforts are undertaken to interpret evidential results within the limits of extant knowledge and experience and that the assays are conducted in accordance with good scientific practices.

Table 1

Elements to be considered when non–fully validated assays are employed.

View this table:

ADVISORY EXPERTISE

In order to be useful, an assay used to provide an investigative lead need not have both optimal sensitivity and specificity nor be fully validated to a discipline’s standards. Subsequent validation of that assay might not be required if the results will not be used for attribution or legal purposes or in situations in which additional evidence is available to support the case (10). In some circumstances, the assay could be used to support attribution in a court case but might need to be subjected to more rigorous testing criteria. This additional testing might be required in an evidence-admissibility hearing in court under more stringent criteria that the judge considers, such as the Daubert standards for admissibility of scientific evidence (testability, peer review and publication, general acceptance in relevant scientific community, and known error rate of method) (10, 13). If the results are to be used for attribution purposes, then it is incumbent on the user to carry out additional appropriate validation studies as required.

Subject-matter experts should be consulted or convened, as previously recommended (12, 14). These experts should provide input for the users to consider before applying the method. The primary roles of the consultants should be to (i) promote objectivity in all circumstances, (ii) identify limitations or uncertainties, (iii) provide guidance for avoiding undue consumption of evidence, (iv) assist in determining interpretation criteria (to include quantitative or qualitative significance), and (v) help in defining the pathway forward for expeditious use.

EXAMPLES AND SCENARIO

Both the U.S. legal (15) and public health systems (16) have recognized that different standards to proceed forward are permissible under exigent circumstances.

The good-faith use and adherence to best-practice principles, by a scientist, when employing a method that has undergone only preliminary validation could provide pertinent information for further investigation. For example, information might lead to the identification of the location of a clandestine laboratory. Once this information is provided, other evidence could be gathered, analyzed, and used to demonstrate that the facility was producing illicit materials. The original information, like any “tip,” may not necessarily need to be relied on at the subsequent attribution and prosecution phases.

Parallels can be drawn to preliminary method validations performed in response to public health threats. For example, the early public health response to avian flu before 2000 involved the application of a pathogen identification method that had not been fully validated (17). As the possibility of an avian flu outbreak grew, so did the need to have an appropriate identification assay. At the time, existing assays were not sufficiently sensitive to identify what appeared to be a flu strain isolated from a Hong Kong child. Using a limited set of data and reagents in a repository at the National Institutes of Health, an assay was developed, preliminarily validated, and applied in the case, allowing appropriate follow-up activities to be initiated in a timely fashion (17). Later, additional time and the acquisition of appropriate positive and negative controls permitted more defined and robust assays to be developed to discipline-specific standards.

The more recent 2009 pandemic influenza A (H1N1) (formerly called swine flu; see http://www.cdc.gov) is an example of an exigent event in which Food and Drug Administration (FDA) officials used the Emergency Use Authorization tool [Project BioShield Act of 2004 (Public Law 108–276)] (16) to employ diagnostic assays that were not fully validated nor approved nor cleared by normal FDA standards (see http://www.fda.gov). Health officials needed to respond promptly, and it was incumbent on health officials and the medical community to understand the limitations of the assays to administer proper, albeit limited, advice to patients.

Whereas the 2009 H1N1 outbreak is a good example of the need to apply preliminarily validated assays in an exigent circumstance, it also could serve as a model for an infectious outbreak that might be an intentional event. Such a possibility might provide an impetus and guidance for collaborative efforts between forensic and epidemiological scientists.

The following scenario also illustrates how an assay that has not been fully validated might be used. A large number of people became ill at a hotel on the third day of a conference that they were attending. The affected participants were rushed to the hospital, most with gastrointestinal symptoms, many with neurologic and cardiac symptoms (18, 19). Physical examinations and initial clinical laboratory testing of the patients’ specimens indicated that they might have been poisoned by digoxin, a cardiac glycoside. Oleandrin, another cardiac glycoside present in oleander leaves, was also considered a possibility by one of the physicians. A common activity identified among the patients was the consumption of iced tea on the second day of the conference. Tea remaining in the hotel refrigerator was collected and sent to a laboratory for forensic analysis. Initial visual inspection at the laboratory identified the presence of small leaf-like fragments in the tea, which led the laboratory to analyze the tea for the presence of oleandrin. At that time, the laboratory did not have a validated method for the identification of oleandrin, nor was one found in a literature survey. To further complicate the issue, a commercial reference standard of oleandrin had to be ordered and would not be available for approximately 8 weeks. Therefore, a panel of experts was consulted. The experts were likewise unaware of an oleandrin-specific analytical method; however, one of the experts was familiar with a published method for the analysis of digoxin, a compound structurally similar to oleandrin. The experts agreed that the digoxin detection method, which used high-performance liquid chromatography mass spectrometry (HPLC-MS), might be adapted for the analysis of oleandrin.

The advisory panel could have suggested the following stepwise plan: (i) Before consuming the evidence and performing tests on the limited supply of leaf fragments in the tea, and in lieu of waiting 8 weeks for a commercial oleandrin standard, acquire known oleander leaves from a local nursery for preliminary validation studies. (ii) Acquire a control sample of the same brand of tea used by the hotel to use as a negative control sample during the preliminary validation. (iii) Optimize the parameters of the digoxin HPLC-MS method for the target analyte, oleandrin. (iv) Perform preliminary validation studies on extracts of the authentic oleander leaves and the control tea sample using the modified HPLC-MS method. (v) If the preliminary validation studies give satisfactory performance results, apply the method to a limited sample of the evidence. (vi) If the results from this sample are consistent with the presence of oleandrin, report the findings and their limitations to the authorities. (vii) Proceed with optimization and studies to validate the HPLC-MS method for oleandrin fully, which may be useful to support future adjudication and casework.

These steps were followed, and the preliminary identification of oleandrin in the iced tea enabled investigators to concentrate on suspects who had access to the hotel kitchen. A disgruntled hotel employee was identified as a potential person of interest. A search of the subject’s residence resulted in the discovery of oleander shrubs and literature on oleandrin and other poisons. Subsequently, the laboratory received the commercial reference standard of oleandrin and fully validated the HPLC-MS method for the identification of oleandrin in extracts of oleander leaves and in several beverages, as well as in blood and urine specimens. Although this is only an illustrative scenario, it does emphasize some points of guidance detailed above and in Table 1.

CONCLUSIONS

This document provides rationale and guidance on the use of methods that may not have had full validation in exigent circumstances. It is also a reminder to our colleagues in multiple scientific disciplines that many of their research techniques have the potential to be applied to other fields, including forensic medicine and science, even if they are still in a research and development phase. The document is particularly focused on the need to generate leads in support of CBRN investigations while maintaining high standards of quality work. The use of methods for the purpose of generating investigative leads and intelligence does not obviate the need for evaluation before use. Understanding the context of use of any method is paramount (2). The development and use of fully validated methods (20) that conform to the relevant discipline-specific standards should remain a priority. If results are used for purposes beyond the provision of an investigative lead, such as to provide evidence for prosecution or other actions by decision-makers, further validation might be necessary (21). Whether the information is used alone or in conjunction with other information, the reliability and defensibility of the science and its limitations are critical for court purposes or for a national response.

Acknowledgments

We gratefully acknowledge the input and suggestions of our colleagues who participated in the FBI-hosted Scientific Working Groups and those of the reviewers.

Footnotes

  • Citation: S. E. Schutzer, P. Keim, K. Czerwinski, B. Budowle, Use of forensic methods under exigent circumstances without full validation. Sci. Transl. Med. 1, 8cm7 (2009).

References

View Abstract

Navigate This Article