PerspectivePerformance Metrics

Network Dynamics to Evaluate Performance of an Academic Institution

See allHide authors and affiliations

Science Translational Medicine  13 Oct 2010:
Vol. 2, Issue 53, pp. 53ps49
DOI: 10.1126/scitranslmed.3001580

Abstract

Statistical assessments of performance are common in industry and for individual scientists, but the use of such measures to assess productivity in scientific organizations has lagged behind. The need for defined performance measures has grown as team science has begun to play a larger role in biomedical research, such as in the area of translational medicine. We used a metric, node degree over time, to measure the change in the rate of collaboration over the past five years within an organization, the University of Pennsylvania’s Institute for Translational Medicine and Therapeutics (ITMAT). The number of collaborative papers and grants roughly doubled over the past five years among investigators within but not outside of ITMAT. Also, collaborations within institutions and departments were more frequent than those between them—an actionable area of improvement.

JUDGING JOINT PERFORMANCE

Quantitative and objective measurements of performance are essential for the management of any enterprise. Despite this fact, academic science only recently began applying such metrics to individual scientists. Several approaches such as the H-index and the related G-index (1, 2), as well as a variety of bibliometric methods based on publication and citation counts, have been proposed as methods for the study and evaluation of individual scientific performance (36). These measurements are still not widely used. Key decisions about recruitment, promotion, and retention of faculty are still often made on the basis of subjective rather than objective and data-driven criteria. There is one exception: Grant funding is evaluated quantitatively.

But these measures focus on individuals. Collaborative research has become more popular as many large-scale projects either require different skills sets or are too large for a single laboratory to complete. In recognition of this, academia responded with institutes and centers that coalesce diverse researchers and focus them on common goals. These consortia can exist within a university or between institutions. But how do we evaluate their performance? Even the most diligent reviewer will find it impossible to comprehensively evaluate the scientific output of an institution that comprises dozens or even hundreds of active researchers. Comparing the performances of centers and institutes with similar missions at different universities lags even further behind such comparisons within a single institution. Here, we present methods to measure network dynamics as a first step toward assessing the scientific performance of cross-disciplinary institutes and centers. Translational science—research that moves basic biomedical discoveries toward improvements in clinical medicine—represents just the type of collaborative discipline that requires new metrics for success.

ASSESSING NETWORK DYNAMICS

We applied these methods to the study of the Institute for Translational Medicine and Therapeutics (ITMAT), the academic home of the University of Pennsylvania (Penn) Clinical and Translational Science Award (CTSA) (7). As an early CTSA recipient, we have the opportunity to evaluate how Penn’s organization has changed over a five-year period. The CTSA program of the U.S. National Institutes of Health (NIH) is growing in importance across the United States and now supports 46 research institutions (8). It is important to assess the status of programs at institutes that received the first CTSAs because it takes a few years for the effects of the award to be measurable. Given the huge investment, much is at stake.

In addition to Penn, ITMAT includes the neighboring Children’s Hospital of Philadelphia (CHOP), the Wistar Institute, and the University of the Sciences in Philadelphia. ITMAT was founded in 2004 as the world’s first translational medicine institute and as of January 2009 includes more than 500 active investigators that span these four institutions and dozens of academic departments.

We used ITMAT’s roster as it changed over time to gather data with which to inform ITMAT’s collective performance. To collect these data, we generated Ruby scripts to automatically harvest publication information from the U.S. National Center for Biotechnology Information’s PubMed database. For Penn, we also analyzed grant proposals submitted by its faculty members over the past five years and additional data from NIH RePORT (Research Portfolio Online Reporting Tools) (9). These data were deposited in a MySQL database, and all necessary queries as well as downstream analyses were written in MySQL and Ruby.

Using this approach, we quantified the number of papers and grants by ITMAT investigators over time. Translational medicine is inherently cross-disciplinary. As this so-called Big Science approach extends into more facets of modern biology, successful research is becoming increasingly collaborative. We reasoned that ITMAT’s productivity could be partly measured by how it facilitates collaborations among its members.

We performed a network-based analysis of ITMAT’s performance over time. Papers and grant proposals coauthored by two or more ITMAT investigators were identified and used to construct interaction networks (Fig. 1A). These networks revealed that ITMAT’s overall size and complexity grew substantially since its inception. Not only have the total number of investigators actively collaborating within ITMAT increased since the creation of the consortium (Fig. 1B), the overall cohesiveness of the network grew dramatically as well, as measured by the average number of edges per node (that is, the number of different investigators with which the average ITMAT faculty member has collaborated) (Fig. 1C). This latter result is not surprising: As the size of the network grew, the probability that two investigators (nodes) would interact to copublish papers or cosubmit grants (edges) grew (10). The number of edges per node grew by a factor of four during an interval in which the size of ITMAT’s roster doubled (Fig. 1C), suggesting that ITMAT’s expansion increasingly spurred collaboration. Moreover, the percentage of ITMAT investigators actively collaborating within ITMAT grew every year, with nearly two thirds actively engaged in collaborations in 2009 (Fig. 1D).

Fig. 1. Growth spurt.

ITMAT’s collaborative network has grown dramatically since its inception. Publications and grant proposals coauthored by two or more active ITMAT investigators between 2006 and 2009 were identified and used to generate interaction networks. (A) Every round circle (node) represents a single ITMAT investigator, color-coded by primary departmental affiliation. Lines (edges) that connect the investigators (nodes) represent coauthored works; light blue edges (light blue lines) indicate shared grant proposals, and black edges (black lines) indicate coauthored publications. The width of each edge (line) represents the strength of the interaction; that is, thick edges (lines) indicate that two investigators share more than one collaborative work. (B) The total number of actively collaborating investigators (connected nodes) and the number of interactions (edges) were quantified as a function of time. (C) To assess the interconnectedness of these networks independently of their size, the average number of edges per node was calculated by year. (D) Shown are the total percentages of ITMAT faculty who are actively coauthoring publications (blue bars), grant proposals (red bars), or either publications or grant proposals (green bars).

CREDIT: C. BICKEL/SCIENCE TRANSLATIONAL MEDICINE

But were these results a product of establishing the ITMAT or simply the product of a growing organization? To address this question, we compared collaborations among ITMAT members versus non-ITMAT members at Penn. To be included in the comparison, the collaborating groups were required to have active laboratories. Because there are twice as many non-ITMAT investigators at Penn as ITMAT investigators, we sampled an equivalently sized subset from each group to study connectivity. One hundred fifty investigators were chosen at random from each group, and copublication interaction networks were independently generated 1000 times (Fig. 2A). The average number of nodes and edges for each group was calculated, and these data showed that the size, complexity, and cohesiveness of ITMAT’s interaction network were substantially greater than those of the non-ITMAT group over time; indeed, we calculated the average number of edges per node for each network, and in every case, the ITMAT group had a considerably larger degree of interconnectedness (Fig. 2B). Moreover, within the ITMAT set there was a positive trend over time in the average number of edges per node. We took the slope of a regression line through the data points in Fig. 2B and computed an estimate of node degree over time (NDT), the average edges per node per year. ITMAT investigators showed a positive NDT: On average, each ITMAT investigator added more than 0.05 collaborations per year (Fig. 2C). In contrast, non-ITMAT control groups showed an NDT value indistinguishable from zero. Put simply, ITMAT investigators were increasingly collaborative over time.

Fig. 2. Group dynamics.

The NDT is larger for ITMAT networks than for non-ITMAT institution-matched controls. (A) Copublication interaction networks were generated for investigators with active laboratories at the University of Pennsylvania School of Medicine. Every round circle (node) represents a single ITMAT investigator, and gray lines (edges) that connect the investigators (nodes) represent coauthored publications. Interaction networks from groups of investigators not affiliated with ITMAT (left) were compared with active ITMAT investigators (right). To normalize the difference between the non-ITMAT and ITMAT roster sizes, 150 investigators from each group were chosen at random for network construction. (A) Two examples of networks generated from this analysis. (B) To quantitatively assess the differences in network connectivity between these two groups, 150 investigators from each group were randomly selected 1000 times, and the average number of edges per node from the resulting interaction networks was calculated for every year between 2006 and 2009 (error bars = ∓1 standard deviation from the mean). (C) Shown are plots of the NDTs for both the ITMAT and non-ITMAT interaction networks.

CREDIT: C. BICKEL/SCIENCE TRANSLATIONAL MEDICINE

Three possibilities can explain the difference between the connectivity of ITMAT versus non-ITMAT interaction networks: (i) ITMAT may have self-selected the most productive or most senior investigators at Penn; (ii) ITMAT may have recruited faculty with similar interests and predeveloped collaborative relationships; or (iii) membership in ITMAT may stimulate the development of new collaborations. The first explanation by itself cannot explain these results because the total number of publications per investigator was not dramatically different between the ITMAT and non-ITMAT groups. Sixty-six percent of ITMAT faculty members are tenured versus 55% of non-ITMAT faculty, indicating a modest increase in the average academic rank of ITMAT investigators. But we do not believe this difference is large enough to account for the fivefold increase in collaborative efforts in ITMAT versus institutional controls.

Distinguishing between the second and third possibility is more problematic. Science, especially collaborative research, is a social activity, and funding opportunities such as ITMAT will encourage the enrollment of investigators with common interests. These may include investigators with pre-established collaborations. On the other hand, the positive NDT value may argue that membership in ITMAT may actively promote new collaborations. Regardless of the underlying dynamic, however, the available data strongly suggest that ITMAT investigators are a more highly collaborative group than institutional controls and that this effect is growing over time.

ROOM FOR IMPROVEMENT

In looking over the network structures, we noticed a tendency for investigators to interact more frequently with members of their own departments than they do with scientists outside the department. As expected, every department we examined showed more frequent collaborations among their members than the null hypothesis (Fig. 3). There was considerable divergence in self-association indices among disciplines. The Departments of Neurology, Pediatrics, and Medicine showed little preference for intradepartmental collaborations (self-association index ~ 1.5), whereas the Departments of Psychiatry and Radiation/Oncology showed a six- and 10-fold enrichment for self-association, respectively. Although understandable, and not by itself at odds with translational research, awareness of this property makes it actionable. For example, a funding mechanism that requires the involvement of investigators from more than one department or institution would increase the probability that diverse researchers would interact.

Fig. 3. Birds of a feather.

Collaborative efforts most frequently occur within departments. (A) The cumulative interaction network for every ITMAT investigator since its inception. Primary departmental affiliations for each node are color-coded, revealing substantial clustering of investigators who are in the same department. (B) To quantitatively assess the degree of interconnectivity within departments, the frequency of intradepartmental collaborations was compared with their expected frequency, assuming a null hypothesis that collaborations are independent of departmental boundaries. The expected probability of self-association (blue bars) is calculated by dividing the number of faculty within a department by the total number of ITMAT faculty. The actual probability of self-association (red bars) is calculated by dividing the number of intradepartmental collaborations by the sum of the intra- and interdepartmental collaborations for each department. (C) The ratios between actual and expected levels of intradepartmental collaboration (the self-association indices) are plotted for the nine largest departments within the University of Pennsylvania School of Medicine. The red horizontal line represents the null hypothesis that the actual number of intradepartmental collaborations is equal to the expected number of intradepartmental collaborations. Rad, Radiation.

CREDIT: C. BICKEL/SCIENCE TRANSLATIONAL MEDICINE

In the course of this work, we ran into many technical hurdles that limit this research. One problem is being able to specifically ascribe a paper or grant to a single individual. Although this glitch could be mitigated by a unique identifier [for example, Researcher ID (11) or OpenID (12)], such systems have not yet achieved wide acceptance. Some groups are approaching this problem computationally with promising results (13). In addition, funding agencies such as the Wellcome Trust, NIH, and the Howard Hughes Medical Institute now require investigators to deposit their manuscripts in PubMed Central, which will aid future efforts to unambiguously attribute credit for publications.

There is also a surprising lack of available data required to perform the kind of analysis described here. For example, in NIH RePORT (9) only the names of principal investigators for funded grants are available; key personnel (collaborators) are not. In fact, we were able to find online rosters for only a few of the 46 CTSAs, and none of them were presented on a year-by-year basis. Having this information as well as other types of data (for example, Institutional Review Board applications and research impact measures) would enable comparative performance evaluation between institutes and centers and, eventually, lead to of the identification of operating principles and mechanisms that foster superior research.

Footnotes

  • Citation: M. E. Hughes, J. Peeler, J. B. Hogenesch, Network dynamics to evaluate performance of an academic institution. Sci. Transl. Med. 2, 53ps49 (2010).

References and Notes

  1. Acknowledgments: We would like to thank Garret FitzGerald, Tilo Grosser, Andy Su, Abby Cohen, and John Farrar for helpful discussions and Lorri Schieri for data access. Funding: This work was supported by the University of Pennsylvania’s ITMAT through a grant from National Center for Research Resources (UL1-RR024134, Garret FitzGerald, P.I.). J.B.H. is also supported by the National Heart, Lung, and Blood Institute (1R01HL097800, Hogenesch, P.I.) and the National Institute of Neurological Disorders and Stroke (1R01NS054794, Hogenesch, P.I.). Competing interests: The authors declare that they have no competing interests.
View Abstract

Stay Connected to Science Translational Medicine

Navigate This Article