ReportsCROWDSOURCING

Systematic Analysis of Challenge-Driven Improvements in Molecular Prognostic Models for Breast Cancer

See allHide authors and affiliations

Science Translational Medicine  17 Apr 2013:
Vol. 5, Issue 181, pp. 181re1
DOI: 10.1126/scitranslmed.3006112
  • Fig. 1 Timeline and phases of BCC.

    At initiation (phase 1), a subset of the METABRIC data set was provided along with orientation on how to use the Synapse platform for accessing data and submitting models and source code. Phase 2 provided a new randomization of samples, to eliminate biases in the distribution of clinical variables across training and test data, and renormalization of METABRIC mRNA expression and DNA copy number data, to reduce batch effects and harmonize data with the OsloVal data used in phase 3. During phase 2, there was a live “pre–15 October 2012” leaderboard that provided real-time scores for each submission against the held-out test set of 500 samples. At the conclusion of phase 2 on 15 October 2012, all models in the leaderboard were tested against the remaining held-out 481 samples. In the final validation round (phase 3), participants were invited to retrain up to five models on the entire METABRIC data set. Each model was then assigned a final CI score and consequently a rank based on the model’s performance against the independent OsloVal test set.

  • Fig. 2 BCC through time.

    (A) During phase 2, the highest-scoring model scores were recorded for each date until the leaderboard was closed. Each colored segment represents a top-scoring team at any given point for the period extending from late September 2012 until the final deadline of phase 2 (15 October 2012). The plot records only the times when there was an increase in the best score, whereas the teams that achieved this score are labeled with different colors. The sequence of colors highlights an important aspect of the real-time feedback, where teams were encouraged to improve their models after being bested on the leaderboard by another team. Inset, the same plot with a y-axis scale ranging from of 0.5 to 1.0 maximum CI. (B) Probability density function plots of model scores posted on the live pre–15 October leaderboard evaluated against (i) the first test set of 500 samples (blue), (ii) the second test set of 481 samples (yellow), and (iii) the OsloVal data set (red). The null hypothesis probability density, which corresponds to random predictions evaluated against the OsloVal data set, is shown in purple. (C) Scatter plot of pre–15 October 2012 model performance versus 15 October 2012 performance. Colors represent quantiles, meaning that the ordered data are divided into four equal groups numbered consecutively from the bottom-scoring models (1) to the top-scoring models (4) for pre–15 October model performance. (D) Scatter plot of pre–15 October 2012 model performance versus final OsloVal performance. Colors represent quantiles of pre–15 October model performance. Asterisk represents the highest-scoring submitted model.

  • Fig. 3 Rank stability of final models.

    The OsloVal test data were randomly subsampled 100 times using 80% of the samples. Model rank was recalculated at each iteration. Models are ordered by their final posted leaderboard score (P values for the top three models, which were submitted by the same team, versus the fourth place model were as follows = 5.1 × 10−28, 1.8 × 10−22, and 1.7 × 10−20 by Wilcoxon rank-sum tests). With these box plots, the middle horizontal line represents the median, the upper whisker extends to the highest value within a 1.5× distance between the first and third quantiles, and the lower whisker extends to the lowest value within a 1.5× distance. Data beyond the ends of the whiskers are outliers plotted as points.

  • Fig. 4 Individual and community scores for METABRIC and OsloVal.

    (A) Individual model scores are ordered by their rank on the pre–15 October 2012 METABRIC leaderboard (red line). For each model rank (displayed on the x axis), the blue line plots the aggregate model score based on combining all models less than or equal to the given rank. (B) Individual and aggregate model scores based on evaluation in the OsloVal data set. (C) Individual model scores (that is, community = 1) from the pre–15 October 2012 METABRIC leaderboard (http://leaderboards.bcc.sagebase.org/pre_oct15/index.html) are plotted alongside the community aggregate scores obtained when 5, 10, 20, and 50 randomly chosen models were considered. (D) Individual model scores (that is, community = 1) from the final OsloVal leaderboard (http://leaderboards.bcc.sagebase.org/final/index.html) are plotted alongside the community aggregate scores obtained when 5, 10, 20, and 50 randomly chosen predictions were considered. The colors correspond to community size: red = 1, yellow = 5, green = 10, blue = 20, purple = 50.

  • Fig. 5 Model performance and clinical characteristics.

    (A) Percentage of CI variance explained by each clinical variable. (B) CIs were calculated for OsloVal models according to subsets of patients by histological grade. (C) CIs were calculated for OsloVal models according to subsets of patients by LN status. (D) CIs were calculated for OsloVal models according to subsets of patients by follow-up time (OS). (E) CIs were calculated for OsloVal models according to subsets of patients by age, ER status, and HER2 status. Patients were divided into subsets according to each of the above clinical characteristics. Individual model predictions were generated for patients belonging to each subset, and the CI was calculated by comparison with the actual survival for each patient.

  • Table 1 Clinical characteristics of METABRIC and OsloVal data sets.

    NA, not available.

    CategoriesMETABRICOsloVal
    Cohort size1981184
    Age, years (%)
      ≤5021.433.1
      50–6022.518.5
      ≥6056.148.4
    Tumor size, cm (%)
      ≤243.338.0
      2–548.242.4
      ≥57.57.1
      NA1.012.5
    Node status (%)
      Node negative52.349.5
      1–3 nodes31.421.2
      4–9 nodes11.49.8
      ≥10 nodes4.68.2
      NA0.311.3
    ER status (%)
      ER+76.360.9
      ER23.739.1
    PR status (%)
      PR+52.721.2
      PR47.378.8
    HER2 copy status (%)
      HER2 amplification22.113.6
      HER2 neutral72.686.4
      HER2 loss5.00.0
      NA0.30.0
    Tumor grade (%)
      18.66.5
      239.137.0
      348.130.4
      NA4.226.1

Additional Files

  • Supplementary Material for:

    Systematic Analysis of Challenge-Driven Improvements in Molecular Prognostic Models for Breast Cancer

    Adam A. Margolin,* Erhan Bilal, Erich Huang, Thea C. Norman, Lars Ottestad, Brigham H. Mecham, Ben Sauerwine, Michael R. Kellen, Lara M. Mangravite, Matthew D. Furia, Hans Kristian Moen Vollan, Oscar M. Rueda, Justin Guinney, Nicole A. Deflaux, Bruce Hoff, Xavier Schildwachter, Hege G. Russnes, Daehoon Park, Veronica O. Vang, Tyler Pirtle, Lamia Youseff, Craig Citro, Christina Curtis, Vessela N. Kristensen, Joseph Hellerstein, Stephen H. Friend,* Gustavo Stolovitzky, Samuel Aparicio, Carlos Caldas,
    Anne-Lise Børresen-Dale

    *Corresponding author. E-mail: margolin@sagebase.org (A.A.M.); friend@sagebase.org (S.H.F.)

    Published 17 April 2013, Sci. Transl. Med. 5, 181re1 (2013)
    DOI: 10.1126/scitranslmed.3006112

    This PDF file includes:

    • Materials and Methods
    • Table S1. Univariate and multivariate Cox regression statistics for the clinical covariates in the METABRIC data set.
    • Table S2. Univariate and multivariate Cox regression statistics for the clinical covariates in the OsloVal data set.
    • Table S3. The Breast Cancer Challenge Consortium: Challenge participants who submitted a model to phase 3 of the BCC.

    [Download PDF]

     

Navigate This Article