External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data

Our plain-English summary of External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data, by A. Winter, D.R. Aberle, W. Hsu (Thorax, 2019).

The study

• Authors: A. Winter, D.R. Aberle, W. Hsu
• Journal: Thorax
• Published: 2019;74(6):551-563
• Cited by: 30 articles, as counted by NIH iCite
• Read it: PubMed · Publisher

Abstract

Introduction: We performed an external validation of the Brock model using the National Lung Screening Trial (NLST) data set, following strict guidelines set forth by the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement. We report how external validation results can be interpreted and highlight the role of recalibration and model updating.

Materials And Methods: We assessed model discrimination and calibration using the NLST data set. Adhering to the inclusion/exclusion criteria reported by McWilliams et al, we identified 7879 non-calcified nodules discovered at the baseline low-dose CT screen with 2 years of follow-up. We characterised differences between Pan-Canadian Early Detection of Lung Cancer Study and NLST cohorts. We calculated the slope on the prognostic index and the intercept coefficient by fitting the original Brock model to NLST. We also assessed the impact of model recalibration and the addition of new covariates such as body mass index, smoking status, pack-years and asbestos.

Results: While the area under the curve (AUC) of the model was good, 0.905 (95% CI 0.882 to 0.928), a histogram plot showed that the model poorly differentiated between benign and malignant cases. The calibration plot showed that the model overestimated the probability of cancer. In recalibrating the model, the coefficients for emphysema, spiculation and nodule count were updated. The updated model had an improved calibration and achieved an optimism-corrected AUC of 0.912 (95% CI 0.891 to 0.932). Only pack-year history was found to be significant (p<0.01) among the new covariates evaluated.

Conclusion: While the Brock model achieved a high AUC when validated on the NLST data set, the model benefited from updating and recalibration. Nevertheless, covariates used in the model appear to be insufficient to adequately discriminate malignant cases.

© Author(s) (or their employer(s)) 2019. No commercial re-use. See rights and permissions. Published by BMJ.

In plain English

The authors tested how well an existing tool, the Brock model, estimates the chance that a lung nodule found on a first low-dose CT screening scan is cancer. They applied it to US National Lung Screening Trial data: 7879 non-calcified nodules in 5018 participants followed for 2 years. Of these nodules, 2.8% were malignant. The model's area under the curve was 0.905 (95% CI 0.882 to 0.928). However, it overestimated the probability of cancer. At the authors' chosen cut-off of a 6.5% predicted probability, its positive predictive value was low: 18.31%, with a reported interval of 16.92% to 19.7%. The authors then updated the model and tested four extra factors, including occupational exposure to asbestos for 1 or more years. Only pack-years of smoking improved the model, and asbestos exposure was not added. The updated model was better calibrated, with an optimism-corrected AUC of 0.912 (95% CI 0.891 to 0.932). The authors concluded the model benefited from recalibration, but its factors appear insufficient to adequately discriminate malignant cases. They noted that which nodule was cancer had to be inferred, and that 30% of screenings were excluded for missing data.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: External validation, recalibration and extension of an existing lung-nodule cancer prediction model (the Brock model), using data from an earlier randomised screening trial
• Who was studied: 7879 non-calcified nodules found at the baseline low-dose CT screen in National Lung Screening Trial participants (5018 participants: 194 with cancer, 4824 without), with 2 years of follow-up. Screens were done from August 2002 to September 2007 in the USA.
• Exposure: Not an asbestos exposure study. The model used patient and nodule factors. Four extra candidate factors were tested: body mass index, smoking status at randomisation, pack-years, and occupational exposure to asbestos for 1 or more years.
• Main results:
  • The original Brock model applied to the NLST nodules had an AUC of 0.905 (95% CI 0.882 to 0.928)
  • Only 2.8% of the nodules were malignant
  • At a cut-off of 6.5%, the original model had a negative predictive value of 99.36% (99.2 to 99.53) and a positive predictive value of 18.31% (16.92 to 19.7)
  • The original model overestimated the probability of lung cancer, most at higher levels of risk
  • The distributions of all model covariates differed significantly between the original Canadian (PanCan) data and the NLST data
  • Of the new covariates tested (including asbestos exposure), only pack-year history was added to the updated model
  • The best updated model (method 6) had a c-statistic of 0.914 (95% CI 0.892 to 0.936), the lowest AIC (1288) and the lowest Brier score (0.021)
  • Compared with the original model, the updated model's calibration improved but its discrimination did not change
  • After internal validation with 500 bootstrap replications, the updated model's optimism-corrected performance was 0.912 (95% CI 0.891 to 0.932)
  • When predicting cancer over 3 and 4 years of follow-up, discrimination dropped as follow-up lengthened
• The authors' conclusion: The authors concluded that the Brock model achieved a high AUC on NLST and benefited from updating and recalibration, but that its covariates appear insufficient to adequately discriminate malignant cases. They stated that further validation is required before any revised model is used.
• Limitations the authors state:
  • NLST data were at the participant level, so whether each nodule was cancer had to be inferred from locations. This may have caused misclassification and selection bias.
  • 30% of LDCT screenings were excluded because of missing data, which may have contributed to selection bias
  • Logistic regression does not account for censoring, so the authors advised care in estimating cancer probability beyond 2 years
  • The original model's assumptions about its covariates, such as linearity, were not checked

Topics

The National Library of Medicine indexes this study under subjects our site covers: Lung cancer.

Medical terms in this study, explained in our Lexicon: Prognosis, Lung Cancer, Emphysema.

Related studies

• Probability of cancer in lung nodules using sequential volumetric screening up to 12 months: the UKLS trial — M.W. Marcus, S.W. Duffy, et al. (2019)
• Solitary pulmonary nodule malignancy predictive models applicable to routine clinical practice: a systematic review — M. Senent-Valero, J. Librero, et al. (2021)
• Predictive accuracy of the Liverpool Lung Project risk model for stratifying patients for computed tomography screening for lung cancer: a case-control and cohort validation study — O.Y. Raji, S.W. Duffy, et al. (2012)
• Solitary pulmonary nodule in a patient without asbestos exposure — J.F. Turner, C. Gronbeck, et al. (1992)
• Variations in lung cancer risk among smokers — P.B. Bach, M.W. Kattan, et al. (2003)
• A probability model for evaluating building contamination from an environmental event — R.C. Spicer, H.J. Gangloff (2000)

Browse more research in Thorax, or return to medical research.