External Validation of a Breath-Based Prediction Model for Malignant Pleural Mesothelioma

Our plain-English summary of External Validation of a Breath-Based Prediction Model for Malignant Pleural Mesothelioma, by E. Janssens, E. Schillebeeckx, K. Zwijsen, et al (Cancers, 2022).

The study

• Authors: E. Janssens, E. Schillebeeckx, K. Zwijsen, J. Raskin, J. Van Cleemput, V.F. Surmont, K. Nackaerts, E. Marcq, J.P. van Meerbeeck, K. Lamote
• Journal: Cancers
• Published: 2022;14(13)
• Cited by: 9 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

During the past decade, volatile organic compounds (VOCs) in exhaled breath have emerged as promising biomarkers for malignant pleural mesothelioma (MPM). However, as these biomarkers lack external validation, no breath test for MPM has been implemented in clinical practice. To address this issue, we performed the first external validation of a VOC-based prediction model for MPM. The external validation cohort was prospectively recruited, consisting of 47 MPM patients and 76 asbestos-exposed (AEx) controls. The predictive performance of the previously developed model was assessed by determining the degree of agreement between the predicted and actual outcome of the participants (patient/control). Additionally, to optimise the performance, the model was updated by refitting it to the validation cohort. External validation revealed a poor performance of the original model as the accuracy was estimated at only 41%, indicating poor generalisability. However, subsequent updating of the model improved the differentiation between MPM patients and AEx controls significantly (73% accuracy, 92% sensitivity, and 92% negative predictive value), substantiating the validity of the original predictors. This updated model will be more generalisable to the target population and exhibits key characteristics of a potential screening test for MPM, which could significantly impact MPM management.

In plain English

This study tested whether a breath-test model, built earlier by the same group from exhaled volatile organic compounds, could distinguish people with malignant pleural mesothelioma (MPM) from asbestos-exposed people without it. From October 2018 to November 2021 the authors recruited 47 MPM patients and 76 asbestos-exposed controls (some with pleural plaques or asbestosis) in Belgium. Applied to this new group, the original model's accuracy was 40.7% (95% CI 32.3 to 49.5), sensitivity 53.2% (39.0 to 67.0) and specificity 32.9% (23.1 to 44.0); in the earlier group, by internal cross-validation, accuracy had been 86.8% (80.8 to 91.5). The authors refitted the model to the new group, keeping 4 of its 11 compounds. Tested by internal cross-validation in the new group, the updated model had 73.2% accuracy (64.9 to 80.4), 91.5% sensitivity (80.8 to 97.2), 61.8% specificity (50.6 to 72.2) and 92.2% negative predictive value (82.2 to 97.4). The authors concluded the updated model was promising but must itself be externally validated, ideally by an independent group. They noted the compounds' chemical identity was unknown and the patient group smaller than recommended.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Multicentre, cross-sectional, case-control study used to externally validate, and then update, a previously developed breath-based (volatile organic compound) prediction model for MPM
• Who was studied: 123 participants in Belgium, recruited October 2018 to November 2021. There were 47 MPM patients, with histologically confirmed disease and no prior treatment, referred through the Antwerp University Hospital. There were 76 asbestos-exposed controls, recruited through the occupational health departments of two companies that used asbestos until 1997 and through an online advertisement. None had taken part in the earlier discovery study (52 MPM patients, 100 controls).
• Exposure: Controls had a known history of asbestos exposure. They were either asymptomatic or had benign asbestos-related disease: in the validation cohort, pleural plaques 6, asbestosis 2, pleuritis 3. Fiber types and doses were not reported. The test studied was exhaled-breath volatile organic compounds measured by multicapillary column/ion mobility spectrometry.
• Main results:
  • In the earlier discovery cohort, the original 11-compound model had an estimated accuracy of 86.8% (95% CI 80.8 to 91.5) and an area under the curve of 92% (95% CI: 86 to 96%). Both figures came from internal leave-one-out cross-validation.
  • Applied to the new validation cohort (47 MPM patients, 76 controls), the original model had these results: accuracy 40.7% (95% CI 32.3 to 49.5), sensitivity 53.2% (39.0 to 67.0), specificity 32.9% (23.1 to 44.0), positive predictive value 32.9% (23.1 to 44.0) and negative predictive value 53.2% (39.0 to 67.0).
  • The model was refitted to the validation cohort and retained 4 compounds (P9, P88, P101, P122). Estimated by internal cross-validation in that cohort, the updated model had these results: accuracy 73.2% (95% CI 64.9 to 80.4), sensitivity 91.5% (80.8 to 97.2), specificity 61.8% (50.6 to 72.2), positive predictive value 59.7% (48.2 to 70.6), negative predictive value 92.2% (82.2 to 97.4), and area under the curve 75% (95% CI: 66 to 83%).
  • MPM patients in the validation cohort were older than those in the discovery cohort (69.99 vs. 66.43 years, p = 0.018). Of the 11 compounds, only P15 showed a weak correlation with age (Kendall's tau = 0.172, p = 0.012).
  • Test samples came from one healthy subject: 3 taken with and 3 without the viral filter added because of COVID-19. No significant effect of the filter on the compounds was observed.
• The authors' conclusion: The authors concluded that the original model performed poorly on an independent group, showing a problem of reproducibility and generalisability in breath research. Updating the model led to promising outcomes, but the updated model must in turn be validated externally, ideally by an independent research group.
• Limitations the authors state:
  • The chemical identity of the breath compounds is unknown, because the method only allows pseudo-identification of peaks.
  • The sample size was smaller than the rule-of-thumb minimum of 100 events in a validation cohort; only 47 MPM patients were included.
  • An additional viral filter was used because of COVID-19. The authors say this protocol change may have contributed to the differences from the discovery results.
  • Different analysis experts selected the compound peaks manually in the discovery and validation studies, which increased the risk of interobserver variability.
  • The updated model itself still requires external validation.

Topics

Medical terms in this study, explained in our Lexicon: Pleural Mesothelioma, Biomarker.

Related studies

• Exhaled breath to screen for malignant pleural mesothelioma: a validation study — K. Lamote, M. Vynck, et al. (2017)
• External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data — A. Winter, D.R. Aberle, et al. (2019)
• Development and validation of a lung cancer risk prediction model for African-Americans — C.J. Etzel, S. Kachroo, et al. (2008)
• External validation of prognostic indices for overall survival of malignant pleural mesothelioma — Y. Kataoka, Y. Yamamoto, et al. (2017)
• Breath analysis by gas chromatography-mass spectrometry and electronic nose to screen for pleural mesothelioma: a cross-sectional case-control study — K. Lamote, P. Brinkman, et al. (2017)
• Detection of malignant pleural mesothelioma in exhaled breath by multicapillary column/ion mobility spectrometry (MCC/IMS) — K. Lamote, M. Vynck, et al. (2016)

Browse more research in Cancers, or return to medical research.