Convolutional Neural Networks for Segmentation of Malignant Pleural Mesothelioma: Analysis of Probability Map Thresholds (CALGB 30901, Alliance)

Our plain-English summary of Convolutional Neural Networks for Segmentation of Malignant Pleural Mesothelioma: Analysis of Probability Map Thresholds (CALGB 30901, Alliance), by M. Shenouda, E. Gudmundsson, F. Li, et al (ArXiv, 2023).

The study

• Authors: M. Shenouda, E. Gudmundsson, F. Li, C.M. Straus, H.L. Kindler, A.Z. Dudek, T. Stinchcombe, X. Wang, A. Starkey, S.G. Armato
• Journal: ArXiv
• Published: 2023
• Read it: PubMed · Free full text

Abstract

Malignant pleural mesothelioma (MPM) is the most common form of malignant mesothelioma, with exposure to asbestos being the primary cause of the disease. To assess response to treatment, tumor measurements are acquired and evaluated based on a patient's longitudinal computed tomography (CT) scans. Tumor volume, however, is the more accurate metric for assessing tumor burden and response. Automated segmentation methods using deep learning can be employed to acquire volume, which otherwise is a tedious task performed manually. The deep learning-based tumor volume and contours can then be compared with a standard reference to assess the robustness of the automated segmentations. The purpose of this study was to evaluate the impact of probability map threshold on MPM tumor delineations generated using a convolutional neural network (CNN). Eighty-eight CT scans from 21 MPM patients were segmented by a VGG16/U-Net CNN. A radiologist modified the contours generated at a 0.5 probability threshold. Percent difference of tumor volume and overlap using the Dice Similarity Coefficient (DSC) were compared between the standard reference provided by the radiologist and CNN outputs for thresholds ranging from 0.001 to 0.9. CNN annotations consistently yielded smaller tumor volumes than radiologist contours. Reducing the probability threshold from 0.5 to 0.1 decreased the absolute percent volume difference, on average, from 43.96% to 24.18%. Median and mean DSC ranged from 0.58 to 0.60, with a peak at a threshold of 0.5; no distinct threshold was found for percent volume difference. The CNN exhibited deficiencies with specific disease presentations, such as severe pleural effusion or disease in the pleural fissure. No single output threshold in the CNN probability maps was optimal for both tumor volume and DSC. This study emphasized the importance of considering both figures of merit when evaluating deep learning-based tumor segmentations across probability thresholds. This work underscores the need to simultaneously assess tumor volume and spatial overlap when evaluating CNN performance. While automated segmentations may yield comparable tumor volumes to that of the reference standard, the spatial region delineated by the CNN at a specific threshold is equally important.

In plain English

This preprint was a computer-imaging study, not a study of asbestos exposure. The authors used a previously published deep-learning program (a convolutional neural network, or CNN) to outline tumor on 88 CT scans from 21 patients with malignant pleural mesothelioma who had been in an earlier trial, CALGB 30901. The program rates each pixel's probability of being tumor, and a cut-off (threshold) decides what counts as tumor. A radiologist corrected outlines made at a 0.5 threshold to create the reference. At 0.5, the radiologist's tumor volumes were larger than the program's in all but eight cases. Lowering the threshold to 0.1 reduced the average absolute volume difference from 43.96% to 24.18%. Across those thresholds, median and average overlap scores stayed between 0.58 and 0.60. The program performed poorly on scans with severe pleural effusion and with disease near other chest structures. The authors concluded that no single threshold was best for both volume and overlap. They noted that having the radiologist start from the 0.5 outlines biased the comparison.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Retrospective computer-imaging study evaluating an existing, previously trained deep-learning (convolutional neural network) tumor-outlining program at different probability thresholds, with no new training
• Who was studied: 88 baseline and follow-up CT scans from 21 patients with malignant pleural mesothelioma, drawn from the CALGB 30901 trial. That trial had 49 patients with unresectable epithelioid, sarcomatoid, or mixed-type MPM randomized to maintenance pemetrexed or observation. The scans came from 21 different scanner models.
• Exposure: No asbestos exposure was studied. The factor varied was the probability threshold (0.001 to 0.9) applied to the program's output to decide which pixels counted as tumor.
• Main results:
  • At the 0.5 threshold, the radiologist's tumor volumes were larger than the program's in all but eight cases.
  • At the 0.5 threshold, the mean percent difference in volume between the radiologist's and the program's outlines was 43% (range -17% to 126%, median 34%). Eleven scans (12.5%) were within plus or minus 5% of no difference.
  • Across all 88 scans, the average absolute percent difference in volume fell as the threshold was lowered. It was 43.96 (plus or minus 34.48) at 0.5 and 24.18 (plus or minus 18.87) at 0.1, then rose to 28.16 (plus or minus 18.41) at 0.01.
  • Overlap with the radiologist's outlines (Dice similarity coefficient) changed little across thresholds. Average and median values stayed between 0.58 and 0.60 for thresholds 0.5 to 0.1. The average fell to 0.54 at the 0.01 threshold.
  • When the best-performing threshold was tallied for each comparison, overlap scores showed a peak at 0.5. Volume difference showed no distinct best threshold.
  • The program identified tumor unreliably in scans with severe pleural effusion and in disease above the aortic arch or below the pulmonary vein. In four scans from three patients that fell outside the agreement limits, it missed disease around the spleen, vertebral column, and heart, and it labeled severe pleural effusion as tumor.
  • When the analysis was limited to CT sections between the aortic arch and the pulmonary vein, volume differences were generally smaller and overlap scores higher. Average absolute volume difference was 44.00 (plus or minus 31.10) at 0.5 and 22.18 (plus or minus 19.26) at 0.1. Median Dice was 0.65 at 0.5. The 0.01 threshold did worse in this subset.
• The authors' conclusion: The authors concluded that overlap showed a clear peak at the 0.5 threshold, but no threshold clearly minimized the volume difference. They concluded that volume and spatial overlap must both be assessed when evaluating the program, and they called the work a pilot study.
• Limitations the authors state:
  • The radiologist modified the program's 0.5-threshold outlines rather than drawing tumor from scratch. The authors said this introduced bias and explains the overlap peak at 0.5.
  • The program used a two-dimensional design rather than more advanced techniques such as a three-dimensional design.
  • P-values were not adjusted for multiple comparisons. The authors gave the small sample size and the hypothesis-generating nature of the study as the reason.
  • The program had not been trained on regions at the level of the diaphragm or near the lung apices, so poor performance there was expected.

Topics

Medical terms in this study, explained in our Lexicon: Pleural Mesothelioma, Malignant Mesothelioma, Tumor Burden, CT Scan, Pleural Effusion.

Related studies

• Combining a gravitational search algorithm, particle swarm optimization, and fuzzy rules to improve the classification performance of a feed-forward neural network — M.L. Huang, Y.C. Chou (2019)
• Randomized Study of Maintenance Pemetrexed Versus Observation for Treatment of Malignant Pleural Mesothelioma: CALGB 30901 — A.Z. Dudek, X. Wang, et al. (2020)
• Computerized segmentation and measurement of malignant pleural mesothelioma — W.F. Sensakovic, S.G. Armato, et al. (2011)
• Validation of Progression-Free Survival as a Surrogate Endpoint for Overall Survival in Malignant Mesothelioma: Analysis of Cancer and Leukemia Group B and North Central Cancer Treatment Group (Alliance) Trials — X. Wang, X. Wang, et al. (2017)
• Vatalanib in malignant mesothelioma: a phase II trial by the Cancer and Leukemia Group B (CALGB 30107) — T. Jahan, L. Gu, et al. (2012)
• The influence of initial outlines on manual segmentation — W.F. Sensakovic, A. Starkey, et al. (2010)

Browse return to medical research.