Influence of inter-observer delineation variability on radiomics stability in different tumor sites

Our plain-English summary of Influence of inter-observer delineation variability on radiomics stability in different tumor sites, by M. Pavic, M. Bogowicz, X. Würms, et al (Acta Oncologica, 2018).

The study

• Authors: M. Pavic, M. Bogowicz, X. Würms, S. Glatz, T. Finazzi, O. Riesterer, J. Roesch, L. Rudofsky, M. Friess, P. Veit-Haibach, M. Huellner, I. Opitz, W. Weder, T. Frauenfelder, M. Guckenberger, S. Tanadini-Lang
• Journal: Acta Oncologica
• Published: 2018;57(8):1070-1074
• Cited by: 168 articles, as counted by NIH iCite
• Read it: PubMed · Publisher

Abstract

Background: Radiomics is a promising methodology for quantitative analysis and description of radiological images using advanced mathematics and statistics. Tumor delineation, which is still often done manually, is an essential step in radiomics, however, inter-observer variability is a well-known uncertainty in radiation oncology. This study investigated the impact of inter-observer variability (IOV) in manual tumor delineation on the reliability of radiomic features (RF).

Methods: Three different tumor types (head and neck squamous cell carcinoma (HNSCC), malignant pleural mesothelioma (MPM) and non-small cell lung cancer (NSCLC)) were included. For each site, eleven individual tumors were contoured on CT scans by three experienced radiation oncologists. Dice coefficients (DC) were calculated for quantification of delineation variability. RF were calculated with an in-house developed software implementation, which comprises 1404 features: shape (n = 18), histogram (n = 17), texture (n = 137) and wavelet (n = 1232). The IOV of RF was studied using the intraclass correlation coefficient (ICC). An ICC >0.8 indicates a good reproducibility. For the stable RF, an average linkage hierarchical clustering was performed to identify classes of uncorrelated features.

Results: Median DC was high for NSCLC (0.86, range 0.57-0.90) and HNSCC (0.72, 0.21-0.89), whereas it was low for MPM (0.26, 0-0.9) indicating substantial IOV. Stability rate of RF correlated with DC and depended on tumor site, showing a high stability in NSCLC (90% of total parameters), acceptable stability in HNSCC (59% of total parameters) and low stability in MPM (36% of total parameters). Shape features showed the weakest stability across all tumor types. Hierarchical clustering revealed 14 groups of correlated and stable features for NSCLC and 6 groups for both HNSCC and MPM.

Conclusion: Inter-observer delineation variability has a relevant influence on radiomics analysis and is strongly influenced by tumor type. This leads to a reduced number of suitable imaging features.

In plain English

This study did not examine asbestos exposure. It looked at how much the measurements used in radiomics (computer analysis of medical images) change when different doctors outline the same tumor. Three radiation oncologists in a single-center study each independently outlined eleven tumors of each of three types on pretreatment CT scans: malignant pleural mesothelioma (MPM), non-small cell lung cancer (NSCLC) and head and neck squamous cell carcinoma (HNSCC). The software calculated 1404 image features. Overlap between outlines, measured by the Dice coefficient (0 means no overlap, 1 means identical), had a median of 0.86 for NSCLC and 0.72 for HNSCC, but only 0.26 for MPM. Features counted as stable (ICC above 0.8) made up 90% in NSCLC, 59% in HNSCC and only 36% in MPM. In MPM, wavelet features had the highest share of stable features (39%). Among shape features, only volume was stable in all three tumor types. The authors concluded that differences in tumor outlining have relevant influences on radiomics results. They noted that the single-center setting might overstate the number of stable features.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Single-center inter-observer variability (reproducibility) study of CT-based radiomic features
• Who was studied: Eleven randomly selected tumors each of head and neck squamous cell carcinoma, non-small cell lung cancer and malignant pleural mesothelioma (all eleven MPM tumors were T3), from University Hospital Zurich, Switzerland; dates of patient care not stated
• Exposure: No asbestos exposure was studied. The factor examined was variation between three radiation oncologists in manually outlining the gross tumor volume on pretreatment planning CT scans.
• Main results:
  • Median Dice coefficient (agreement between outlines) was 0.86 (range 0.57–0.90) for NSCLC and 0.72 (range 0.21–0.89) for HNSCC
  • Median Dice coefficient was low for MPM: 0.26 (range 0–0.9)
  • Of all extracted radiomic features, 90% were stable for NSCLC and 59% for HNSCC, compared with only 36% for MPM
  • Dice coefficient correlated highly with the stability of radiomic features
  • In MPM images, wavelet features had the highest percentage of stable features (39%)
  • Among shape features, only volume was stable across all three tumor sites
  • Hierarchical clustering found 14 groups of correlated and stable features for NSCLC and 6 groups for both HNSCC and MPM
• The authors' conclusion: The authors concluded that differences in tumor delineation lead to relevant influences on radiomics results. They added that some stable features could still be found even for MPM, despite its large inter-observer differences.
• Limitations the authors state:
  • Single-center setting. Observers at the same institution may produce more similar outlines, possibly overestimating the number of stable features compared with a multi-institutional study.
  • Small cohorts of patients (the authors stated that their strict ICC threshold kept type I and type II errors at the 0.05 and 0.20 levels)

Topics

The National Library of Medicine indexes this study under subjects our site covers: Lung cancer, Mesothelioma.

Medical terms in this study, explained in our Lexicon: Squamous Cell Carcinoma, Pleural Mesothelioma, Non-Small Cell Lung Cancer, CT Scan.

Related studies

• Assessing Tumor-Infiltrating Lymphocytes in Solid Tumors: A Practical Review for Pathologists and Proposal for a Standardized Method from the International Immuno-Oncology Biomarkers Working Group: Part 2: TILs in Melanoma, Gastrointestinal Tract Carcinomas, Non-Small Cell Lung Carcinoma and Mesothelioma, Endometrial and Ovarian Carcinomas, Squamous Cell Carcinoma of the Head and Neck, Genitourinary Carcinomas, and Primary Brain Tumors — S. Hendry, R. Salgado, et al. (2017)
• Observer variability in mesothelioma tumor thickness measurements: defining minimally measurable lesions — S.G. Armato, A.K. Nowak, et al. (2014)
• Occupational dust exposure and head and neck squamous cell carcinoma risk in a population-based case-control study conducted in the greater Boston area — S.M. Langevin, M.D. McClean, et al. (2013)
• EphB4: A promising target for upper aerodigestive malignancies — R. Salgia, P. Kulkarni, et al. (2018)
• Inter-reader variability in chest radiography and HRCT for the early detection of asbestos-related lung and pleural abnormalities in a cohort of 636 asbestos-exposed subjects — E. Ochsmann, T. Carl, et al. (2010)
• Variability of tumor area measurements for response assessment in malignant pleural mesothelioma — Z.E. Labby, C. Straus, et al. (2013)

Browse more research in Acta Oncologica, or return to medical research.