Observer variability in mesothelioma tumor thickness measurements: defining minimally measurable lesions

Our plain-English summary of Observer variability in mesothelioma tumor thickness measurements: defining minimally measurable lesions, by S.G. Armato, A.K. Nowak, R.J. Francis, et al (Journal of Thoracic Oncology, 2014).

The study

• Authors: S.G. Armato, A.K. Nowak, R.J. Francis, M. Kocherginsky, M.J. Byrne
• Journal: Journal of Thoracic Oncology
• Published: 2014;9(8):1187-94
• Cited by: 26 articles, as counted by NIH iCite
• Read it: PubMed · Publisher

Abstract

Introduction: Single time-point unidimensional tumor thickness measurements define measurable disease for clinical trial inclusion and also constitute a field in the International Association for the Study of Lung Cancer prospective mesothelioma staging database. The modified Response Evaluation Criteria in Solid Tumors (RECIST) guidelines for mesothelioma did not alter the 10-mm minimum tumor measurement recommendation. However, as computed tomography technology has advanced, we sought to examine whether interobserver agreement was acceptable at smaller tumor thickness in mesothelioma.

Methods: The primary observer selected 170 discrete measurement sites from 105 thoracic computed tomography scans from 50 consenting patients with pleural mesothelioma. Sites represented a range of tumor thickness, lesion morphology, and location. The outer (parietal) tumor margin was marked at each site and presented to five additional observers, who then selected the visceral margin of the tumor to create a line segment that captured tumor thickness. Relative differences among the observer measurements were estimated using a random-effects analysis of variance model to identify the smallest tumor thickness at which linear measurements could be made reliably.

Results: Systematic bias was observed, with some observers consistently measuring larger or smaller thicknesses than the thickness measured by others. The mean range across all 170 sites was 15.1% of the mean per-site measurement (SD = 9.1%; median range, 13.1%). Measurements acquired at sites with mean tumor thickness less than 7.5 mm demonstrated interobserver variability with a 75th percentile included 20% of the tumor thickness. The 95% confidence interval for relative interobserver measurement differences obtained for mean measurement lengths in the range 5 to 7.5 mm does not exceed the RECIST thresholds.

Conclusions: This study has implications for the definition of minimally measurable tumor adopted by clinical trial and staging protocols. Although the statistical findings suggest that a reduction in "minimally measurable disease" from 10 mm to 5 or 7.5 mm might be warranted, clinical factors may ultimately dictate the most appropriate definition.

In plain English

This study measured how much doctors' measurements of pleural mesothelioma tumor thickness on CT scans differ from one another. The authors used 105 existing chest CT scans from 50 patients at a hospital in Perth, Western Australia. A medical oncologist chose 170 tumor sites covering a range of thicknesses and shapes and marked the tumor's outer edge at each one. Five other physicians then each marked the inner edge, so six observers measured every site. Some observers measured consistently larger or smaller than the others. Across all sites, the gap between the largest and smallest measurement averaged 15.1% of the site's mean thickness. Disagreement was greatest for thin tumors. For sites under 5 mm, the 95% confidence interval for differences between observers was -25.6% to 34.5%. For sites of 5 to 7.5 mm it was -14.6% to 17.1%, which falls inside the thresholds used to call tumor shrinkage (-30%) or growth (+20%). The authors concluded that the 10 mm minimum for a "measurable" tumor might be lowered to 5 or 7.5 mm. They added that real-world variability absent from this study, and clinical factors, may decide the best definition. The study did not examine asbestos exposure.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Retrospective observer-variability (measurement agreement) study using previously acquired CT scans
• Who was studied: 105 thoracic CT scans from 50 patients with pleural mesothelioma at Sir Charles Gairdner Hospital, Perth, Western Australia (45 men, 5 women). A primary observer chose 170 measurement sites, which were measured by six observers in total. Study dates were not given.
• Exposure: Not applicable: the study did not examine asbestos or any other exposure. It examined differences between observers measuring tumor thickness on CT scans using the modified RECIST approach.
• Main results:
  • The primary observer (Observer 0) measured less than the average of the other observers at about 154 of 170 sites (90.6%).
  • Observers 2 and 5 measured more than the average of the other observers at 136 (80.0%) and 143 (84.1%) of sites, respectively.
  • Across the 170 sites, the mean per-site tumor thickness was 11.61 mm (range 2.60 to 54.35 mm, SD 8.19 mm).
  • Across all 170 sites, the range between the largest and smallest observer measurement averaged 15.1% of the mean per-site measurement (SD 9.1%; median 13.1%).
  • Across all measurements, the 95% confidence interval for interobserver thickness differences was -16.8% to 20.1%.
  • Variability between observers decreased as tumor thickness increased, and became more consistent at sites thicker than 7.5 mm.
  • For sites with mean thickness under 5 mm, the 95% confidence interval for interobserver differences was -25.6% to 34.5%.
  • For sites with mean thickness of 5 to 7.5 mm, the 95% confidence interval was -14.6% to 17.1%. This lies inside the RECIST response thresholds of -30% and +20%.
  • The confidence interval was narrower for convex mass lesions (-10.6% to 11.9%), which tended to have longer measurements.
  • The mean angular spread between observers' measurement lines was 10.4 degrees, plus or minus 5.6 degrees (range 3.4 to 37.0 degrees).
• The authors' conclusion: The authors concluded that their statistical findings suggest the minimum measurable tumor thickness in mesothelioma might be reduced from 10 mm to 5 or 7.5 mm. They noted that real-world sources of variability and clinical factors may ultimately determine the most appropriate definition.
• Limitations the authors state:
  • Real-world sources of variability were not present in this study and may require a larger margin above the minimum. The authors list differences in patient positioning, differences in imaging settings, and different readers across a patient's sequence of scans.
  • Measurement sites were chosen to cover a range of thicknesses and shapes, not for clinical relevance. The site distributions therefore do not reflect tumor morphology or location in patients generally.
  • The findings apply to single time-point measurements on a baseline scan.
  • The study did not test whether linear measurements are a valid stand-in for total tumor burden.

Topics

The National Library of Medicine indexes this study under subjects our site covers: Mesothelioma.

Medical terms in this study, explained in our Lexicon: Mesothelioma, Clinical Trial, Lung Cancer, Mesothelioma Staging, Modified RECIST (mRECIST), Pleural Mesothelioma, confidence interval, Staging.

Related studies

• Influence of inter-observer delineation variability on radiomics stability in different tumor sites — M. Pavic, M. Bogowicz, et al. (2018)
• Variability of tumor area measurements for response assessment in malignant pleural mesothelioma — Z.E. Labby, C. Straus, et al. (2013)
• Observer variation in computed tomography of pleural lesions in subjects exposed to indoor asbestos — H. De Raeve, J.A. Verschakelen, et al. (2001)
• Assessment of therapy responses and prediction of survival in malignant pleural mesothelioma through computer-aided volumetric measurement on computed tomography scans — F. Liu, B. Zhao, et al. (2010)
• Correlation of patient survival with clinical tumor measurements in malignant pleural mesothelioma — F. Li, M. Ahmad, et al. (2019)
• Serum soluble mesothelin concentrations in malignant pleural mesothelioma: relationship to tumor volume, clinical stage and changes in tumor burden — J. Creaney, R.J. Francis, et al. (2011)

Browse more research in Journal of Thoracic Oncology, or return to medical research.