The influence of initial outlines on manual segmentation

Our plain-English summary of The influence of initial outlines on manual segmentation, by W.F. Sensakovic, A. Starkey, R. Roberts, et al (Medical Physics, 2010).

The study

• Authors: W.F. Sensakovic, A. Starkey, R. Roberts, C. Straus, P. Caligiuri, M. Kocherginsky, S.G. Armato
• Journal: Medical Physics
• Published: 2010;37(5):2153-8
• Cited by: 6 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

Purpose: Initial outlines are often presented as an aid to reduce the time-cost associated with manual segmentation and measurement of structures in medical images. This study evaluated the influence of initial outlines on manual segmentation intraobserver and interobserver precision.

Methods: Three observers independently outlined all pleural mesothelioma tumors present in five computed tomography (CT) sections in each of 30 patient scans. After a lapse of time, each observer was presented with the same series of CT sections with the outlines of each observer superimposed as initial outlines. Each observer created altered outlines by altering the initial outlines to reflect their perception of the tumor boundary. Altered outlines were compared to original outlines using the Jaccard similarity coefficient (J). Intraobserver and interobserver precision of observer outlines were calculated by applying linear mixed effects analysis of variance models to the J values. The percent of minor alterations (alterations that resulted in only slight changes in the initial outline) was also recorded.

Results: The average J value between pairs of observer original outlines was 0.371. The average J value between pairs of observer outlines when altered from an identical initial outline was 0.796, indicating increased interobserver precision. The average difference between J values of an observer's segmentation created by altering their own initial outline and when altering a different observer's initial outline was 0.476, indicating initial outlines strongly influence intraobserver precision. Observers made minor alterations on 74.5% of initial outlines with which they were presented.

Conclusions: Intraobserver and interobserver precision were strongly dependent on the initial outline. These effects are likely due to the tendency of observers to make only minor corrections to initial outlines. This finding could impact observer study design, tumor growth assessment, computer-aided diagnosis system validation, and radiation therapy target volume definition when initial outlines are used as an observer aid.

In plain English

This study did not examine asbestos exposure. It tested how showing a pre-drawn outline affects the way doctors trace pleural mesothelioma tumors on CT scans. The authors used 30 existing chest CT scans from patients with malignant pleural mesothelioma (25 men and five women, aged 50 to 83). Two attending radiologists and one resident first outlined tumor on five selected sections per scan. After one month, each observer re-outlined the same sections, starting from each observer's earlier outline, shown anonymously. Agreement was scored from 0 (no overlap) to 1 (identical). Of 150 sections, 118 were analyzed. Without starting outlines, agreement between observers averaged 0.371. When they started from the same outline, it averaged 0.796. New outlines matched an observer's own first outlines at 0.860 to 0.957 when editing their own work, but only 0.320 to 0.559 when editing someone else's. Observers made only minor changes to 74.5% of starting outlines. The authors concluded that precision depended strongly on the starting outline. They noted that only three observers took part.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Observer study of manual tumor outlining on CT images, in two phases: outlining without initial outlines, then altering superimposed initial outlines after a lapse of time
• Who was studied: 30 diagnostic thoracic helical CT scans from patients with malignant pleural mesothelioma (25 males, five females; age range 50 to 83 years; mean 68), enrolled in chemotherapy clinical trials and scanned at the authors' institution (University of Chicago). Three observers outlined the scans: two attending radiologists (Observers A and C) and one resident (Observer B). Five randomly selected sections per scan gave 150 sections.
• Exposure: No asbestos or fiber exposure was studied. The condition tested was the presence of an initial outline (each observer's Phase 1 outline) shown to observers when they outlined tumor in Phase 2.
• Main results:
  • 118 of the original 150 sections were analyzed. Seventeen sections were excluded because all three observers agreed no tumor was present, and 15 because only one observer outlined tumor.
  • Without initial outlines (Phase 1), the average Jaccard agreement between pairs of observers was 0.371. Pairwise values were 0.325 (95% CI 0.251 to 0.398) for A vs B, 0.496 (0.422 to 0.569) for A vs C, and 0.291 (0.217 to 0.365) for B vs C.
  • In Phase 1, the J value for the two attending radiologists (A and C) was on average 0.188 higher than the J values involving the resident (Observer B), a statistically significant difference (p less than 0.01).
  • When observers altered an identical initial outline, the average J value between pairs of observers was 0.796. Pairwise values were 0.763 for A vs B, 0.824 for A vs C, and 0.802 for B vs C. These were higher than the Phase 1 values (p less than 0.001 for all comparisons).
  • When observers altered their own Phase 1 outlines, the J value against their original outlines ranged from 0.860 to 0.957.
  • When observers altered another observer's outline, the J value against their own original outlines ranged from 0.320 to 0.559. The average difference between the two conditions was 0.476.
  • Observers made only minor alterations (an altered outline with Jaccard agreement greater than 0.9 with the initial outline) on 74.5% of initial outlines. By observer and outline source, the percentages ranged from 50.8% to 91.5%.
  • Phase 1 outlines that were altered in Phase 2 were less complex than those that were not altered. Median maximal entropy was 1.1721 for altered outlines vs 1.2108 for unaltered outlines (p=0.016).
• The authors' conclusion: The authors concluded that intraobserver and interobserver precision depended strongly on the initial outline. They attributed this to observers tending to make only small alterations to initial outlines, even when those outlines differed considerably from their own independent outlines.
• Limitations the authors state:
  • Only three observers provided outlines.
  • The less experienced observer's original outlines differed significantly from those of the more experienced observers.
  • Time savings were not measured.
  • How often and how much observers altered outlines may depend on the subtlety of the tumor boundary. The mesothelioma tumors had both high-contrast and low-contrast boundaries.
  • Effects on clinical tasks such as tumor volume calculation or response assessment were not directly investigated.

Topics

Medical terms in this study, explained in our Lexicon: Pleural Mesothelioma.

Related studies

• Computerized segmentation and measurement of malignant pleural mesothelioma — W.F. Sensakovic, S.G. Armato, et al. (2011)
• Three-dimensional stereoscopic volume rendering of malignant pleural mesothelioma — N.M. Mollberg, N.M. Parsad, et al. (2012)
• Characterization of mesothelioma and tissues present in contrast-enhanced thoracic CT scans — N. Corson, W.F. Sensakovic, et al. (2011)
• Convolutional Neural Networks for Segmentation of Malignant Pleural Mesothelioma: Analysis of Probability Map Thresholds (CALGB 30901, Alliance) — M. Shenouda, E. Gudmundsson, et al. (2023)
• PET-guided dose escalation tomotherapy in malignant pleural mesothelioma — A. Fodor, C. Fiorino, et al. (2011)
• Computer-assisted diagnosis for early stage pleural mesothelioma: towards automated detection and quantitative assessment of pleural thickening from thoracic CT images — K. Chaisaowong, T. Aach, et al. (2007)

Browse more research in Medical Physics, or return to medical research.