Variability in interpretation of radiographs for asbestosis abnormalities: problems and solutions

Our plain-English summary of Variability in interpretation of radiographs for asbestosis abnormalities: problems and solutions, by A.M. Ducatman (Annals of the New York Academy of Sciences, 1991).

The study

• Author: A.M. Ducatman
• Journal: Annals of the New York Academy of Sciences
• Published: 1991;643:108-20
• Type of article: Review
• Cited by: 14 articles, as counted by NIH iCite
• Read it: PubMed · Publisher
• In J-M's library: Johns-Manville's company medical library held a copy (J-M no. 004853).

Abstract

Interobserver variation continues to be a substantial problem for interpretation of asbestos-workers' X-rays, despite incremental improvements in the International Labour Office (ILO) classification system. The apparent degree of agreement/variability is affected by the population disease prevalence. For most populations, experts will be able to agree routinely about normal films. Yet designed trials show that selected experts exhibit a two-fold or greater range for perceived prevalence of abnormal profusions. This degree of variability would be adequate for many population studies were it achieved under real-life conditions of interpretation. Unfortunately, the realistic degree of interobserver variability is probably much higher under actual population surveillance conditions. Asbestos population surveillance programs have provided little or no quality assurance for ILO interpretations, so the very high variability suggested by available data are not surprising and are not the fault of the classification. Recommendations include required participation in quality-assurance panels for those who wish to be certified as interpreters, with consistent feedback and the dropping of outliers. If quality-assurance programs are implemented, the classification will achieve its intended epidemiologic purposes.

In plain English

This article mainly reviewed earlier studies, adding some new analysis of published data and some previously unpublished data. The author searched the English-language literature for studies of how consistently doctors read workers' chest X-rays under the International Labour Office (ILO) classification. Asbestos-worker studies were emphasized; coal- and silica-worker studies were also used. In formal trials, readers disagreed widely. In one trial by Rossiter, 12 readers called between 28 and 71 of 100 films normal. In a later international trial, 12 readers rated 29% to 65% of films category 1/1 or more. Under real surveillance conditions the spread was larger. In an earlier U.S. Navy study by Ducatman and colleagues, 23 certified readers received randomly distributed films of 105,029 asbestos-exposed workers. The share they rated category 1/0 or greater ranged from 0.05% to 10.93%. In a NIOSH coal-miner survey, first- and second-reader agreement (kappa 0.29) was called not much better than chance. The author concluded that the problem lay with the readers rather than the classification. The author recommended required quality-assurance panels with feedback and dropping outlier readers.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Literature review of studies on intra- and interobserver variability in ILO chest X-ray classification, with some new analysis of previously published data and some previously unpublished data
• Who was studied: Published radiograph-reading variability studies, emphasizing asbestos-worker studies using recent ILO modifications. Coal- and silica-worker studies were also evaluated.
• Exposure: Not an exposure study. The subject was how readers classify asbestos surveillance chest X-rays under the ILO system. No fiber types were specified.
• Main results:
  • In Rossiter's UICC/Cincinnati trial, 12 readers classified between 28 and 71 of 100 test films as normal.
  • In a later trial by Rossiter and coworkers of a 100-film set of asbestos-related abnormalities, the prevalence of abnormality (category 1/1 or more) ranged from 29% to 65% among 12 readers from 10 countries.
  • NIOSH coal-miner surveillance data covered films from 14,000 miners. The prevalence of category 0/1 or greater ranged from 0.6% to 94.8% among first readers and from 3.1% to 60.6% among certified B-readers. Agreement between first and second readers (kappa) was 0.29.
  • In the U.S. Navy asbestos surveillance program, films of 105,029 asbestos-exposed workers went to 23 certified readers in rotation. The share rated category 1/0 or greater ranged from 0.05% to 10.93% (median 1.71%).
  • In the Navy data, one reader contributed a third of all abnormalities of category 1/1 or greater. The most experienced readers (course instructors) were about half as likely as other certified readers to see category 1/1 or greater abnormalities.
  • In previously unpublished data from the author on B-readers living in two cities, the mean prevalence of category 1/0 or greater was 0.65% for one group (4 readers, 15,234 films) and 4.35% for the other (5 readers).
  • The author's new analysis of previously published data indicated that the 'middling tendency' (a bias favoring the central minor category) appears to apply least to the most expert observers.
  • Musch and coworkers reported a kappa of 0.60 among three certified readers of coal-workers' films. The article described this as the most encouraging formal-trial result.
  • Parker and colleagues re-examined 566 surveillance films. Pleural abnormality prevalence fell from 30% with a single reader to 4% when a blind, independent consensus of two of three NIOSH experts was required.
  • In the Canadian Pneumoconiosis Reading Panel, which gave readers ongoing feedback, 14 of 16 readers showed increasing agreement with the standard expert over time, while 2 showed decreasing agreement.
• The authors' conclusion: The author concluded that the ILO classification should be kept and that the problems lay with its users rather than the system. The author recommended required participation in quality-assurance panels with feedback. The author stated that, with such programs, the classification would achieve its intended epidemiologic purposes.
• Limitations the authors state:
  • The author stated that the benefit of ongoing quality-assurance testing, feedback and dropping outliers was possible but not yet documented.
  • The author stated that the wide range in the NIOSH coal data reflected an unknown mixture of observer variability and nonrandom distribution of films. The author added that uncertified first readers likely contributed extra variability.

Topics

The National Library of Medicine indexes this study under subjects our site covers: Asbestosis.

Related studies

• Variability in interpretation among B-readers in the U.S. Navy Asbestos Medical Surveillance Program — J.G. Muller, J.M. Lieske, et al. (2008)
• Relationships (I) of International Classification of High-resolution Computed Tomography for Occupational and Environmental Respiratory Diseases with the ILO International Classification of Radiographs of Pneumoconioses for parenchymal abnormalities — T. Tamura, N. Suganuma, et al. (2015)
• Lung function and chest radiograph abnormalities among construction insulators — S.M. Kennedy, S. Vedal, et al. (1991)
• Risk and significance of chest radiograph and pulmonary function abnormalities in an elderly cohort of former nuclear weapons workers — M.A. Mikulski, P.G. Hartley, et al. (2011)
• Chest radiographs in subjects with asbestos-related abnormalities: comparison between ILO categorizations and clinical reading — B. Hilt, A. Borgerson, et al. (1992)
• Comparison of "B" readers' interpretations of chest radiographs for asbestos related changes — J.N. Gitlin, L.L. Cook, et al. (2004)

Browse more research in Annals of the New York Academy of Sciences, or return to medical research.