Inter-rater agreement for a retrospective exposure assessment of asbestos, chromium, nickel and welding fumes in a study of lung cancer and ionizing radiation

Our plain-English summary of Inter-rater agreement for a retrospective exposure assessment of asbestos, chromium, nickel and welding fumes in a study of lung cancer and ionizing radiation, by E.A. Seel, D.D. Zaebst, M.J. Hein, et al (The Annals of Occupational Hygiene, 2007).

The study

• Authors: E.A. Seel, D.D. Zaebst, M.J. Hein, J. Liu, S.J. Nowlin, P. Chen
• Journal: The Annals of Occupational Hygiene
• Published: 2007;51(7):601-10
• Cited by: 9 articles, as counted by NIH iCite
• Read it: PubMed · Publisher

Abstract

A retrospective exposure assessment of asbestos, welding fumes, chromium and nickel (in welding fumes) was conducted at the Portsmouth Naval Shipyard for a nested case-control study of lung cancer risk from external ionizing radiation. These four contaminants were included because of their potential to confound or modify the effect of a lung cancer-radiation relationship. The exposure assessment included three experienced industrial hygienists from the shipyard who independently assessed exposures for 3519 shop/job/time period combinations. A consensus process was used to resolve estimates with large differences. Final exposure estimates were linked to employment histories of the 4388 study subjects to calculate their cumulative exposures. Inter-rater agreement analyses were performed on the original estimates to better understand the estimation process. Although concordance was good to excellent (78-99%) for intensity estimates and excellent (96-99%) for frequency estimates, overall simple kappa statistics indicated only slight agreement beyond chance (kappa < 0.2). Unbalanced distributions of exposure estimates partly contributed to the weak observed overall inter-rater agreement. Pairwise weighted kappa statistics revealed better agreement between two of the three panelists (kappa = 0.19-0.65). The final consensus estimates were similar to the estimates made by these same two panelists. Overall welding fume exposures were fairly stable across time at the shipyard while asbestos exposures were higher in the early years and fell in the mid-1970s. Mean cumulative exposure for all study subjects was 520 fiber-days cc(-1) for asbestos and 1000 mg-days m(-3) for welding fumes. Mean exposure was much lower for nickel (140 microg-days m(-3)) and chromium (45 microg-days m(-3)). Asbestos and welding fume exposure estimates were positively associated with lung cancer in the nested case-control study. The radiation-lung cancer relationship was attenuated by the inclusion of these two confounders. This exposure assessment provided exposure estimates that aided in understanding of the lung cancer-radiation relationship at the shipyard.

In plain English

This paper described how three industrial hygienists estimated past exposures to asbestos, welding fumes, and chromium and nickel in welding fumes at the Portsmouth Naval Shipyard in Kittery, Maine. The estimates were for a nested case-control study of lung cancer and ionizing radiation. Each hygienist had 25 years of experience at the shipyard. Working independently, they rated 3519 combinations of shop, job and three-year time period, covering work from 1945 to 1996. Consensus meetings resolved large disagreements. Most of their exposure-intensity ratings were within one category of each other. Even so, overall kappa statistics for intensity and frequency ratings, which correct for chance agreement, showed only slight agreement. Panelists 2 and 3 agreed better with each other than with panelist 1, who had read the instructions differently. Of 4388 study subjects, 2796 were ever exposed to asbestos above background. Estimated asbestos exposures fell in the 1970s as asbestos was phased out. The authors concluded that without these estimates, the lung cancer risk from radiation would have been overestimated. They noted the estimates could not be checked against air measurements, so some exposures may have been misclassified.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Retrospective exposure assessment by an expert panel using shop/job/time-period exposure matrices, with analysis of agreement between the raters; part of an updated nested case-control study of lung cancer and external ionizing radiation
• Who was studied: Employment histories of 4388 subjects in a nested case-control study at the Portsmouth Naval Shipyard (Kittery, Maine), with work histories from 1945 to 1996. Exposures were rated for 3519 shop/job/time period combinations by three shipyard industrial hygienists, each with 25 years of experience there.
• Exposure: Asbestos (fiber type not specified), used to insulate boilers and water pipes and present in welding aprons, gloves, millboard, thermal curtains and blankets. Also welding fumes (iron oxide), and chromium and nickel in welding fumes. Ratings covered 8-hour exposure intensity, days per year exposed, and, for asbestos only, respirator use.
• Main results:
  • Raw agreement (concordance) among the three panelists on intensity estimates was 78 to 99% across the four substances. However, overall simple kappa statistics, which correct for chance agreement, indicated only slight agreement for intensity and for frequency estimates for all four contaminants.
  • For asbestos respirator use, concordance was high, while the overall simple kappa indicated barely fair agreement.
  • Agreement was fair to substantial between panelists 2 and 3, but only slight between panelist 1 and each of the others. For asbestos intensity, the weighted kappa for panelists 2 and 3 was 0.42 (unweighted 0.22).
  • Follow-up meetings found that panelist 1 had interpreted the instructions differently from the other two panelists, for example by giving supervisors high exposures.
  • Most ratings fell into the lowest exposure categories. The authors stated that these unbalanced distributions partly explained the weak overall agreement.
  • Final asbestos estimates were slightly higher than the estimates of panelists 2 and 3.
  • Of the 4388 study subjects, 2796 were ever exposed to asbestos above background.
  • Estimated welding fume exposures stayed fairly constant over time, while estimated asbestos exposures declined in the 1970s as asbestos was phased out at the shipyard.
  • Among the 10 most highly asbestos-exposed shop/jobs, pipefitters in the pipe shop and welders in the welding shop had the greatest percentage of highly exposed workers (11% and 6% respectively).
  • Reported from the companion case-control analysis (Yiin et al., 2007) that used these estimates, not this paper's own analysis: lung cancer risk was higher with asbestos exposure than with no exposure (odds ratio 1.04 at 1000 fiber-days per cubic centimeter, 95% confidence interval 1.00 to 1.09).
• The authors' conclusion: The authors concluded that without these asbestos and welding fume estimates, the lung cancer risk from radiation would have been overestimated in the shipyard case-control analysis.
• Limitations the authors state:
  • The estimates could not be validated against air monitoring data, so exposure misclassification remains possible.
  • Air sampling was complaint-based until the mid-1970s, then compliance-driven, and rarely linked to job titles, so it could not be used directly.
  • The panelists were less familiar with exposures from the earlier years and were not used to estimating exposures at the level of detail an epidemiology study needs.
  • Using standardized job titles instead of the original titles may have made the panelists' estimation harder.
  • Panelists did not repeat any of their estimates, so the consistency of each panelist's own ratings (intra-rater reliability) was not assessed.

Topics

The National Library of Medicine indexes this study under subjects our site covers: Lung cancer.

Medical terms in this study, explained in our Lexicon: Lung Cancer, nested case-control study, Cumulative Exposure, Asbestos Exposure.

Related studies

• A nested case-control study of lung cancer risk and ionizing radiation exposure at the portsmouth naval shipyard — J.H. Yiin, S.R. Silver, et al. (2007)
• Ionizing radiation: a risk factor for mesothelioma — J.E. Goodman, M.A. Nascarella, et al. (2009)
• Welding fumes and lung cancer: a meta-analysis of case-control and cohort studies — M.K. Honaryar, R.M. Lunn, et al. (2019)
• Heavy metals, welding fumes, and other occupational exposures, and the risk of kidney cancer: A population-based nested case-control study in three Nordic countries — I.M. Michalek, J.I. Martinsen, et al. (2019)
• Risk of cancer from occupational exposure to ionising radiation: retrospective cohort study of workers in France, the United Kingdom, and the United States (INWORKS) — D.B. Richardson, E. Cardis, et al. (2015)
• Development of an exposure measurement database on five lung carcinogens (ExpoSYN) for quantitative retrospective occupational exposure assessment — S. Peters, R. Vermeulen, et al. (2012)

Browse more research in The Annals of Occupational Hygiene, or return to medical research.