Automated ancillary cancer history classification for mesothelioma patients from free-text clinical reports

Our plain-English summary of Automated ancillary cancer history classification for mesothelioma patients from free-text clinical reports, by R.A. Wilson, W.W. Chapman, S.J. Defries, et al (Journal of Pathology Informatics, 2010).

The study

• Authors: R.A. Wilson, W.W. Chapman, S.J. Defries, M.J. Becich, B.E. Chapman
• Journal: Journal of Pathology Informatics
• Published: 2010;1:24
• Cited by: 12 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

Background: Clinical records are often unstructured, free-text documents that create information extraction challenges and costs. Healthcare delivery and research organizations, such as the National Mesothelioma Virtual Bank, require the aggregation of both structured and unstructured data types. Natural language processing offers techniques for automatically extracting information from unstructured, free-text documents.

Methods: Five hundred and eight history and physical reports from mesothelioma patients were split into development (208) and test sets (300). A reference standard was developed and each report was annotated by experts with regard to the patient's personal history of ancillary cancer and family history of any cancer. The Hx application was developed to process reports, extract relevant features, perform reference resolution and classify them with regard to cancer history. Two methods, Dynamic-Window and ConText, for extracting information were evaluated. Hx's classification responses using each of the two methods were measured against the reference standard. The average Cohen's weighted kappa served as the human benchmark in evaluating the system.

Results: Hx had a high overall accuracy, with each method, scoring 96.2%. F-measures using the Dynamic-Window and ConText methods were 91.8% and 91.6%, which were comparable to the human benchmark of 92.8%. For the personal history classification, Dynamic-Window scored highest with 89.2% and for the family history classification, ConText scored highest with 97.6%, in which both methods were comparable to the human benchmark of 88.3% and 97.2%, respectively.

Conclusion: We evaluated an automated application's performance in classifying a mesothelioma patient's personal and family history of cancer from clinical reports. To do so, the Hx application must process reports, identify cancer concepts, distinguish the known mesothelioma from ancillary cancers, recognize negation, perform reference resolution and determine the experiencer. Results indicated that both information extraction methods tested were dependant on the domain-specific lexicon and negation extraction. We showed that the more general method, ConText, performed as well as our task-specific method. Although Dynamic- Window could be modified to retrieve other concepts, ConText is more robust and performs better on inconclusive concepts. Hx could greatly improve and expedite the process of extracting data from free-text, clinical records for a variety of research or healthcare delivery organizations.

In plain English

This study did not examine asbestos. It tested a computer program called Hx. The program reads hospital history and physical reports and decides whether a mesothelioma patient has a personal history of another cancer, and whether there is a family history of any cancer. The authors randomly selected 120 mesothelioma patients at the University of Pittsburgh Medical Center and tested the program on 300 of their reports. The correct answers were the majority view of three annotators. Of the 300 test reports, 20% showed a personal history of another cancer and 27% a family history of cancer. Hx used two methods, Dynamic-Window and ConText, to build "cancer frames" (structured notes on each cancer mention). Each method had an overall accuracy of 96.2% and made 23 errors out of 600 answers. Their F-measures (91.8% and 91.6%) were similar to the human benchmark, the annotators' average kappa of 92.8%. Gaps in the word list caused 17 of the 46 total errors. The authors concluded that ConText built cancer frames as well as their task-specific method. They noted that the number of unique patients was limited.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Development and evaluation of an automated natural language processing (computer text-reading) program, compared against a human-annotated reference standard
• Who was studied: 120 randomly selected patients from the University of Pittsburgh Medical Center Medical Archival System who had a positive surgical pathology report of mesothelioma and a hospital admission after diagnosis. Their history and physical reports (508 total) were split into 208 development and 300 test reports.
• Exposure: Not applicable. The study evaluated a computer program that classifies cancer history in clinical reports. Asbestos exposure was not studied.
• Main results:
  • In the reference standard for the 300 test reports, 20% had a personal history of ancillary cancer and 27% had a family history of any cancer.
  • Hx was tested with two methods for building cancer frames, Dynamic-Window and ConText. Both used the same report parsing and frame evaluation modules.
  • Overall accuracy on both questions was 96.2% for both the Dynamic-Window and ConText methods.
  • Overall F-measures were 91.8% for Dynamic-Window and 91.6% for ConText. The human benchmark (average kappa among annotators) was 92.8%.
  • For personal history, Dynamic-Window scored 89.2% against a human benchmark of 88.3%. For family history, ConText had an F-measure of 97.6% against a benchmark of 97.2%.
  • Each method made 23 errors out of 600 responses (300 reports, two questions each).
  • Gaps in the program's word list (lexicon) accounted for 17 of the 46 total errors across both methods.
  • The authors calculated that correcting the lexicon would raise the overall F-measure from 91.8% to 93.8% for Dynamic-Window and from 91.6% to 95.7% for ConText.
• The authors' conclusion: The authors concluded that the more general ConText method built cancer frames as well as their task-specific Dynamic-Window method, and that with improvements it would outperform Dynamic-Window.
• Limitations the authors state:
  • The low incidence of mesothelioma limited the number of unique patients available.
  • The test set included only 70 unique patients, with a median of three reports per patient.
  • One error for each method came from an incorrect reference standard answer: two of the three annotators missed a cancer term.

Topics

Medical terms in this study, explained in our Lexicon: Mesothelioma.

Related studies

• Mesothelin family proteins and diagnosis of mesothelioma: analytical evaluation of an automated immunoassay and preliminary clinical results — F. Di Serio, A. Fontana, et al. (2007)
• Awake Thoracic Surgery with Thoracic Epidural Anesthesia in High-Risk Patients: Three Case Reports and Clinical Insights — S. Kesici, F. Uysal, et al. (2025)
• Phase II study of first-line bortezomib and cisplatin in malignant pleural mesothelioma and prospective validation of progression free survival rate as a primary end-point for mesothelioma clinical trials (European Organisation for Research and Treatment of Cancer 08052) — M.E. O'Brien, R.M. Gaafar, et al. (2013)
• Electron microscopy: a brief history and review of current clinical application — R.E. Gordon (2014)
• Clinical study of asbestos-related lung cancer in Japan with special reference to occupational history — T. Kishimoto, K. Gemba, et al. (2010)
• Ancillary studies in pleural, pericardial, and peritoneal effusion cytology — K.E. Sundling, E.S. Cibas (2018)

Browse return to medical research.