Medical College of Wisconsin
CTSICores SearchResearch InformaticsREDCap

Comprehensive Clinical Usability-Oriented Contour Quality Evaluation for Deep Learning Auto-segmentation: Combining Multiple Quantitative Metrics Through Machine Learning. Pract Radiat Oncol 2024 Sep 02

Date

09/05/2024

Pubmed ID

39233005

DOI

10.1016/j.prro.2024.07.007

Scopus ID

2-s2.0-85204920429 (requires institutional sign-in at Scopus site)   1 Citation

Abstract

PURPOSE: The current commonly used metrics for evaluating the quality of auto-segmented contours have limitations and do not always reflect the clinical usefulness of the contours. This work aims to develop a novel contour quality classification (CQC) method by combining multiple quantitative metrics for clinical usability-oriented contour quality evaluation for deep learning-based auto-segmentation (DLAS).

METHODS AND MATERIALS: The CQC was designed to categorize contours on slices as acceptable, minor edit, or major edit based on the expected editing effort/time with supervised ensemble tree classification models using 7 quantitative metrics. Organ-specific models were trained for 5 abdominal organs (pancreas, duodenum, stomach, small, and large bowels) using 50 magnetic resonance imaging (MRI) data sets. Twenty additional MRI and 9 computed tomography (CT) data sets were employed for testing. Interobserver variation (IOV) was assessed among 6 observers and consensus labels were established through majority vote for evaluation. The CQC was also compared with a threshold-based baseline approach.

RESULTS: For the 5 organs, the average area under the curve was 0.982 ± 0.01 and 0.979 ± 0.01, the mean accuracy was 95.8% ± 1.7% and 94.3% ± 2.1%, and the mean risk rate was 0.8% ± 0.4% and 0.7% ± 0.5% for MRI and CT testing data set, respectively. The CQC results closely matched the IOV results (mean accuracy of 94.2% ± 0.8% and 94.8% ± 1.7%) and were significantly higher than those obtained using the threshold-based method (mean accuracy of 80.0% ± 4.7%, 83.8% ± 5.2%, and 77.3% ± 6.6% using 1, 2, and 3 metrics).

CONCLUSIONS: The CQC models demonstrated high performance in classifying the quality of contour slices. This method can address the limitations of existing metrics and offers an intuitive and comprehensive solution for clinically oriented evaluation and comparison of DLAS systems.

Author List

Zhang Y, Amjad A, Ding J, Sarosiek C, Zarenia M, Conlin R, Hall WA, Erickson B, Paulson E

Authors

Asma Amjad PhD Instructor in the Radiation Oncology department at Medical College of Wisconsin
Beth A. Erickson MD Professor in the Radiation Oncology department at Medical College of Wisconsin
William Adrian Hall MD Professor in the Radiation Oncology department at Medical College of Wisconsin
Eric Paulson PhD Chief, Professor in the Radiation Oncology department at Medical College of Wisconsin
Christina Sarosiek Medical Physicist Assistant in the Radiation Oncology department at Medical College of Wisconsin