Comprehensive Clinical Usability-Oriented Contour Quality Evaluation for Deep Learning Auto-segmentation: Combining Multiple Quantitative Metrics Through Machine Learning. Pract Radiat Oncol 2025;15(1):93-102
Date
09/05/2024Pubmed ID
39233005Pubmed Central ID
PMC11711007DOI
10.1016/j.prro.2024.07.007Scopus ID
2-s2.0-85204920429 (requires institutional sign-in at Scopus site) 2 CitationsAbstract
PURPOSE: The current commonly used metrics for evaluating the quality of auto-segmented contours have limitations and do not always reflect the clinical usefulness of the contours. This work aims to develop a novel contour quality classification (CQC) method by combining multiple quantitative metrics for clinical usability-oriented contour quality evaluation for deep learning-based auto-segmentation (DLAS).
METHODS AND MATERIALS: The CQC was designed to categorize contours on slices as acceptable, minor edit, or major edit based on the expected editing effort/time with supervised ensemble tree classification models using 7 quantitative metrics. Organ-specific models were trained for 5 abdominal organs (pancreas, duodenum, stomach, small, and large bowels) using 50 magnetic resonance imaging (MRI) data sets. Twenty additional MRI and 9 computed tomography (CT) data sets were employed for testing. Interobserver variation (IOV) was assessed among 6 observers and consensus labels were established through majority vote for evaluation. The CQC was also compared with a threshold-based baseline approach.
RESULTS: For the 5 organs, the average area under the curve was 0.982 ± 0.01 and 0.979 ± 0.01, the mean accuracy was 95.8% ± 1.7% and 94.3% ± 2.1%, and the mean risk rate was 0.8% ± 0.4% and 0.7% ± 0.5% for MRI and CT testing data set, respectively. The CQC results closely matched the IOV results (mean accuracy of 94.2% ± 0.8% and 94.8% ± 1.7%) and were significantly higher than those obtained using the threshold-based method (mean accuracy of 80.0% ± 4.7%, 83.8% ± 5.2%, and 77.3% ± 6.6% using 1, 2, and 3 metrics).
CONCLUSIONS: The CQC models demonstrated high performance in classifying the quality of contour slices. This method can address the limitations of existing metrics and offers an intuitive and comprehensive solution for clinically oriented evaluation and comparison of DLAS systems.
Author List
Zhang Y, Amjad A, Ding J, Sarosiek C, Zarenia M, Conlin R, Hall WA, Erickson B, Paulson EAuthors
Asma Amjad PhD Assistant Professor in the Radiation Oncology department at Medical College of WisconsinBeth A. Erickson MD Professor in the Radiation Oncology department at Medical College of Wisconsin
William Adrian Hall MD Chair, Professor in the Radiation Oncology department at Medical College of Wisconsin
Eric Paulson PhD Chief, Professor in the Radiation Oncology department at Medical College of Wisconsin
Christina Sarosiek Medical Physicist Assistant II in the Radiation Oncology department at Medical College of Wisconsin
MESH terms used to index this publication - Major topics in bold
HumansImage Processing, Computer-Assisted
Machine Learning
Magnetic Resonance Imaging
Tomography, X-Ray Computed
