Surgical Phase and Instrument Recognition: How to identify appropriate Dataset Splits

作     者：Kostiuchik, Georgii Sharan, Lalith Mayer, Benedikt Wolf, Ivo Preim, Bernhard Engelhardt, Sandy 

作者机构：Department of Cardiac Surgery Heidelberg University Hospital Heidelberg Germany  Partner Site Heidelberg Mannheim Germany Department of Simulation and Graphics University of Magdeburg Magdeburg Germany Department of Computer Science Mannheim University of Applied Sciences Mannheim Germany 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2023年

核心收录：

主　　题：Data visualization 

摘      要：Purpose: Machine learning approaches can only be reliably evaluated if training, validation and test data splits are representative and not affected by the absence of classes that are of interest. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, the issue becomes difficult as sub-properties that help to define phases, like instrument (co-)occurrence, are usually not particularly considered when defining the split. We argue that such sub-properties must be equally considered. Methods: We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits for surgical workflow recognition, especially regarding identification of sub-optimal dataset splits. Results: To validate the dedicated interactive visualizations, we performed analysis of the common Cholec80 dataset splits using the proposed application. We were able to uncover phase transitions, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements of the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks using the proposed application. Conclusion: In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split. Our interactive data visualization tool presents a promising approach for the assessment of dataset splits for the tasks of surgical phase and instrument recognition. Evaluat