检索结果-内蒙古大学图书馆

AN INTELLIGENT DECISION-MAKING SYSTEM BASED ON MULTIPLE CLASSIFIERS UPDATED USING CONFIDENCE RATES AND STRESS PARAMETERS

学校读者我要写书评

暂无评论

CONTROL AND INTELLIGENT systems 2011年第4期39卷 213-223页

作者： Hamdani, Tarek M. Khabou, Mohamed A. Alimi, Adel M. Karray, Fakhri REGIM: Research Group on Intelligent Machines University of Sfax National School of Engineers (ENIS Sfax 3038 BP 1173 Tunisia Electrical and Computer Engineering Dept University of West Florida 11000 University Pkwy Pensacola FL 32514 United States Pattern Analysis and Machine Intelligence Research Group Department of Electrical and Computer Engineering University of Waterloo Waterloo ON N2L 3G1 Canada

We develop an intelligent multi-classifier decision-making system for multi-class classification tasks. The proposed system called I-MDS (Intelligent Multiple Decision System) uses a dynamic scheme to combine the information provided by the individual classifiers and make a classification decision. The individual classifiers in the system are interconnected and use a negotiation scheme to come up with a unified decision. During the interactive and reactive negotiation process, individual classifiers are allowed to revaluate their confidence in their individual decisions and to respond to a system-wise stress parameter that keeps increasing as long as the system does not reach a unified decision. If after a certain number of negotiation rounds the system can not reach a unified decision, the input pattern is rejected. The proposed systems were tested on multi-class classification problems from the UCI repository and were shown to produce better classification rates and fewer misclassifications than majority voting combination technique.

关键词： pattern recognition multiple classifier systems multi-agent design negotiation confidence rate stress rate

Physics-Driven Spectrum-Consistent Federated Learning for Palmprint Verification

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yang, Ziyuan Teoh, Andrew Beng Jin Zhang, Bob Leng, Lu Zhang, Yi College of Computer Science Sichuan University Chengdu China School of Electrical and Electronic Engineering College of Engineering Yonsei University Seoul Korea Republic of Pattern Analysis and Machine Intelligence Group Department of Computer and Information Science University of Macau Macau China School of Software Nanchang Hangkong University Nanchang China School of Cyber Science and Engineering Sichuan University Chengdu China

Palmprint as biometrics has gained increasing attention recently due to its discriminative ability and robustness. However, existing methods mainly improve palmprint verification within one spectrum, which is challenging to verify across different spectrums. Additionally, in distributed server-client-based deployment, palmprint verification systems predominantly necessitate clients to transmit private data for model training on the centralized server, thereby engendering privacy apprehensions. To alleviate the above issues, in this paper, we propose a physics-driven spectrum-consistent federated learning method for palmprint verification, dubbed as PSFed-Palm. PSFed-Palm draws upon the inherent physical properties of distinct wavelength spectrums, wherein images acquired under similar wavelengths display heightened resemblances. Our approach first partitions clients into short- and long-spectrum groups according to the wavelength range of their local spectrum images. Subsequently, we introduce anchor models for short- and long-spectrum, which constrain the optimization directions of local models associated with long- and short-spectrum images. Specifically, a spectrum-consistent loss that enforces the model parameters and feature representation to align with their corresponding anchor models is designed. Finally, we impose constraints on the local models to ensure their consistency with the global model, effectively preventing model drift. This measure guarantees spectrum consistency while protecting data privacy, as there is no need to share local data. Extensive experiments are conducted to validate the efficacy of our proposed PSFed-Palm approach. The proposed PSFed-Palm demonstrates compelling performance despite only a limited number of training data. The codes will be released at https://***/Zi-YuanYang/PSFed-Palm. Copyright © 2023, The Authors. All rights reserved.

关键词： Biometrics

A Dual-Level Cancelable Framework for Palmprint Verification and Hack-Proof Data Storage

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yang, Ziyuan Kang, Ming Teoh, Andrew Beng Jin Gao, Chengrui Chen, Wen Zhang, Bob Zhang, Yi The College of Computer Science Key Laboratory of Data Protection and Intelligent Management Ministry of Education Sichuan University Chengdu610065 China The School of Cyber Science and Engineering Sichuan University Chengdu610065 China The School of Electrical and Electronic Engineering College of Engineering Yonsei University Seoul Korea Republic of The College of Computer Science Sichuan University Chengdu610065 China The Pattern Analysis and Machine Intelligence Group Department of Computer and Information Science University of Macau Taipa China The School of Cyber Science and Engineering Key Laboratory of Data Protection and Intelligent Management Ministry of Education Sichuan University Chengdu610065 China

In recent years, palmprints have been widely used for individual verification. The rich privacy information in palmprint data necessitates its protection to ensure security and privacy without sacrificing system performance. Existing systems often use cancelable technologies to protect templates, but these technologies ignore the potential risk of data leakage. Upon breaching the system and gaining access to the stored database, a hacker could easily manipulate the stored templates, compromising the security of the verification system. To address this issue, we propose a dual-level cancelable palmprint verification framework in this paper. Specifically, the raw template is initially encrypted using a competition hashing network with a first-level token, facilitating the end-to-end generation of cancelable templates. Different from previous works, the protected template undergoes further encryption to differentiate the second-level protected template from the first-level one. The system specifically creates a negative database (NDB) with the second-level token for dual-level protection during the enrollment stage. Reversing the NDB is NP-hard and a fine-grained algorithm for NDB generation is introduced to manage the noise and specified bits. During the verification stage, we propose an NDB matching algorithm based on matrix operation to accelerate the matching process of previous NDB methods caused by dictionary-based matching rules. This approach circumvents the need to store templates identical to those utilized for verification, reducing the risk of potential data leakage. Extensive experiments conducted on public palmprint datasets have confirmed the effectiveness and generality of the proposed framework. Upon acceptance of the paper, the code will be accessible at https://***/Deep-Imaging-group/NPR. Copyright © 2024, The Authors. All rights reserved.

关键词： Digital storage

Continual Learning in the Presence of Repetition

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hemati, Hamed Pellegrini, Lorenzo Duan, Xiaotian Zhao, Zixuan Xia, Fangfang Masana, Marc Tscheschner, Benedikt Veas, Eduardo Zheng, Yuxiang Zhao, Shiji Li, Shao-Yuan Huang, Sheng-Jun Lomonaco, Vincenzo van de Ven, Gido M. Institute for Computer Science University of St. Gallen Rosenbergstrasse 30 St. Gallen9000 Switzerland Department of Computer Science University of Bologna Via dell’Università 50 Cesena47521 Italy The University of Chicago 5801 S Ellis Ave Chicago60637 United States Argonne National Laboratory 9700 S Cass Ave Lemont60439 United States Graz University of Technology Rechbauerstraße 12 Graz8010 Austria TU Graz - SAL Dependable Embedded Systems Lab Silicon Austria Labs Graz8010 Austria Know-Center GmbH Sandgasse 36/4 Graz8010 Austria MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing University of Aeronautics and Astronautics Nanjing211106 China Department of Computer Science University of Pisa Piano Secondo Largo Bruno Pontecorvo 3 Pisa56127 Italy Department of Electrical Engineering KU Leuven Kasteelpark Arenberg 10 Leuven3001 Belgium

Continual learning (CL) provides a framework for training models in ever-evolving environments. Although re-occurrence of previously seen objects or tasks is common in real-world problems, the concept of repetition in the data stream is not often considered in standard benchmarks for CL. Unlike with the rehearsal mechanism in buffer-based strategies, where sample repetition is controlled by the strategy, repetition in the data stream naturally stems from the environment. This report provides a summary of the CLVision challenge at CVPR 2023, which focused on the topic of repetition in class-incremental learning. The report initially outlines the challenge objective and then describes three solutions proposed by finalist teams that aim to effectively exploit the repetition in the stream to learn continually. The experimental results from the challenge highlight the effectiveness of ensemble-based solutions that employ multiple versions of similar modules, each trained on different but overlapping subsets of classes. This report underscores the transformative potential of taking a different perspective in CL by employing repetition in the data stream to foster innovative strategy design. © 2024, CC BY.

关键词： Contrastive Learning

Deep ROC analysis and AUC as balanced average accuracy to improve model selection, understanding and interpretation

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Carrington, André M. Manuel, Douglas G. Fieguth, Paul W. Ramsay, Tim Osmani, Venet Wernly, Bernhard Bennett, Carol Hawken, Steven McInnes, Matthew Magwood, Olivia Sheikh, Yusuf Holzinger, Andreas Ottawa Hospital Research Institute Institute of Clinical Evaluative Sciences Ottawa Canada Ottawa Hospital Research Institute Department of Family Medicine School of Epidemiology and Public Health University of Ottawa Institute of Clinical Evaluative Sciences Bruyère Research Institute Canada Department of Systems Design Engineering Vision and Image Processing Lab Faculty of Engineering University of Waterloo Canada e-health group Fondazione Bruno Kessler Research Institute department of Psychology and Cognitive Science University of Trento Italy Department of Cardiology Paracelsus Medical University of Salzburg Salzburg Austria Ottawa Hospital Research Institute University of Ottawa Canada Ottawa Hospital Research Institute University of Ottawa Canada Ottawa Hospital Research Institute University of Ottawa Canada Ottawa Hospital Research Institute Institute of Clinical Evaluative Sciences Ottawa Canada Bruyere Research Institute University of Ottawa Canada Ottawa Hospital Research Institute Department of Biology University of Ottawa Canada Alberta Machine Intelligence Institute University of Alberta Canada Human-Centered AI Lab Medical University Graz Austria

Optimal performance is critical for decision-making tasks from medicine to autonomous driving, however common performance measures may be too general or too specific. For binary classifiers, diagnostic tests or prognosis at a timepoint, measures such as the area under the receiver operating characteristic curve, or the area under the precision recall curve, are too general because they include unrealistic decision thresholds. On the other hand, measures such as accuracy, sensitivity or the F1 score are measures at a single threshold that reflect an individual single probability or predicted risk, rather than a range of individuals or risk. We propose a method in between, deep ROC analysis, that examines groups of probabilities or predicted risks for more insightful analysis. We translate esoteric measures into familiar terms: AUC and the normalized concordant partial AUC are balanced average accuracy (a new finding);the normalized partial AUC is average sensitivity;and the normalized horizontal partial AUC is average specificity. Along with post-test measures, we provide a method that can improve model selection in some cases and provide interpretation and assurance for patients in each risk group. We demonstrate deep ROC analysis in two case studies and provide a toolkit in Python. © 2021, CC BY-NC-SA.

关键词： Decision making

Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Bassi, Pedro R.A.S. Li, Wenxuan Tang, Yucheng Isensee, Fabian Wang, Zifu Chen, Jieneng Chou, Yu-Cheng Roy, Saikat Kirchhoff, Yannick Rokuss, Maximilian Huang, Ziyan Ye, Jin He, Junjun Wald, Tassilo Ulrich, Constantin Baumgartner, Michael Maier-Hein, Klaus H. Jaeger, Paul Ye, Yiwen Xie, Yutong Zhang, Jianpeng Chen, Ziyang Xia, Yong Xing, Zhaohu Zhu, Lei Sadegheih, Yousef Bozorgpour, Afshin Kumari, Pratibha Azad, Reza Merhof, Dorit Shi, Pengcheng Ma, Ting Du, Yuxin Bai, Fan Huang, Tiejun Zhao, Bo Wang, Haonan Li, Xiaomeng Gu, Hanxue Dong, Haoyu Yang, Jichen Mazurowski, Maciej A. Gupta, Saumya Wu, Linshan Zhuang, Jiaxin Chen, Hao Roth, Holger Xu, Daguang Blaschko, Matthew B. Decherchi, Sergio Cavalli, Andrea Yuille, Alan L. Zhou, Zongwei Department of Computer Science Johns Hopkins University United States Department of Pharmacy and Biotechnology University of Bologna Italy Center for Biomolecular Nanotechnologies Istituto Italiano di Tecnologia Italy NVIDIA United States Germany Germany ESAT-PSI KU Leuven Belgium Faculty of Mathematics and Computer Science Heidelberg University Germany HIDSS4Health - Helmholtz Information and Data Science School for Health Germany Shanghai Jiao Tong University China Shanghai Artificial Intelligence Laboratory China Pattern Analysis and Learning Group Department of Radiation Oncology Heidelberg University Hospital Germany DKFZ Germany School of Computer Science and Engineering Northwestern Polytechnical University China Australian Institute for Machine Learning The University of Adelaide Australia College of Computer Science and Technology Zhejiang University China Hong Kong University of Science and Technology Guangzhou China Hong Kong University of Science and Technology Hong Kong Faculty of Informatics and Data Science University of Regensburg Germany Faculty of Electrical Engineering and Information Technology RWTH Aachen University Germany Fraunhofer Institute for Digital Medicine MEVIS Germany Electronic & Information Engineering School Harbin Institute of Technology Shenzhen China China The Chinese University of Hong Kong Hong Kong Peking University China Department of Electrical and Computer Engineering Duke University United States Stony Brook University United States Department of Computer Science and Engineering Department of Chemical and Biological Engineering Division of Life Science Hong Kong University of Science and Technology Hong Kong Data Science and Computation Facility Fondazione Istituto Italiano di Tecnologia Italy Ecole Polytechnique Fédérale de Lausanne Switzerland

How can we test AI performance? This question seems trivial, but it isn’t. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms. In addition, we also evaluated pre-existing AI frameworks—which, differing from algorithms, are more flexible and can support different algorithms—including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain. Copyright © 2024, The Authors. All rights reserved.

关键词： Open systems

The Liver Tumor Segmentation Benchmark (LiTS)

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Bilic, Patrick Christ, Patrick Li, Hongwei Bran Vorontsov, Eugene Ben-Cohen, Avi Kaissis, Georgios Szeskin, Adi Jacobs, Colin Mamani, Gabriel Efrain Humpire Chartrand, Gabriel Lohöfer, Fabian Holch, Julian Walter Sommer, Wieland Hofmann, Felix Hostettler, Alexandre Lev-Cohain, Naama Drozdzal, Michal Amitai, Michal Marianne Vivanti, Refael Sosna, Jacob Ezhov, Ivan Sekuboyina, Anjany Navarro, Fernando Kofler, Florian Paetzold, Johannes C. Shit, Suprosanna Hu, Xiaobin Lipková, Jana Rempfler, Markus Piraud, Marie Kirschke, Jan Wiestler, Benedikt Zhang, Zhiheng Hülsemeyer, Christian Beetz, Marcel Ettlinger, Florian Antonelli, Michela Bae, Woong Bellver, Míriam Bi, Lei Chen, Hao Chlebus, Grzegorz Dam, Erik B. Dou, Qi Fu, Chi-Wing Georgescu, Bogdan Giró-I-Nieto, Xavier Gruen, Felix Han, Xu Heng, Pheng-Ann Hesser, Jürgen Moltz, Jan Hendrik Igel, Christian Isensee, Fabian Jäger, Paul Jia, Fucang Kaluva, Krishna Chaitanya Khened, Mahendra Kim, Ildoo Kim, Jae-Hun Kim, Sungwoong Kohl, Simon Konopczynski, Tomasz Kori, Avinash Krishnamurthi, Ganapathy Li, Fan Li, Hongchao Li, Junbo Li, Xiaomeng Lowengrub, John Ma, Jun Maier-Hein, Klaus Maninis, Kevis-Kokitsi Meine, Hans Merhof, Dorit Pai, Akshay Perslev, Mathias Petersen, Jens Pont-Tuset, Jordi Qi, Jin Qi, Xiaojuan Rippel, Oliver Roth, Karsten Sarasua, Ignacio Schenk, Andrea Shen, Zengming Torres, Jordi Wachinger, Christian Wang, Chunliang Weninger, Leon Wu, Jianrong Xu, Daguang Yang, Xiaoping Yu, Simon Chun-Ho Yuan, Yading Yue, Miao Zhang, Liping Cardoso, Jorge Bakas, Spyridon Braren, Rickmer Heinemann, Volker Pal, Christopher Tang, An Kadoury, Samuel Soler, Luc van Ginneken, Bram Greenspan, Hayit Joskowicz, Leo Menze, Bjoern Department of Informatics Technical University of Munich Germany Department of Quantitative Biomedicine University of Zurich Switzerland Ecole Polytechnique de Montréal Canada Department of Medical Imaging Radboud University Medical Center Nijmegen Netherlands Department of Biomedical Engineering Tel-Aviv University Israel Germany Pattern Analysis and Learning Group Department of Radiation Oncology Heidelberg University Hospital Heidelberg Germany Philips Research China Philips China Innovation Campus Shanghai China School of Biomedical Engineering & Imaging Sciences King0s College London London United Kingdom Institute for AI in Medicine Technical University of Munich Germany Department of Computer Science Guangdong University of Foreign Studies China Institute for diagnostic and interventional radiology Klinikum rechts der Isar Technical University of Munich Germany Institute for Diagnostic and Interventional Neuroradiology Klinikum Rechts der Isar Technical University of Munich Germany Department of Hepatobiliary Surgery The Affiliated Drum Tower Hospital Nanjing University Medical School China Department of Computing Imperial College London London United Kingdom Institute for Tissue Engineering and Regenerative Medicine Helmholtz Zentrum München Neuherberg Germany Brigham and Women's Hospital Harvard Medical School United States School of Computer Science and Engineering The Hebrew University of Jerusalem Israel University of Pennsylvania PA United States Pte. Ltd. Singapore Medical Imaging and Reconstruction Lab Department of Engineering Design Indian Institute of Technology Madras India Sensetime Shanghai China Department of Radiology Perelman School of Medicine University of Pennsylvania United States Department of Pathology and Laboratory Medicine Perelman School of Medicine University of Pennsylvania PA United States Co. Ltd China MontréalQC Canada Department of Radiology Radiation Oncology and Nuclear Medicine University of Montréal

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://***/. In addition, both data and online evaluation are accessible via ***. © 2019, CC BY-NC-ND.

关键词： Deep learning

Why is the Winner the Best?

学校读者我要写书评

暂无评论

Why is the Winner the Best?

Conference on Computer Vision and pattern Recognition (CVPR)

作者： M. Eisenmann A. Reinke V. Weru M. D. Tizabi F. Isensee T. J. Adler S. Ali V. Andrearczyk M. Aubreville U. Baid S. Bakas N. Balu S. Bano J. Bernal S. Bodenstedt A. Casella V. Cheplygina M. Daum M. De Bruijne A. Depeursinge R. Dorent J. Egger D. G. Ellis S. Engelhardt M. Ganz N. Ghatwary G. Girard P. Godau A. Gupta L. Hansen K. Harada M. Heinrich N. Heller A. Hering A. Huaulmé P. Jannin A. E. Kavur O. Kodym M. Kozubek J. Li H. Li J. Ma C. Martín-Isla B. Menze A. Noble V. Oreiller N. Padoy S. Pati K. Payette T. Rädsch J. Rafael-Patiño V. Singh Bawa S. Speidel C. H. Sudre K. Van Wijnen M. Wagner D. Wei A. Yamlahi M. H. Yap C. Yuan M. Zenk A. Zia D. Zimmerer D. Aydogan B. Bhattarai L. Bloch R. Brüngel J. Cho C. Choi Q. Dou I. Ezhov C. M. Friedrich C. Fuller R. R. Gaire A. Galdran Á. García Faura M. Grammatikopoulou S. Hong M. Jahanifar I. Jang A. Kadkhodamohammadi I. Kang F. Kofler S. Kondo H. Kuijf M. Li M. Luu T. Martinčič P. Morais M. A. Naser B. Oliveira D. Owen S. Pang J. Park S. Park S. Płotka E. Puybareau N. Rajpoot K. Ryu N. Saeed A. Shephard P. Shi D. Štepec R. Subedi G. Tochon H. R. Torres H. Urien J. L. Vilaça K. A. Wahid H. Wang J. Wang L. Wang X. Wang B. Wiestler M. Wodzinski F. Xia J. Xie Z. Xiong S. Yang Y. Yang Z. Zhao K. Maier-Hein P. F. Jäger A. Kopp-Schneider L. Maier-Hein Division of Intelligent Medical Systems German Cancer Research Center (DKFZ) Heidelberg Germany Helmholtz Imaging German Cancer Research Center (DKFZ) Heidelberg Germany Faculty of Mathematics and Computer Science Heidelberg University Heidelberg Germany Division of Biostatistics German Cancer Research Center (DKFZ) Heidelberg Germany Division of Medical Image Computing German Cancer Research Center (DKFZ) Heidelberg Germany Faculty of Engineering and Physical Sciences School of Computing University of Leeds Leeds UK Institute of Informatics School of Management HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland Sierre Switzerland Department of Nuclear Medicine and Molecular Imaging Lausanne University Hospital Lausanne Switzerland Technische Hochschule Ingolstadt Ingolstadt Germany Center for Artificial Intelligence and Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA) University of Pennsylvania Philadelphia PA USA Department of Pathology and Laboratory Medicine Perelman School of Medicine University of Pennsylvania Philadelphia PA USA Department of Radiology Perelman School of Medicine University of Pennsylvania Philadelphia PA USA Department of Radiology University of Washington Seattle WA USA Department of Computer Science Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) University College London London UK Universitat Autònoma de Barcelona & Computer Vision Center Barcelona Spain Division of Translational Surgical Oncology National Center for Tumor Diseases (NCT/UCC) Dresden Dresden Germany Department of Advanced Robotics Istituto Italiano di Tecnologia Italy Department of Electronics Information and Bioengineering Politecnico di Milano Milan Italy IT University of Copenhagen Copenhagen Denmark Department of General Visceral and Transplantation Surgery Heidelberg University Hospital Heidelberg Germany Department of Radiology and Nuc

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multicenter study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and post-processing (66%). The “typical” lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

关键词：

Understanding metric-related pitfalls in image analysis validation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Reinke, Annika Tizabi, Minu D. Baumgartner, Michael Eisenmann, Matthias Heckmann-Nötzel, Doreen Kavur, A. Emre Rädsch, Tim Sudre, Carole H. Acion, Laura Antonelli, Michela Arbel, Tal Bakas, Spyridon Benis, Arriel Blaschko, Matthew B. Buettner, Florian Cardoso, M. Jorge Cheplygina, Veronika Chen, Jianxu Christodoulou, Evangelia Cimini, Beth A. Collins, Gary S. Farahani, Keyvan Ferrer, Luciana Galdran, Adrian van Ginneken, Bram Glocker, Ben Godau, Patrick Haase, Robert Hashimoto, Daniel A. Hoffman, Michael M. Huisman, Merel Isensee, Fabian Jannin, Pierre Kahn, Charles E. Kainmueller, Dagmar Kainz, Bernhard Karargyris, Alexandros Karthikesalingam, Alan Kenngott, Hannes Kleesiek, Jens Kofler, Florian Kooi, Thijs Kopp-Schneider, Annette Kozubek, Michal Kreshuk, Anna Kurc, Tahsin Landman, Bennett A. Litjens, Geert Madani, Amin Maier-Hein, Klaus Martel, Anne L. Mattson, Peter Meijering, Erik Menze, Bjoern Moons, Karel G.M. Müller, Henning Nichyporuk, Brennan Nickel, Felix Petersen, Jens Rafelski, Susanne M. Rajpoot, Nasir Reyes, Mauricio Riegler, Michael A. Rieke, Nicola Saez-Rodriguez, Julio Sánchez, Clara I. Shetty, Shravya Summers, Ronald M. Taha, Abdel A. Tiulpin, Aleksei Tsaftaris, Sotirios A. van Calster, Ben Varoquaux, Gaël Yaniv, Ziv R. Jäger, Paul F. Maier-Hein, Lena Faculty of Mathematics and Computer Science Heidelberg University Heidelberg Germany Heidelberg Division of Intelligent Medical Systems Germany NCT Heidelberg A Partnership Between DKFZ University Medical Center Heidelberg Germany Heidelberg Division of Medical Image Computing Germany Heidelberg Division of Intelligent Medical Systems Germany MRC Unit for Lifelong Health and Ageing UCL Centre for Medical Image Computing Department of Computer Science University College London London United Kingdom School of Biomedical Engineering and Imaging Science King’s College London London United Kingdom Instituto de Cálculo CONICET – Universidad de Buenos Aires Buenos Aires Argentina Centre for Medical Image Computing University College London London United Kingdom McGill University Montreal Canada Division of Computational Pathology Dept of Pathology & Laboratory Medicine Indiana University School of Medicine IU Health Information and Translational Sciences Building Indianapolis United States University of Pennsylvania Richards Medical Research Laboratories FL7 PhiladelphiaPA United States Department of Digital Medical Technologies Holon Institute of Technology Holon Israel European Federation for Medical Informatics Le Mont-sur-Lausanne Switzerland Center for Processing Speech and Images Department of Electrical Engineering KU Leuven Leuven Belgium partner site Frankfurt/Mainz a partnership between DKFZ and UCT Frankfurt Marburg Germany Heidelberg Germany Goethe University Frankfurt Department of Medicine Germany Goethe University Frankfurt Department of Informatics Germany and Frankfurt Cancer Insititute Germany Department of Computer Science IT University of Copenhagen Copenhagen Denmark Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V. Dortmund Germany Imaging Platform Broad Institute of MIT and Harvard CambridgeMA United States Centre for Statistics in Medicine University of Oxford Oxford United Kingdom Center for Biomedical In

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately. While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. While focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation. © 2023, CC BY.

关键词： Computer vision