检索结果-内蒙古大学图书馆

Self-supervised multi-modality learning for multi-label skin lesion classification

COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2025年 265卷 108729-108729页

作者： Wang, Hao Ahn, Euijoon Bi, Lei Kim, Jinman Univ Sydney Fac Engn Sch Comp Sci Sydney NSW 2006 Australia James Cook Univ Coll Sci & Engn Cairns Qld 4870 Australia Shanghai Jiao Tong Univ Inst Translat Med Natl Ctr Translat Med Shanghai Peoples R China

Background: The clinical diagnosis of skin lesions involves the analysis of dermoscopic and clinical modalities. Dermoscopic images provide detailed views of surface structures, while clinical images offer complementary macroscopic information. Clinicians frequently use the seven-point checklist as an auxiliary tool for melanoma diagnosis and identifying lesion attributes. Supervised deep learning approaches, such as convolutional neural networks, have performed well using dermoscopic and clinical modalities (multi-modality) and further enhanced classification by predicting seven skin lesion attributes (multi-label). However, the performance of these approaches is reliant on the availability of large-scale labeled data, which are costly and time-consuming to obtain, more so with annotating multi-attributes Methods: To reduce the dependency on large labeled datasets, we propose a self-supervised learning (SSL) algorithm for multi-modality multi-label skin lesion classification. Compared with single-modality SSL, our algorithm enables multi-modality SSL by maximizing the similarities between paired dermoscopic and clinical images from different views. We introduce a novel multi-modal and multi-label SSL strategy that generates surrogate pseudo-multi-labels for seven skin lesion attributes through clustering analysis. A label-relation-aware module is proposed to refine each pseudo-label embedding, capturing the interrelationships between pseudo-multi-labels. We further illustrate the interrelationships of skin lesion attributes and their relationships with clinical diagnoses using an attention visualization technique. Results: The proposed algorithm was validated using the well-benchmarked seven-point skin lesion dataset. Our results demonstrate that our method outperforms the state-of-the-art SSL counterparts. Improvements in the area under receiver operating characteristic curve, precision, sensitivity, and specificity were observed across various lesion attributes a

关键词： Skin lesion classification Self-supervised learning multi-label learning multi-modality learning

来源：评论

学校读者我要写书评

暂无评论

multi-modality learning for Non-Rigid 3D Shape Retrieval via Structured Sparsity Regularizations

引用

IEEE SENSORS JOURNAL 2021年第20期21卷 22985-22994页

作者： Luo, Luqing Tang, Lulu Liu, Rui Zhang, Xiaoli Yang, Zhi-Xin Univ Macau Dept Electromech Engn State Key Lab Internet Things Smart City Macau 999078 Peoples R China Kent State Univ Coll Aeronaut & Engn Cognit Robot & AI Lab CRAI Kent OH 44240 USA Colorado Sch Mines Dept Mech Engn Golden CO 80401 USA

Big challenges are usually occurring in non-rigid 3D shape retrieval, for the shapes undergoing arbitrarily non-affine transformations. In this work, a novel design of feature learning approach is proposed for non-rigid 3D shape retrieval, dubbed Structured Sparsity Regularized multi-modality Method (SSR-MM). The shape signatures which capture the deformation-invariant characteristics are averaged and stacked for a multi-modality machine learning approach, and a transform matrix based on the structure sparsity regularization is utilized to map those signatures obtaining the discriminative features for retrieval. The proposed framework is evaluated on the publicly available non-rigid 3D human benchmarks, and the experimental results show the efficacy of our contributions and the advantages of our method over existing ones.

关键词： Shape Three-dimensional displays Manifolds Kernel Sensors Strain Eigenvalues and eigenfunctions 3D shape retrieval multi-modality learning non-rigid shapes

来源：评论

学校读者我要写书评

暂无评论

Confidence-aware multi-modality learning for eye disease screening

引用

MEDICAL IMAGE ANALYSIS 2024年 96卷 103214页

作者： Zou, Ke Lin, Tian Han, Zongbo Wang, Meng Yuan, Xuedong Chen, Haoyu Zhang, Changqing Shen, Xiaojing Fu, Huazhu Sichuan Univ Natl Key Lab Fundamental Sci Synthet Vis Chengdu 610065 Peoples R China Sichuan Univ Coll Comp Sci Chengdu 610065 Peoples R China Shantou Univ Joint Shantou Int Eye Ctr Shantou 515041 Peoples R China Chinese Univ Hong Kong Shantou 515041 Peoples R China Shantou Univ Med Coll Shantou 515041 Peoples R China Tianjin Univ Coll Intelligence & Comp Tianjin 300350 Peoples R China Agcy Sci Technol & Res Inst High Performance Comp Singapore 138632 Singapore Sichuan Univ Coll Math Chengdu 610065 Peoples R China

multi -modal ophthalmic image classification plays a key role in diagnosing eye diseases, as it integrates information from different sources to complement their respective performances. However, recent improvements have mainly focused on accuracy, often neglecting the importance of confidence and robustness in predictions for diverse modalities. In this study, we propose a novel multi -modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multimodality information using a multi -distribution fusion perspective. Specifically, our method first utilizes normal inverse gamma prior distributions over pre -trained models to learn both aleatoric and epistemic uncertainty for uni-modality. Then, the normal inverse gamma distribution is analyzed as the Student's t distribution. Furthermore, within a confidence -aware fusion framework, we propose a mixture of Student's t distributions to effectively integrate different modalities, imparting the model with heavy -tailed properties and enhancing its robustness and reliability. More importantly, the confidence -aware multi -modality ranking regularization term induces the model to more reasonably rank the noisy single -modal and fused -modal confidence, leading to improved reliability and accuracy. Experimental results on both public and internal datasets demonstrate that our model excels in robustness, particularly in challenging scenarios involving Gaussian noise and modality missing conditions. Moreover, our model exhibits strong generalization capabilities to out -of -distribution data, underscoring its potential as a promising solution for multimodal eye disease screening.

关键词： Uncertainty estimation Eye disease multi-modality learning

来源：评论

学校读者我要写书评

暂无评论

INPUTMIX: A STRATEGY TO REGULARIZE AND BALANCE multi-modality AND multi-VIEW MODEL learning 49

INPUTMIX: A STRATEGY TO REGULARIZE AND BALANCE MULTI-MODALIT...

引用

49th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Wang, Junjie Nordstrom, Tomas Umea Univ Dept Appl Phys & Elect Umea Sweden RISE Res Inst Sweden Stockholm Sweden

ISBN: (纸本)9798350344868;9798350344851

Real-world perception tasks often involve multiple modalities or views of input. While joint training of multiple modality classification models has been explored previously, it has not consistently outperformed the best single modality model. This paper aims to address one of the reasons for this: the difficulty in balancing the contributions of each input in the end-to-end training of multi-input models. Additionally, the increased capacity of multi-input networks can lead to overfitting. To solve these issues, we propose InputMix, a simple yet effective method for optimally mixing different inputs. Our method mixes a certain proportion p of input pairs to relieve the increased capacity problems and assigns a weighting factor lambda for each input to generate a mixed target, allowing us to specify the contributions of each input. Experimental results on three multi-input classification tasks demonstrate that our method significantly improves the generalization performance of multi-input neural networks. Codes are available at https://***/JesseWong333/inputmix/.

关键词： multi-view learning multi-modality learning

来源：评论

学校读者我要写书评

暂无评论

Robust Visual Question Answering: Datasets, Methods, and Future Challenges

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024年第8期46卷 5575-5594页

作者： Ma, Jie Wang, Pinghui Kong, Dechen Wang, Zewei Liu, Jun Pei, Hongbin Zhao, Junzhou Jiaotong Univ Sch Cyber Sci & Engn Key Lab Intelligent Networks & Network Secur Minist Educ Xian 710049 Shaanxi Peoples R China Jiaotong Univ Sch Cyber Sci & Engn Key Lab Intelligent Networks & Network Secur Xian 710049 Shaanxi Peoples R China

Visual question answering requires a system to provide an accurate natural language answer given an image and a natural language question. However, it is widely recognized that previous generic VQA methods often tend to memorize biases present in the training data rather than learning proper behaviors, such as grounding images before predicting answers. Therefore, these methods usually achieve high in-distribution but poor out-of-distribution performance. In recent years, various datasets and debiasing methods have been proposed to evaluate and enhance the VQA robustness, respectively. This paper provides the first comprehensive survey focused on this emerging fashion. Specifically, we first provide an overview of the development process of datasets from in-distribution and out-of-distribution perspectives. Then, we examine the evaluation metrics employed by these datasets. Third, we propose a typology that presents the development process, similarities and differences, robustness comparison, and technical features of existing debiasing methods. Furthermore, we analyze and discuss the robustness of representative vision-and-language pre-training models on VQA. Finally, through a thorough review of the available literature and experimental analysis, we discuss the key areas for future research from various viewpoints.

关键词： Sports Task analysis Robustness Transformers Training Question answering (information retrieval) Knowledge engineering Vision-and-language pre-training bias learning debiasing multi-modality learning visual question answering

来源：评论

学校读者我要写书评

暂无评论

Visual-Textual Attribute learning for Class-Incremental Facial Expression Recognition

引用

IEEE TRANSACTIONS ON multiMEDIA 2024年 26卷 8038-8051页

作者： Lv, Yuanling Huang, Guangyu Yan, Yan Xue, Jing-Hao Chen, Si Wang, Hanzi Xiamen Univ Sch Informat Fujian Key Lab Sensing & Comp Smart City Xiamen 361005 Peoples R China Xiamen Univ Key Lab Multimedia Trusted Percept & Efficient Com Minist Educ China Xiamen 361005 Peoples R China UCL Dept Stat Sci London WC1E 6BT England Xiamen Univ Technol Sch Comp & Informat Engn Xiamen Peoples R China

In this paper, we study facial expression recognition (FER) in the class-incremental learning (CIL) setting, which defines the classification of well-studied and easily-accessible basic expressions as an initial task while learning new compound expressions gradually. Motivated by the fact that compound expressions are meaningful combinations of basic expressions, we treat basic expressions as attributes (i.e., semantic descriptors), and thus compound expressions are represented in terms of attributes. To this end, we propose a novel visual-textual attribute learning network (VTA-Net), mainly consisting of a textual-guided visual module (TVM) and a textual compositional module (TCM), for class-incremental FER. Specifically, TVM extracts textual-aware visual features and classifies expressions by incorporating the textual information into visual attribute learning. Meanwhile, TCM generates visual-aware textual features and predicts expressions by exploiting the dependency between textual attributes and category names of old and new expressions based on a textual compositional graph. In particular, a visual-textual distillation loss is introduced to calibrate TVM and TCM during incremental learning. Finally, the outputs from TVM and TCM are fused to make a final prediction. On the one hand, at each incremental task, the representations of visual attributes are enhanced since visual attributes are shared across old and new expressions. This increases the stability of our method. On the other hand, the textual modality, which involves rich prior knowledge of the relevance between expressions, facilitates our model to identify subtle visual distinctions between compound expressions, improving the plasticity of our method. Experimental results on both in-the-lab and in-the-wild facial expression databases show the superiority of our method against several state-of-the-art methods for class-incremental FER.

关键词： Visualization Compounds Feature extraction Task analysis Face recognition Image recognition Training Facial expression recognition Class-incremental learning multi-modality learning Attribute learning

来源：评论

学校读者我要写书评

暂无评论

Text-Driven Traffic Anomaly Detection With Temporal High-Frequency Modeling in Driving Videos

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2024年第9期34卷 8684-8697页

作者： Liang, Rongqin Li, Yuanman Zhou, Jiantao Li, Xia Shenzhen Univ Guangdong Key Lab Intelligent Informat Proc Coll Elect & Informat Engn Shenzhen Guangdong Peoples R China Univ Macau State Key Lab Internet Things Smart City Macau Peoples R China Univ Macau Dept Comp & Informat Sci Macau Peoples R China

Traffic anomaly detection (TAD) in driving videos is critical for ensuring the safety of autonomous driving and advanced driver assistance systems. Previous single-stage TAD methods primarily rely on frame prediction, making them vulnerable to interference from dynamic backgrounds induced by the rapid movement of the dashboard camera. While two-stage TAD methods appear to be a natural solution to mitigate such interference by pre-extracting background-independent features (such as bounding boxes and optical flow) using perceptual algorithms, they are susceptible to the performance of first-stage perceptual algorithms and may result in error propagation. In this paper, we introduce TTHF, a novel single-stage method aligning video clips with text prompts, offering a new perspective on traffic anomaly detection. Unlike previous approaches, the supervised signal of our method is derived from languages rather than orthogonal one-hot vectors, providing a more comprehensive representation. Further, concerning visual representation, we propose to model the high frequency of driving videos in the temporal domain. This modeling captures the dynamic changes of driving scenes, enhances the perception of driving behavior, and significantly improves the detection of traffic anomalies. In addition, to better perceive various types of traffic anomalies, we carefully design an attentive anomaly focusing mechanism that visually and linguistically guides the model to adaptively focus on the visual context of interest, thereby facilitating the detection of traffic anomalies. It is shown that our proposed TTHF achieves promising performance, outperforming state-of-the-art competitors by +5.4% AUC on the DoTA dataset and achieving high generalization on the DADA dataset.

关键词： Traffic anomaly detection multi-modality learning high frequency attention

来源：评论

学校读者我要写书评

暂无评论

multi-modality Low-Rank learning Fused First-Order and Second-Order Information for Computer-Aided Diagnosis of Schizophrenia 9th

Multi-modality Low-Rank Learning Fused First-Order and Secon...

引用

9th International Conference on Intelligence Science and Big Data Engineering (IScIDE)

作者： Li, Huijie Zhu, Qi Zhang, Rui Zhang, Daoqiang Nanjing Univ Aeronaut & Astronaut Coll Comp Sci & Technol Nanjing 211106 Peoples R China Collaborat Innovat Ctr Novel Software Technol & I Nanjing 210093 Peoples R China

ISBN: (纸本)9783030362041;9783030362034

The brain functional connectivity network (BFCN) based methods for diagnosing brain diseases have shown great advantages. At present, most BFCN construction strategies only calculate the first-order correlation between brain areas, such as the Pearson correlation coefficient method. Although the work of the low-order and high-order BFCN construction methods exists, there is very little work to integrate them, that is, to design a multi-modal BFCN feature selection and classification method to combine low-order and high-order information. This may affect the performance of brain disease diagnosis. To this end, we propose a multi-modality low-rank learning framework jointly learning first-order and second-order BFCN information and apply it to the diagnosis of schizophrenia. The proposed method not only embeds the correlation information of multi-modality data in the learning model, but also encourages the cooperation between the first-order and the second-order BFCN by combining the ideal representation term. The experimental results of the three schizophrenia datasets (totally including 168 patients and 163 normal controls) show that our proposed method achieves promising classification results in the diagnosis of schizophrenia.

关键词： multi-modality learning Low rank representation Brain functional connection network

来源：评论

学校读者我要写书评

暂无评论

Text-driven adaptation of foundation models for few-shot surgical workflow analysis

引用

INTERNATIONAL JOURNAL OF COMPUTER ASSISTED RADIOLOGY AND SURGERY 2025年 1-9页

作者： Chen, Tingxuan Yuan, Kun Srivastav, Vinkle Navab, Nassir Padoy, Nicolas Univ Strasbourg CNRS INSERM ICubeUMR7357 Strasbourg France IHU Strasbourg Strasbourg France Tech Univ Munich CAMP Munich Germany

PurposeSurgical workflow analysis is crucial for improving surgical efficiency and safety. However, previous studies rely heavily on large-scale annotated datasets, posing challenges in cost, scalability, and reliance on expert annotations. To address this, we propose Surg-FTDA (Few-shot Text-driven Adaptation), designed to handle various surgical workflow analysis tasks with minimal paired image-label *** approach has two key components. First, few-shot selection-based modality alignment selects a small subset of images and aligns their embeddings with text embeddings from the downstream task, bridging the modality gap. Second, text-driven adaptation leverages only text data to train a decoder, eliminating the need for paired image-text data. This decoder is then applied to aligned image embeddings, enabling image-related tasks without explicit image-text *** evaluate our approach on generative tasks (image captioning) and discriminative tasks (triplet recognition and phase recognition). Results show that Surg-FTDA outperforms baselines and generalizes well across downstream *** propose a text-driven adaptation approach that mitigates the modality gap and handles multiple downstream tasks in surgical workflow analysis, with minimal reliance on large annotated datasets. The code and dataset will be released in https://***/CAMMApublic/Surg-FTDA.

关键词： Foundation model Surgical data science multi-modality learning Surgical workflow analysis

来源：评论

学校读者我要写书评

暂无评论

Advancing surgical VQA with scene graph knowledge

引用

INTERNATIONAL JOURNAL OF COMPUTER ASSISTED RADIOLOGY AND SURGERY 2024年第7期19卷 1409-1417页

作者： Yuan, Kun Kattel, Manasi Lavanchy, Joel L. Navab, Nassir Srivastav, Vinkle Padoy, Nicolas Univ Strasbourg CNRS ICube UMR7357INSERM Strasbourg France IHU Strasbourg France Tech Univ Munich CAMP Munich Germany

Purpose The modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with natural language capabilities is emerging as a necessity. Our work aims to advance visual question answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question-condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model *** First, we propose a surgical scene graph-based dataset, SSG-VQA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. We then propose SSG-VQA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module, which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene *** Our comprehensive analysis shows that our SSG-VQA dataset provides a more complex, diverse, geometrically grounded, unbiased and surgical action-oriented dataset compared to existing surgical VQA datasets and SSG-VQA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex *** We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. We point out that the bottleneck of the current surgical visual question-answer model lies in learning the encoded representation rather than decoding the sequence. Our SSG-VQA

关键词： Visual question answering multi-modality learning Surgical data science

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：