检索结果-内蒙古大学图书馆

Enhanced Cleft Lip and Palate Classification Using SigLIP 2: A Comparative Study with vision Transformers and Siamese Networks

引用

APPLIED SCIENCES-BASEL 2025年第9期15卷 4766-4766页

作者： Nantha, Oraphan Sathanarugsawait, Benjaporn Praneetpolgrang, Prasong Sripatum Univ Sch Informat Technol Bangkok 10900 Thailand

This paper extends our previous work on cleft lip and/or palate (CL/P) classification, which employed vision transformers (ViTs) and Siamese neural networks. We now integrate SigLIP 2, a state-of-the-art multilingual vision-language model, for feature extraction, replacing the previously utilized BiomedCLIP. SigLIP 2 offers enhanced semantic understanding, improved localization capabilities, and multilingual support, potentially leading to more robust feature representations for CL/P classification. We hypothesize that SigLIP 2's superior feature extraction will improve the classification accuracy of CL/P types (bilateral, unilateral, and palate-only) from the UltraSuite CLEFT dataset, a collection of ultrasound video sequences capturing tongue movements during speech with synchronized audio recordings. A comparative analysis is conducted, evaluating the performance of our original ViT-Siamese network model (using BiomedCLIP) against a new model leveraging SigLIP 2 for feature extraction. Performance is assessed using accuracy, precision, recall, and F1 score, demonstrating the impact of SigLIP 2 on CL/P classification. The new model achieves statistically significant improvements in overall accuracy (86.6% vs. 82.76%) and F1 scores for all cleft types. We discuss the computational efficiency and practical implications of employing SigLIP 2 in a clinical setting, highlighting its potential for earlier and more accurate diagnosis, personalized treatment planning, and broader applicability across diverse populations. The results demonstrate the significant potential of advanced vision-language models, such as SigLIP 2, to enhance AI-powered medical diagnostics.

关键词： cleft lip and palate vision-language models few-shot learning medical image analysis AI in healthcare

来源：评论

学校读者我要写书评

暂无评论

Towards molecular structure discovery from cryo-ET density volumes via modelling auxiliary semantic prototypes

引用

BRIEFINGS IN BIOINFORMATICS 2025年第1期26卷 bbae570页

作者： Nair, Ashwin Li, Xingjian Solanki, Bhupendra Mukhopadhyay, Souradeep Jha, Ankit Uddin, Mostofa Rafid Singha, Mainak Banerjee, Biplab Xu, Min Indian Inst Sci Educ & Res Dept Data Sci Vithura 695551 Kerela India Carnegie Mellon Univ Computat Biol Dept Pittsburgh PA 15213 USA Indian Inst Technol Machine Learning & Visual Comp Lab Powai 400076 Maharashtra India Indian Inst Sci Comp Sci & Automat CV Raman Rd Bengaluru 560012 Karnataka India LNM Inst Informat Technol Comp Sci & Engn Jaipur 302031 Rajasthan India

Cryo-electron tomography (cryo-ET) is confronted with the intricate task of unveiling novel structures. General class discovery (GCD) seeks to identify new classes by learning a model that can pseudo-label unannotated (novel) instances solely using supervision from labeled (base) classes. While 2D GCD for image data has made strides, its 3D counterpart remains unexplored. Traditional methods encounter challenges due to model bias and limited feature transferability when clustering unlabeled 2D images into known and potentially novel categories based on labeled data. To address this limitation and extend GCD to 3D structures, we propose an innovative approach that harnesses a pretrained 2D transformer, enriched by an effective weight inflation strategy tailored for 3D adaptation, followed by a decoupled prototypical network. Incorporating the power of pretrained weight-inflated Transformers, we further integrate CLIP, a vision-language model to incorporate textual information. Our method synergizes a graph convolutional network with CLIP's frozen text encoder, preserving class neighborhood structure. In order to effectively represent unlabeled samples, we devise semantic distance distributions, by formulating a bipartite matching problem for category prototypes using a decoupled prototypical network. Empirical results unequivocally highlight our method's potential in unveiling hitherto unknown structures in cryo-ET. By bridging the gap between 2D GCD and the distinctive challenges of 3D cryo-ET data, our approach paves novel avenues for exploration and discovery in this domain.

关键词： cryo-electron tomography generalized class discovery vision-language models

来源：评论

学校读者我要写书评

暂无评论

Exploring the Limits of Large language models' Ability to Distinguish Between Objects

引用

APPLIED SCIENCES-BASEL 2025年第9期15卷 4620-4620页

作者： Ju, Hyeongjin Park, Incheol Nalcakan, Yagiz Jin, Youngwan Yeo, Sanghyeop Kim, Shiho Yonsei Univ Sch Integrated Technol Incheon 21983 South Korea Yonsei Univ BK21 Grad Program Intelligent Semicond Technol Incheon 21983 South Korea

This paper explores the capability of large language models (LLMs) to accurately classify objects in challenging visual scenarios, focusing on two main tasks: differentiating real objects from artificial replicas and distinguishing human figures from human-like entities (e.g., mannequins, banners). We evaluate a diverse set of vision-language models (VLMs) ranging from large-scale architectures to parameter-efficient systems across multiple question prompts designed to probe object identification, authenticity verification, and multi-object reasoning. Our experiments reveal that while many models perform reasonably well in identifying single objects, their accuracy declines substantially under more complex conditions, such as multi-object scenes or tasks requiring fine-grained judgments of authenticity. Even top-tier models exhibit noticeable performance drops from around 100% to below 15% accuracy when forced to discern real from fake items among multiple candidates, and from 100% to 83.33% accuracy on providing positional details for human-like figures. We further discuss how these performance limitations indicate gaps in current LLM-based vision systems to highlight the need for more robust spatial reasoning and attribute analysis. Our findings underscore the significance of broadening these models' multimodal understanding and refining prompts, with an eye toward improving real-world applications-from automated quality control to surveillance-where nuanced visual classification is crucial. By comparing a variety of architectures under consistent evaluation settings, this study offers insights into the barriers LLMs face when confronted with increasingly complex visual information.

关键词： largelanguage model prompt engineering vision-language models object classification

来源：评论

学校读者我要写书评

暂无评论

Enhancing open-vocabulary object detection through region-word and region-vision matching

引用

MULTIMEDIA SYSTEMS 2025年第3期31卷 1-15页

作者： Chen, Yi Wang, Chong Li, Zhehao Lin, Sunqi Xiang, Jinhui Li, Yuqi Qian, Jiangbo Ningbo Univ Fac Elect Engn & Comp Sci Ningbo 315000 Zhejiang Peoples R China

Open-vocabulary object detection (OVOD) aims to detect novel object categories beyond the training set. Existing OVOD methods have made encouraging progress by leveraging large-scale image-caption pairs and pre-trained vision-language models (VLMs). However, two main limitations exhibit: (1) The potential category-specific concepts in global captions are not fully utilized, resulting in a lack of fine-grained semantic guidance for the detector. (2) The compositional structure of multiple concepts naturally existing in image-caption pairs as represented by VLMs remains insufficiently explored, limiting the model's ability to generalize to novel category concepts. To address these limitations, we propose a novel framework called Region-Word-vision Matching (RWVM) that integrates two core modules: a Region-Word Matching (RWM) module and a Region-vision Matching (RVM) module. Our key insight is to simultaneously guide textual and visual knowledge alignment with region features to strengthen the model's understanding of complex visual scenes. Specifically, the RWM module guides fine-grained semantic aggregation by fusing local region-word matching with global image-caption matching. The RVM module leverages VLMs to capture the compositional structure of single and multiple object concepts, directly enhancing the detector's ability to learn novel category concepts. Additionally, we demonstrate that the RVM module outperforms embeddings extracted from full language models using only simplified region embeddings. Extensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the average precision (AP) for novel categories on COCO and LVIS datasets.

关键词： Open-vocabulary object detection Region-word matching vision-language models Multi-modal training

来源：评论

学校读者我要写书评

暂无评论

Universal and extensible language-vision models for organ segmentation and tumor detection from abdominal computed tomography

引用

MEDICAL IMAGE ANALYSIS 2024年 97卷 103226页

作者： Liu, Jie Zhang, Yixiao Wang, Kang Yavuz, Mehmet Can Chen, Xiaoxi Yuan, Yixuan Li, Haoliang Yang, Yang Yuille, Alan Tang, Yucheng Zhou, Zongwei City Univ Hong Kong Hong Kong Peoples R China Johns Hopkins Univ Baltimore MD 21218 USA Univ Calif San Francisco San Francisco CA USA Univ Illinois Champaign IL USA Chinese Univ Hong Kong Hong Kong Peoples R China NVIDIA Santa Clara CA USA

The advancement of artificial intelligence (AI) for organ segmentation and tumor detection is propelled by the growing availability of computed tomography (CT) datasets with detailed, per-voxel annotations. However, these AI models often struggle with flexibility for partially annotated datasets and extensibility for new classes due to limitations in the one-hot encoding, architectural design, and learning scheme. To overcome these limitations, we propose a universal, extensible framework enabling a single model, termed Universal Model, to deal with multiple public datasets and adapt to new classes (e.g., organs/tumors). Firstly, we introduce a novel language-driven parameter generator that leverages language embeddings from large language models, enriching semantic encoding compared with one-hot encoding. Secondly, the conventional output layers are replaced with lightweight, class-specific heads, allowing Universal Model to simultaneously segment 25 organs and six types of tumors and ease the addition of new classes. We train our Universal Model on 3410 CT volumes assembled from 14 publicly available datasets and then test it on 6173 CT volumes from four external datasets. Universal Model achieves first place on six CT tasks in the Medical Segmentation Decathlon (MSD) public leaderboard and leading performance on the Beyond The Cranial Vault (BTCV) dataset. In summary, Universal Model exhibits remarkable computational efficiency (6x faster than other dataset-specific models), demonstrates strong generalization across different hospitals, transfers well to numerous downstream tasks, and more importantly, facilitates the extensibility to new classes while alleviating the catastrophic forgetting of previously learned classes. Codes, models, and datasets are available at https://***/ljwztc/CLIPDriven-Universal-Model.

关键词： Organ segmentation Tumor detection vision-language models

来源：评论

学校读者我要写书评

暂无评论

Extracting Sparse Specialist models from Generalist models

Extracting Sparse Specialist Models from Generalist Models

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Yu, Tao Zhao, Xu An, Yongqi Zhu, Guibo Tang, Ming Wang, Jinqiao Institute of Automation Chinese Academy of Sciences Beijing China School of Artificial Intelligence University of Chinese Academy of Sciences Beijing China Wuhan AI Research Wuhan China

ISBN: (纸本)9798350368741

Recently, several generalist models such as Contrastive language Image Pre-training (CLIP) have demonstrated their capabilities of performing diverse downstream tasks through zero-shot or few-shot guidance. When these generalist models are used for the specific downstream task where only a fraction of features is relevant, they would suffer from a significant redundancy of parameters. While existing methods aim to achieve sparsity and specialization, they often require additional training and large datasets. In this paper, we propose a novel framework to extract a sparse specialist model from a generalist model using only few-shot samples, without any training. Our task-specific pruning framework defines task relevance metrics and employs weighted layer-wise pruning, preserving relevant features while removing redundancies. Experiments show that our method maintains nearly identical zero-shot accuracy compared to the original generalist models at 30% sparsity, with only minimal decline at 50%. © 2025 IEEE.

关键词： Network Compression Network Pruning vision-language models

来源：评论

学校读者我要写书评

暂无评论

Diffusion models are Zero-Shot Generative Text-vision Retrievers

Diffusion Models are Zero-Shot Generative Text-Vision Retrie...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Li, Bao Xie, Zeke Zhang, Xiaomei Zhu, Xiangyu Lei, Zhen MAIS Institute of Automation Chinese Academy of Sciences Beijing China School of Artifical Intelligence University of Chinese Academy of Sciences Beijing China Guangzhou China CAIR HKISI Chinese Academy of Sciences Hong Kong

ISBN: (纸本)9798350368741

Large-scale text-to-image diffusion models have demonstrated impressive capabilities for downstream tasks by leveraging strong vision-language alignment from generative pre-training. Recently, a number of works have explored how to use the power of text-to-image diffusion models for text-image matching tasks. While previous generative text-image matching methods have shown potential in retrieving the most challenging candidates, they still suffer from extremely slow retrieval speeds and lack the ability to handle temporal dimensions, making them impractical for text-video retrieval. In this paper, we propose ZSGenRet, a simple yet effective zero-shot generative text-vision retrieval framework for both text-image and text-video tasks, based on pre-trained text-to-image diffusion models. We further incorporate inversion saliency detection to identify key frames in videos and enhance the semantic representation of the vision encoder. Experimental results demonstrate that ZSGenRet significantly improves text-video retrieval performance and achieves competitive results on text-image retrieval while remarkably improving efficiency. To the best of our knowledge, the proposed ZSGenRet is the first to explore zero-shot text-video retrieval based on diffusion models. © 2025 IEEE.

关键词： diffusion models Text-vision retrieval vision-language models zero-shot learning

来源：评论

学校读者我要写书评

暂无评论

KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration

KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driv...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Li, Chengyuan Zhou, Suyang Kong, Jieping Qi, Lei Xue, Hui College of Software Engineering Southeast University Nanjing China School of Computer Science and Engineering Southeast University Nanjing China

ISBN: (纸本)9798350368741

Zero-shot anomaly detection (ZSAD) identifies anomalies without needing training samples from the target dataset, essential for scenarios with privacy concerns or limited data. vision-language models like CLIP show potential in ZSAD but have limitations: relying on manually crafted fixed textual descriptions or anomaly prompts is time-consuming and prone to semantic ambiguity, and CLIP struggles with pixel-level anomaly segmentation, focusing more on global semantics than local details. To address these limitations, We introduce KAnoCLIP, a novel ZSAD framework that leverages vision-language models. KAnoCLIP combines general knowledge from a Large language Model (GPT-3.5) and fine-grained, image-specific knowledge from a Visual Question Answering system (Llama3) via Knowledge-Driven Prompt Learning (KnPL). KnPL uses a knowledge-driven (KD) loss function to create learnable anomaly prompts, removing the need for fixed text prompts and enhancing generalization. KAnoCLIP includes the CLIP visual encoder with V-V attention (CLIP-VV), Bi-Directional Cross-Attention for Multi-Level Cross-Modal Interaction (Bi-CMCI), and Conv-Adapter. These components preserve local visual semantics, improve local cross-modal fusion, and align global visual features with textual information, enhancing pixel-level anomaly detection. KAnoCLIP achieves state-of-the-art performance in ZSAD across 12 industrial and medical datasets, demonstrating superior generalization compared to existing methods. © 2025 IEEE.

关键词： Prompt Learning vision-language models Zero-shot Anomaly Detection

来源：评论

学校读者我要写书评

暂无评论

Adversarial Attacks on vision-language Model-Empowered Chatbots in Consumer Electronics

引用

IEEE TRANSACTIONS ON CONSUMER ELECTRONICS 2024年第3期70卷 6075-6083页

作者： Shang, Yingjia Liu, Zhijun Kang, Jiawen Hossain, M. Shamim Wu, Yi Heilongjiang Univ Sch Data Sci & Technol Harbin 150080 Peoples R China Guangdong Univ Technol Sch Automat Guangzhou 510006 Peoples R China King Saud Univ Coll Comp & Informat Sci Dept Software Engn Riyadh 13272 Saudi Arabia

Artificial Intelligence-Generated Content (AIGC) technology has revolutionized content creation, distribution, and engagement in the consumer electronics sector, propelling its applications to unprecedented heights. Within this landscape, AIGC-driven conversational agents, exemplified by renowned chatbots like ChatGPT, have gained widespread popularity globally. These advanced conversational agents are instrumental in significantly enhancing user efficiency and overall experience within consumer electronics applications. However, with the increasing integration of vision-language models (VLMs, a representative of AIGC) in consumer electronics, the vulnerability of chatbots to adversarial attacks has become a critical concern. This paper investigates and analyzes the susceptibility of VLMs-empowered chatbots to adversarial manipulation, particularly in the context of consumer electronics applications. The study employs a comprehensive approach, combining vision and language modalities, to explore potential attack vectors and vulnerabilities. Specifically, we designed three adversarial attacks, which all exploited the insufficient alignment of VLMs on multi-modal data to implement effective attacks and make chatbots output harmful content. A series of experiments demonstrate the efficacy of adversarial attacks on three popular chatbot systems, revealing vulnerabilities that may compromise the reliability and security of these systems in real-world scenarios. The findings emphasize the importance of robust defenses against adversarial attacks in VLMs-driven chatbots, urging the development of enhanced security measures to safeguard users and prevent malicious exploitation of consumer electronics. Our data and code are available at https://***/yxc0731/VLM-Adversarial-Attacks.

关键词： Consumer electronics Chatbots Security Visualization Artificial intelligence Media Data models AIGC consumer electronics vision-language models multimodal attacks adversarial attacks

来源：评论

学校读者我要写书评

暂无评论

Persian Text-Image Retrieval: A Framework Based on Image Captioning and Scalable Vector Search 29

Persian Text-Image Retrieval: A Framework Based on Image Cap...

引用

29th International Computer Conference, Computer Society of Iran, CSICC 2025

作者： Asadian, Rasoul Akhavanpour, Alireza Shenasa AI Tehran Iran

ISBN: (纸本)9798331523114

Text-image retrieval systems have seen significant advancements with CLIP-based models and vision-language models, which rely on vast datasets and powerful computational resources. However, these advancements primarily focus on English and other high-resource languages, leaving a substantial gap for languages like Persian. This paper introduces a pioneering Persian Text-Image Retrieval (PTIR), the first comprehensive framework designed to address Persian’s unique challenges. We present three key innovations: (1) a diverse corpus of 1.2 million Persian image-caption pairs from multiple sources;(2) training a vision-language model combining a vision-encoder and a pre-trained Persian text-decoder to optimize cross-modal alignment and generate detailed captions;and (3) a scalable retrieval pipeline using sentence embedding models and vector databases for efficient Persian query-image matching. Evaluations on the Persian-adapted COCO benchmark dataset demonstrate PTIR achieves a 30-40% improvement over existing Persian image captioning models in standard metrics and a 9% mean improvement in Hit@K compared to the CLIP baseline. The framework’s modular design enables domain-specific adaptations, offering a blueprint for low-resource language retrieval systems. A subset of our implementation, including code and datasets, is available at: https://***/PTIR. © 2025 IEEE.

关键词： Multimodal Learning Persian Image Captioning Persian Text-Image Retrieval Vector Databases vision-language models

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：