检索结果-内蒙古大学图书馆

Multi-view visual semantic embedding for cross-modal image-text retrieval

PATTERN RECOGNITION 2025年 159卷

作者： Li, Zheng Guo, Caili Wang, Xin Zhang, Hao Hu, Lin Beijing Univ Posts & Telecommun Beijing Key Lab Network Syst Architecture & Conver Beijing Peoples R China Beijing Univ Posts & Telecommun Beijing Lab Adv Informat Networks Beijing Peoples R China China Telecom Digital Intelligence Technol Co Ltd Beijing Peoples R China State Key Lab Heavy duty & Express High power Elec Zhuzhou Peoples R China

visual semantic embedding (VSE) is a dominant method for cross-modal image-text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image-text data. Multiple captions describing the same image maybe described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View visual semantic embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image-text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image-text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at https://***/AAA-Zheng/MV-VSE.

关键词： Image-text retrieval Cross-modal retrieval visual semantic embedding Multi-view learning

来源：评论

学校读者我要写书评

暂无评论

VSE-fs: Fast Full-Sample visual semantic embedding

引用

IEEE INTELLIGENT SYSTEMS 2021年第4期36卷 3-12页

作者： Zhai, Songlin Guo, Guibing Yuan, Fajie Liu, Yuan Wang, Xingwei Northeastern Univ Shenyang 110819 Peoples R China

The visual semantic embedding (VSE) aims to construct a joint embedding space between visual features and semantic information, whereby classes can be well retrieved for a given image. However, VSE faces the computational challenge due to the large scale image-class data and the constrained system processing power. To speed up model training, many researchers resort to different sampling strategies by involving only a small portion of the classes at each training step. However, these methods are greatly biased especially when the sampling distribution deviates from the true data distribution. In order to retain VSE models fidelity, we adopt the regular full-sample in our algorithm. We also devise two separate optimization strategies to reduce time complexity, and derive more effective updating rules. The experimental results on four real datasets demonstrate that our approach not only converges much faster than the state-of-the-art sampling models, but also generates more accurate class retrieval.

关键词： visual semantic embedding Full Sample Negative Sampling

来源：评论

学校读者我要写书评

暂无评论

Hypercube Pooling for visual semantic embedding

引用

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 2024年第12期20卷 1-17页

作者： Wang, Hongbin Tang, Rui Li, Fan Kunming Univ Sci & Technol Fac Informat Engn & Automat Kunming Peoples R China Kunming Univ Sci & Technol Yunnan Prov Key Lab Artificial Intelligence Kunming Peoples R China

visual semantic embedding (VSE) is a primary model for cross-modal retrieval, wherein the global feature aggregator is a crucial component of the VSE model. In recent research, the General Pooling Operator (GPO) aggregator, which weighs the features reconstructed from the local feature set to aggregate, facilitates the related models to achieve good retrieval performance. However, the reason for the effectiveness remains to be explored. To enhance the rationality of aggregator designs, we analyze the reason from the perspective of feature space. Indeed, for each data, the local feature set forms a hypercube containing abundant data information, and the feature learned by GPO measures the hypercube, thereby representing the data. The geometric structure of the hypercube implies that the set containing all points within the hypercube is a convex set, so the feature learned by weighted aggregation is an interior point of the hypercube. However, using the interior point to measure the hypercube leads to some problems in feature representation and model optimization, as well as the reduction of retrieval efficiency caused by weight computation. For example, the related pair's features may be far, while the unrelated ones may be close. To measure the hypercube more clearly and alleviate the problems mentioned above, we propose Hypercube Pooling (HCP) aggregator. Specifically, HCP concatenates the Max and Min Pooling features as the global features. This aggregation method has multiple advantages, e.g., the learned global feature represents all hyperplanes of the hypercube that contain critical information and hypercube geometric structure. Moreover, HCP adds normalizationbefore-concatenation and reduces the usual setting of margin in the loss function by half to avoid gradient loss caused by the difference in the feature value and dimensionality doubling. The experimental results on the Flickr30K and MSCOCO datasets show that the HCP model has excellent performance with h

关键词： Cross-Modal Retrieval visual semantic embedding Global Pooling Hypercube

来源：评论

学校读者我要写书评

暂无评论

Regularizing visual semantic embedding With Contrastive Learning for Image-Text Matching

引用

IEEE SIGNAL PROCESSING LETTERS 2022年 29卷 1332-1336页

作者： Liu, Yang Liu, Hong Wang, Huaqiu Liu, Mengyuan Chongqing Univ Technol Sch Artificial Intelligence Chongqing 401135 Peoples R China Peking Univ Key Lab Machine Percept Beijing 100871 Peoples R China SunYat Sen Univ Sch Intelligent Syst Engn Guangzhou 510275 Peoples R China

Learning visual semantic embedding for image-text matching has achieved high success by using triplet loss to pull positive image-text pairs which share similar semantic meaning and to push negative image-text pairs which share different semantic meaning. Without modeling constraints from image-image or text-text pairs, the generated visual semantic embedding inevitably faces the problem of semantic misalignments among similar images or among similar texts. To solve this problem, we present a contrastive visual semantic embedding framework, named ConVSE, which achieves intra-modal semantic alignment by contrastive learning from augmented image-image (or text-text) pairs and achieves inter-modal semantic alignment by applying hardest-negative-enhanced triplet loss on image-text pairs. To the best of our knowledge, we are the first to find that contrastive learning benefits visual semantic embedding. Extensive experiments on large-scale MSCOCO and Flickr30 K datasets verify the effectiveness of our proposed ConVSE by outperforming visual semantic embedding-based methods and achieving new state-of-the-art.

关键词： semantics visualization Integrated circuit modeling Feature extraction Image retrieval Task analysis Computational modeling Cross-modal retrieval image-text matching visual semantic embedding

来源：评论

学校读者我要写书评

暂无评论

Multimodal Learning with Triplet Ranking Loss for visual semantic embedding Learning 12th

Multimodal Learning with Triplet Ranking Loss for Visual Sem...

引用

12th International Conference on Knowledge Science, Engineering and Management (KSEM)

作者： Yang, Zhanbo Li, Li He, Jun Wei, Zixi Liu, Li Liao, Jun Southwest Univ Sch Comp & Informat Sci Chongqing 400715 Peoples R China Chongqing Univ Sch Big Data & Software Engn Chongqing 400044 Peoples R China

ISBN: (纸本)9783030295516;9783030295509

semantic embedding learning for image and text has been well studied in recent years. In this paper, we present a simple while effective dual-encoder (image encoder and text encoder) framework to unify image and text into a common embedding space. Inspired by deep metric learning, we utilize triplet ranking loss to minimize the gap between the two embedding spaces. We train and test our proposed framework on Flickr8k, Flickr30k and MS-COCO datasets respectively, and evaluate the framework on the CoreIlk benchmark dataset as an application. Using VGG-19 for image encoder, GRU for text encoder and triplet ranking loss, we gained obvious improvement versus baseline model on image annotation and image search tasks. Additionally, we explore the vector generated by our image encoder and the one by word embedding of plain word for some arithmetic operations. The above experiments demonstrate the effectiveness of our proposed learning framework.

关键词： visual semantic embedding Triplet ranking loss Multimodal learning Word embedding

来源：评论

学校读者我要写书评

暂无评论

Dynamic Soft Labeling for visual semantic embedding 24

Dynamic Soft Labeling for Visual Semantic Embedding

引用

4th Annual International Conference on Multimedia Retrieval (ICMR)

作者： Yu, Jiaao Ding, Yunlai Dong, Junyu Li, Yuezun Ocean Univ China Coll Comp Sci & Technol Qingdao Shandong Peoples R China Ocean Univ China Sanya Oceanog Inst Sanya Hainan Peoples R China

ISBN: (纸本)9798400706028

visual semantic embedding (VSE) is a prominent approach in image-text retrieval, aiming to learn a deep embedding space that aligns visual data with semantic text labels. However, current VSE methods oversimplify the retrieval task, treating it as a binary classification problem with triplet loss constraints. This ignores the semantic correlation between pairs of mismatched samples and fails to capture the similarity gradient between samples. In addition, hard constraints on negative samples with high semantic relevance can be detrimental to the model's representational capabilities. To address these limitations, we propose a novel training strategy that introduces dynamic soft labels without additional annotations. This captures the correlation between positive and negative sample pairs and guides feature representation learning using the Soft Negative Alignment Loss (SNAL). SNAL fully takes into account the influence by similar negative samples, enhancing the representation of cross-modal data. In addition, we propose the Stepwise Negative Decoupling Loss (SNDL) to increase the distance between positive and negative samples. Stepwise decoupling of negative samples can be adaptively distanced based on their semantic relevance to the anchor, resulting in a wider distribution of sample features in the common space. Experiments on Flickr30K and MS-COCO datasets validate the effectiveness of our dynamic soft labeling (DSL) methods, demonstrating the importance of considering complex relationships between sample pairs and the limitations of rigid negative sample categorization based on subjective annotations.

关键词： visual semantic embedding similar negative samples image-text retrieval

来源：评论

学校读者我要写书评

暂无评论

Multilabel Deep visual-semantic embedding

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2020年第6期42卷 1530-1536页

作者： Yeh, Mei-Chen Li, Yi-Nan Natl Taiwan Normal Univ Dept Comp Sci & Informat Engn Taipei 106 Taiwan ASUS Corp Taipei 112 Taiwan

Inspired by the great success from deep convolutional neural networks (CNNs) for single-label visual-semantic embedding, we exploit extending these models for multilabel images. We propose a new learning paradigm for multilabel image classification, in which labels are ranked according to its relevance to the input image. In contrast to conventional CNN models that learn a latent vector representation (i.e., the image embedding vector), the developed visual model learns a mapping (i.e., a transformation matrix) from an image in an attempt to differentiate between its relevant and irrelevant labels. Despite the conceptual simplicity of our approach, the proposed model achieves state-of-the-art results on three public benchmark datasets.

关键词： semantics Computational modeling visualization Training Task analysis Convolutional neural networks Redundancy Multilabel classification visual semantic embedding convolutional neural networks

来源：评论

学校读者我要写书评

暂无评论

Ambiguity-Aware and High-order Relation learning for multi-grained image-text matching

引用

KNOWLEDGE-BASED SYSTEMS 2025年 316卷

作者： Chen, Junyu Gao, Yihua Ge, Mingyuan Li, Mingyong Chongqing Normal Univ Coll Comp & Informat Sci Chongqing 401331 Peoples R China Chongqing Univ Sch Big Data & Software Engn Chongqing 401331 Peoples R China

Image-text matching is crucial for bridging the semantic gap between computer vision and natural language processing. However, existing methods still face challenges in handling high-order associations and semantic ambiguities among similar instances. These ambiguities arise from subtle differences between soft positive samples (semantically similar but incorrectly labeled) and soft negative samples (locally matched but globally inconsistent), creating matching uncertainties. Furthermore, current methods fail to fully utilize the neighborhood relationships among semantically similar instances within training batches, limiting the model's ability to learn high-order shared knowledge. This paper proposes the Ambiguity-Aware and High-order Relation learning framework (AAHR) to address these issues. AAHR constructs a unified representation space through dynamic clustering prototype contrastive learning, effectively mitigating the soft positive sample problem. The framework introduces global and local feature extraction mechanisms and an adaptive aggregation network, significantly enhancing full-grained semantic understanding capabilities. Additionally, AAHR employs intramodal and inter-modal correlation matrices to investigate neighborhood relationships among sample instances thoroughly. It incorporates GNN to enhance semantic interactions between instances. Furthermore, AAHR integrates momentum contrastive learning to expand the negative sample set. These combined strategies significantly improve the model's ability to discriminate between features. Experimental results demonstrate that AAHR outperforms existing state-of-the-art methods on Flickr30K, MSCOCO, and ECCV Caption datasets, considerably improving the accuracy and efficiency of image-text matching. The code and model checkpoints for this research are available at https://***/Image-Text-Matching/AAHR.

关键词： Image-text matching Ambiguity-aware learning Neighborhood semantic interaction visual semantic embedding

来源：评论

学校读者我要写书评

暂无评论

Selectively Hard Negative Mining for Alleviating Gradient Vanishing in Image-Text Matching

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2025年第2期35卷 1921-1935页

作者： Li, Zheng Guo, Caili Wang, Xin Feng, Zerun Du, Zhongtian Beijing Univ Posts & Telecommun Sch Informat & Commun Engn Beijing 100876 Peoples R China China Telecom Digital Intelligence Technol Co Ltd Beijing 100035 Peoples R China China Telecom Inst Artificial Intelligence TeleAI Beijing 100032 Peoples R China

Most Image-Text Matching (ITM) models adopt Triplet loss with Hard Negative mining (T-HN) as the optimization objective. T-HN mines the hardest negative samples in each batch for training and achieves impressive performance. However, we observe that these ITM models have bad training behaviors in the early phases of training. Model training is difficult to converge, and matching performance is slow to improve. In this paper, we find that the cause of bad training behavior is that the model suffers from gradient vanishing. Optimizing an ITM model using only the hardest negative samples can easily lead to gradient vanishing. Through gradient analysis, we first derive the condition under which the gradient vanishes during training. We explain why the gradient tends to zero under certain conditions. To alleviate gradient vanishing, we propose a Triplet loss with Selectively Hard Negative mining (T-SelHN), which decides whether to mine the hardest negative samples according to the gradient vanishing condition. T-SelHN can be applied to ITM models in a plug-and-play manner to improve their training behaviors. To further ensure the back-propagation of gradients, we construct a Residual visual semantic embedding model with T-SelHN, denoted RVSE++, whichAcirc;has a simple network structure and efficient training and inference speeds. Extensive experiments on two ITM benchmarks demonstrate the strength of RVSE++, achieving state-of-the-art performance. The code is available at https://***/AAA-Zheng/RVSEPP.

关键词： Training visualization semantics Circuits and systems Telecommunications Optimization Measurement Feature extraction Neural networks Integrated circuit modeling Image-text matching hard negative mining gradient vanishing visual semantic embedding

来源：评论

学校读者我要写书评

暂无评论

LUNETR: Language-Infused UNETR for precise pancreatic tumor segmentation in 3D medical image

引用

NEURAL NETWORKS 2025年 187卷 107414页

作者： Shi, Ziyang Zhang, Ruopeng Wei, Xiajun Yu, Cheng Xie, Haojie Hu, Zhen Chen, Xili Zhang, Yongzhong Xie, Bin Luo, Zhengmao Peng, Wanxiang Xie, Xiaochun Li, Fang Long, Xiaoli Li, Lin Hu, Linan Cent South Univ Forestry & Technol Sch Elect Informat & Phys Changsha 410004 Peoples R China Cent South Univ Xiangya Sch Med Dept Radiol Zhuzhou Hosp Zhuzhou 412002 Peoples R China Cent South Univ Xiangya Hosp 2 Dept Radiol Changsha 410011 Peoples R China

The identification of early micro-lesions and adjacent blood vessels in CT scans plays a pivotal role in the clinical diagnosis of pancreatic cancer, considering its aggressive nature and high fatality rate. Despite the widespread application of deep learning methods for this task, several challenges persist: (1) the complex background environment in abdominal CT scans complicates the accurate localization of potential micro-tumors;(2) the subtle contrast between micro-lesions within pancreatic tissue and the surrounding tissues makes it challenging for models to capture these features accurately;and (3) tumors that invade adjacent blood vessels pose significant barriers to surgical procedures. To address these challenges, we propose LUNETR (Language-Infused UNETR), an advanced multimodal encoder model that combines textual and image information for precise medical image segmentation. The integration of an autoencoding language model with cross-attention enabling our model to effectively leverage semantic associations between textual and image data, thereby facilitating precise localization of potential pancreatic micro-tumors. Additionally, we designed a Multi-scale Aggregation Attention (MSAA) module to comprehensively capture both spatial and channel characteristics of global multiscale image data, enhancing the model's capacity to extract features from micro-lesions embedded within pancreatic tissue. Furthermore, in order to facilitate precise segmentation of pancreatic tumors and nearby blood vessels and address the scarcity of multimodal medical datasets, we collaborated with Zhuzhou Central Hospital to construct a multimodal dataset comprising CT images and corresponding pathology reports from 135 pancreatic cancer patients. Our experimental results surpass current state-of-the-art models, with the incorporation of the semantic encoder improving the average Dice score for pancreatic tumor segmentation by 2.23 %. For the Medical Segmentation Decathlon (MSD) li

关键词： Deep learning visual semantic embedding Multi-scale feature aggregation Medical image segmentation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：