检索结果-内蒙古大学图书馆

Joint 30th International Conference on Computational Linguistics and 14th International Conference on Language Resources and Evaluation, LREC-COLING 2024

作者： Yang, Juncheng Li, Zuchao Xie, Shuai Yu, Wei Li, Shijun Du, Bo School of Computer Science Wuhan University Hubei Wuhan China National Engineering Research Center for Multimedia Software Hubei Wuhan China School of Electronic Information Engineering Henan Polytechnic Institute Henan Nanyang China JD Explore Academy Beijing China

ISBN: (纸本)9782493814104

The chain-of-thought technique has been received well in multi-modal tasks. It is a step-by-step linear reasoning process that adjusts the length of the chain to improve the performance of generated prompts. However, human thought processes are predominantly non-linear, as they encompass multiple aspects simultaneously and employ dynamic adjustment and updating mechanisms. Therefore, we propose a novel Aggregation-Graph-of-Thought (AGoT) mechanism for soft-prompt tuning in multi-modal representation learning. The proposed AGoT models the human thought process not only as a chain but also models each step as a reasoning aggregation graph to cope with the overlooked multiple aspects of thinking in single-step reasoning. This turns the entire reasoning process into prompt aggregation and prompt flow operations. Experiments show that our multi-modal model enhanced with AGoT soft-prompting achieves good results in several tasks such as text-image retrieval, visual question answering, and image recognition. In addition, we demonstrate that it has good domain generalization performance due to better reasoning. © 2024 ELRA Language Resource Association: CC BY-NC 4.0.

关键词： Image enhancement

来源：评论

学校读者我要写书评

暂无评论

Empowering visible-infrared person re-identification with large foundation models 24

Empowering visible-infrared person re-identification with la...

引用

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Zhangyi Hu Bin Yang Mang Ye National Engineering Research Center for Multimedia Software School of Computer Science Wuhan University Wuhan China

ISBN: (纸本)9798331314385

Visible-Infrared Person Re-identification (VI-ReID) is a challenging cross-modal retrieval task due to significant modality differences, primarily resulting from the absence of color information in the infrared modality. The development of large foundation models like Large Language Models (LLMs) and Vision Language Models (VLMs) motivates us to explore a feasible solution to empower VI-ReID with off-the-shelf large foundation models. To this end, we propose a novel Text-enhanced VI-ReID framework driven by Large Foundation Models (TVI-LFM). The core idea is to enrich the representation of the infrared modality with textual descriptions automatically generated by VLMs. Specifically, we incorporate a pre-trained VLM to extract textual features from texts generated by VLM and augmented by LLM, and incrementally fine-tune the text encoder to minimize the domain gap between generated texts and original visual modalities. Meanwhile, to enhance the infrared modality with extracted textual representations, we leverage modality alignment capabilities of VLMs and VLM-generated feature-level filters. This enables the text model to learn complementary features from the infrared modality, ensuring the semantic structural consistency between the fusion modality and the visible modality. Furthermore, we introduce modality joint learning to align features across all modalities, ensuring that textual features maintain stable semantic representation of overall pedestrian appearance during complementary information learning. Additionally, a modality ensemble retrieval strategy is proposed to leverage complementary strengths of each query modality to improve retrieval effectiveness and robustness. Extensive experiments on three expanded VI-ReID datasets demonstrate that our method significantly improves the retrieval performance, paving the way for the utilization of large foundation models in downstream multi-modal retrieval tasks.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Sample complexity for distributionally robust learning under χ2-divergence

引用

The Journal of Machine Learning Research 2023年第1期24卷 10845-10871页

作者： Zhengyu Zhou Weiwei Liu School of Computer Science National Engineering Research Center for Multimedia Software Institute of Artificial Intelligence Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan University Wuhan China

This paper investigates the sample complexity of learning a distributionally robust predictor under a particular distributional shift based on χ2-divergence, which is well known for its computational feasibility and statistical properties. We demonstrate that any hypothesis class ℌ with finite VC dimension is distributionally robustly learnable. Moreover, we show that when the perturbation size is smaller than a constant, finite VC dimension is also necessary for distributionally robust learning by deriving a lower bound of sample complexity in terms of VC dimension.

关键词： distributionally robustness PAC learning sample complexity χ2-divergence

来源：评论

学校读者我要写书评

暂无评论

Adversarial Guided Gradient Estimation Hashing for Cross-modal Retrieval 8

Adversarial Guided Gradient Estimation Hashing for Cross-mod...

引用

8th IEEE International Conference on Cloud Computing and Intelligence Systems, CCIS 2022

作者： Lu, Kangkang Liang, Meiyu Xue, Zhe Cao, Xiaowen Yin, Mengran Zhao, Zehua Beijing University of Posts and Telecommunications Beijing Key Laboratory of Intelligent Communication Software and Multimedia School of Computer Science National Pilot Software Engineering School Beijing100876 China

ISBN: (纸本)9781665477352

Due to low storage cost and fast query speed, deep hashing methods are widely used in cross-modal retrieval. However, the 'heterogeneous gap' between multi-modal data is still a challenge. Moreover, a major difficulty in deep hashing lies in the discrete constraints imposed on the network output. Existing solutions usually use relaxation techniques, but this inevitably produces quantization error, leading to sub-optimal hash code. In this paper, we propose Adversarial Guided Gradient Estimation Hashing (AGEH). Firstly, in order to bridge the heterogeneous gap between different modal data, a cross-modal adversarial feature learning network is constructed to learn cross-modal semantic associations. Secondly, to solve the discrete optimization problem of hash code, we propose a hashing optimization strategy based on gradient estimation for sign function, which strictly uses sign function to maintain discrete constraints in forward propagation, while in back propagation, the gradient is directly transmitted to the previous layer and thus avoid quantization error. Extensive experiments conducted on two cross-modal benchmark datasets show that our proposed AGEH outperforms several state-of-the-art methods. © 2022 IEEE.

关键词： Backpropagation

来源：评论

学校读者我要写书评

暂无评论

Cross-modal Search Method of Technology Video based on Adversarial Learning and Feature Fusion 8

Cross-modal Search Method of Technology Video based on Adver...

引用

8th IEEE International Conference on Cloud Computing and Intelligence Systems, CCIS 2022

作者： Liu, Xiangbin Du, Junping Liang, Meiyu Li, Ang Beijing University of Posts and Telecommunications Beijing Key Laboratory of Intelligent Communication Software and Multimedia School of Computer Science National Pilot Software Engineering School Beijing100876 China

ISBN: (纸本)9781665477352

Technology videos contain rich multi-modal information. In cross-modal information search, the data features of different modalities cannot be compared directly, so the semantic gap between different modalities is a key problem that needs to be solved. To address the above problems, this paper proposes a novel Feature Fusion based Adversarial Cross-modal Retrieval method (FFACR) to achieve text-to-video matching, ranking and searching. The proposed method uses the framework of adversarial learning to construct a video multimodal feature fusion network and a feature mapping network as generator, a modality discrimination network as discriminator. Generator and discriminator are trained alternately based on adversarial learning, so that the data obtained by the feature mapping network is semantically consistent with the original data and the modal features are eliminated. Experimental results demonstrate that the proposed method performs better in text-to-video search on the self-built datasets of technology videos. © 2022 IEEE.

关键词： Semantics

来源：评论

学校读者我要写书评

暂无评论

Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data

arXiv

引用

arXiv 2024年

作者： Zou, Xin Liu, Weiwei School of Computer Science Institute of Artificial Intelligence National Engineering Research Center for Multimedia Software Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan University China

Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. In this paper, we study the confidence set prediction problem in the OOD generalization setting. Split conformal prediction (SCP) is an efficient framework for handling the confidence set prediction problem. However, the validity of SCP requires the examples to be exchangeable, which is violated in the OOD setting. Empirically, we show that trivially applying SCP results in a failure to maintain the marginal coverage when the unseen target domain is different from the source domain. To address this issue, we develop a method for forming confident prediction sets in the OOD setting and theoretically prove the validity of our method. Finally, we conduct experiments on simulated data to empirically verify the correctness of our theory and the validity of our proposed method. Copyright © 2024, The Authors. All rights reserved.

关键词： Forecasting

来源：评论

学校读者我要写书评

暂无评论

MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification

arXiv

引用

arXiv 2025年

作者： Li, Yapeng Luo, Yong Zhang, Lefei Wang, Zengmao Du, Bo National Engineering Research Center for Multimedia Software School of Computer Science Institute of Artificial Intelligence Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan University Wuhan430072 China

Transformer has been extensively explored for hyperspectral image classification. However, transformer poses challenges in terms of speed and memory usage because of its quadratic computational complexity. Recently the Mamba model has emerged as a promising approach, which has strong long-distance modeling capabilities while maintaining a linear computational complexity. However, representing the hyperspectral image is challenging for the Mamba due to the requirement for an integrated spatial and spectral understanding. To remedy these drawbacks, we propose a novel hyperspectral image classification model based on a Mamba model, named MambaHSI, which can simultaneously model long-range interaction of the whole image and integrate spatial and spectral information in an adaptive manner. Specifically, we design a spatial Mamba block to model the long-range interaction of the whole image at the pixel-level. Then we propose a spectral Mamba block to split the spectral vector into multiple groups, mine the relations across different spectral groups, and extract spectral features. Finally, we propose a spatial-spectral fusion module to adaptively integrate spatial and spectral features of a hyperspectral image. To our best knowledge, this is the first image-level hyperspectral image classification model based on the Mamba. We conduct extensive experiments on four diverse hyperspectral image datasets. The results demonstrate the effectiveness and superiority of the proposed model for hyperspectral image classification. This reveals the great potential of Mamba to be the next-generation backbone for hyperspectral image models. Codes are available at https://***/li-yapeng/MambaHSI. Copyright © 2025, The Authors. All rights reserved.

关键词： Image classification

来源：评论

学校读者我要写书评

暂无评论

MMSite: A Multi-modal Framework for the Identification of Active Sites in Proteins 38

MMSite: A Multi-modal Framework for the Identification of Ac...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Ouyang, Song Cai, Huiyu Luo, Yong Su, Kehua Zhang, Lefei Du, Bo School of Computer Science National Engineering Research Center for Multimedia Software Institute of Artificial Intelligence Wuhan University China BioGeometry China Mila - Québec AI Institute Canada Université de Montréal Canada

The accurate identification of active sites in proteins is essential for the advancement of life sciences and pharmaceutical development, as these sites are of critical importance for enzyme activity and drug design. Recent advancements in protein language models (PLMs), trained on extensive datasets of amino acid sequences, have significantly improved our understanding of proteins. However, compared to the abundant protein sequence data, functional annotations, especially precise per-residue annotations, are scarce, which limits the performance of PLMs. On the other hand, textual descriptions of proteins, which could be annotated by human experts or a pretrained protein sequence-to-text model, provide meaningful context that could assist in the functional annotations, such as the localization of active sites. This motivates us to construct a ProTein-Attribute text Dataset (ProTAD), comprising over 570,000 pairs of protein sequences and multi-attribute textual descriptions. Based on this dataset, we propose MMSite, a multi-modal framework that improves the performance of PLMs to identify active sites by leveraging biomedical language models (BLMs). In particular, we incorporate manual prompting and design a MACross module to deal with the multi-attribute characteristics of textual descriptions. MMSite is a two-stage ("First Align, Then Fuse") framework: first aligns the textual modality with the sequential modality through soft-label alignment, and then identifies active sites via multi-modal fusion. Experimental results demonstrate that MMSite achieves state-of-the-art performance compared to existing protein representation learning methods. The dataset and code implementation are available at https://***/Gift-OYS/MMSite. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer 37

DPText-DETR: Towards Better Scene Text Detection with Dynami...

引用

37th AAAI Conference on Artificial Intelligence, AAAI 2023

作者： Ye, Maoyuan Zhang, Jing Zhao, Shanshan Liu, Juhua Bo, Du Tao, Dacheng Research Center for Graphic Communication Printing and Packaging Institute of Artificial Intelligence Wuhan University China The University of Sydney Australia JD Explore Academy National Engineering Research Center for Multimedia Software Institute of Artificial Intelligence School of Computer Science Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan University China

ISBN: (纸本)9781577358800

Recently, Transformer-based methods, which predict polygon points or Bezier curve control points for localizing texts, are popular in scene text detection. However, these methods built upon detection transformer framework might achieve sub-optimal training efficiency and performance due to coarse positional query modeling. In addition, the point label form exploited in previous works implies the reading order of humans, which impedes the detection robustness from our observation. To address these challenges, this paper proposes a concise Dynamic Point Text DEtection TRansformer network, termed DPText-DETR. In detail, DPText-DETR directly leverages explicit point coordinates to generate position queries and dynamically updates them in a progressive way. Moreover, to improve the spatial inductive bias of non-local self-attention in Transformer, we present an Enhanced Factorized Self-Attention module which provides point queries within each instance with circular shape guidance. Furthermore, we design a simple yet effective positional label form to tackle the side effect of the previous form. To further evaluate the impact of different label forms on the detection robustness in real-world scenario, we establish an Inverse-Text test set containing 500 manually labeled images. Extensive experiments prove the high training efficiency, robustness, and state-of-the-art performance of our method on popular benchmarks. The code and the Inverse-Text test set are available at https://***/ymy-k/DPText-DETR. Copyright © 2023, Association for the Advancement of Artificial Intelligence (***). All rights reserved.

关键词： Efficiency

来源：评论

学校读者我要写书评

暂无评论

Sequential Kernel Goodness-of-fit testing 24

Sequential Kernel Goodness-of-fit testing

引用

Proceedings of the 41st International Conference on Machine Learning

作者： Zhengyu Zhou Weiwei Liu School of Computer Science National Engineering Research Center for Multimedia Software Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan University Wuhan China

Goodness-of-fit testing, a classical statistical tool, has been extensively explored in the batch setting, where the sample size is predetermined. However, practitioners often prefer methods that adapt to the complexity of a problem rather than fixing the sample size beforehand. Classical batch tests are generally unsuitable for streaming data, as valid inference after data peeking requires multiple testing corrections, resulting in reduced statistical power. To address this issue, we delve into the design of consistent sequential goodness-of-fit tests. Following the principle of testing by betting, we reframe this task as selecting a sequence of payoff functions that maximize the wealth of a fictitious bettor, betting against the null in a repeated game. We conduct experiments to demonstrate the adaptability of our sequential test across varying difficulty levels of problems while maintaining control over type-I errors.

关键词：

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：