检索结果-内蒙古大学图书馆

Zero-Shot Audio Captioning Using Soft and Hard Prompts

学校读者我要写书评

暂无评论

IEEE Transactions on Audio, Speech and Language Processing

IEEE Transactions on Audio, Speech and Language Processing 2025年 33卷 2045-2058页

作者： Yiming Zhang Xuenan Xu Ruoyi Du Haohe Liu Yuan Dong Zheng-Hua Tan Wenwu Wang Zhanyu Ma Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing China Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Centre for Vision Speech and Signal Processing University of Surrey Guildford U.K. Department of Electronic Systems Aalborg University Aalborg Denmark

In traditional audio captioning methods, a model is usually trained in a fully supervised manner using a human-annotated dataset containing audio-text pairs and then evaluated on the test set from the same dataset. Such methods have two limitations. First, these methods are often data-hungry and require time-consuming and expensive human annotations to obtain audio-text pairs. Second, these models often suffer from performance degradation in cross-domain scenarios, i.e., when the input audio comes from a different domain than the training set, and this issue has received little attention. To address these issues, we propose a new zero-shot method for audio captioning. Our method is built on the contrastive language-audio pre-training (CLAP) model. During training, the model reconstructs the ground-truth caption using the CLAP text encoder. In the inference stage, the model generates text descriptions from the CLAP audio embeddings of given audio inputs. To enhance the ability of the model in transitioning from text-to-text generation to audio-to-text generation, we propose to use the mixed-augmentations-based soft prompt to learn more robust latent representations, leveraging instance replacement and embedding augmentation. Additionally, we introduce the retrieval-based acoustic-aware hard prompt to improve the cross-domain performance of the model by employing the domain-agnostic label information of sound events. Extensive experiments on AudioCaps and Clotho benchmarks show the effectiveness of our proposed method, which outperforms other zero-shot audio captioning approaches for in-domain scenarios and outperforms the compared methods for cross-domain scenarios, underscoring the generalization ability of our method.

关键词： Training Decoding Semantics Data models Acoustics Electronic mail Benchmark testing Transformers Robustness Perturbation methods

SFace: Sigmoid-Constrained Hypersphere Loss for Robust Face recognition

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhong, Yaoyao Deng, Weihong Hu, Jiani Zhao, Dongyue Li, Xian Wen, Dongchao The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China Co. Ltd China

Deep face recognition has achieved great success due to large-scale training databases and rapidly developing loss functions. The existing algorithms devote to realizing an ideal idea: minimizing the intra-class distance and maximizing the inter-class distance. However, they may neglect that there are also low quality training images which should not be optimized in this strict way. Considering the imperfection of training databases, we propose that intra-class and inter-class objectives can be optimized in a moderate way to mitigate overfitting problem, and further propose a novel loss function, named sigmoid-constrained hypersphere loss (SFace). Specifically, SFace imposes intra-class and inter-class constraints on a hypersphere manifold, which are controlled by two sigmoid gradient re-scale functions respectively. The sigmoid curves precisely re-scale the intra-class and inter-class gradients so that training samples can be optimized to some degree. Therefore, SFace can make a better balance between decreasing the intra-class distances for clean examples and preventing overfitting to the label noise, and contributes more robust deep face recognition models. Extensive experiments of models trained on CASIA-WebFace, VGGFace2, and MS-Celeb-1M databases, and evaluated on several face recognition benchmarks, such as LFW, MegaFace and IJB-C databases, have demonstrated the superiority of SFace. Copyright © 2022, The Authors. All rights reserved.

关键词： Face recognition

Oracle Character recognition using Unsupervised Discriminative Consistency Network

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, Mei Deng, Weihong Su, Sen The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications Beijing100876 China

Ancient history relies on the study of ancient characters. However, real-world scanned oracle characters are difficult to collect and annotate, posing a major obstacle for oracle character recognition (OrCR). Besides, serious abrasion and inter-class similarity also make OrCR more challenging. In this paper, we propose a novel unsupervised domain adaptation method for OrCR, which enables to transfer knowledge from labeled handprinted oracle characters to unlabeled scanned data. We leverage pseudo-labeling to incorporate the semantic information into adaptation and constrain augmentation consistency to make the predictions of scanned samples consistent under different perturbations, leading to the model robustness to abrasion, stain and distortion. Simultaneously, an unsupervised transition loss is proposed to learn more discriminative features on the scanned domain by optimizing both between-class and within-class transition probability. Extensive experiments show that our approach achieves state-of-the-art result on Oracle-241 dataset and substantially outperforms the recently proposed structure-texture separation network by 15.1%. © 2023, CC BY.

关键词： Character recognition

GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Ou, Ruizhe Hu, Yuan Zhang, Fan Chen, Jiaxin Liu, Yu Pattern Recognition and Intelligent System Lab School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China Institute of Remote Sensing and Geographic Information Systems School of Earth and Space Sciences Peking University Beijing100871 China Peking University Ordos Research Institute of Energy Ordos017000 China

Multi-modal large language models (MLLMs) have achieved remarkable success in image- and region-level remote sensing (RS) image understanding tasks, such as image captioning, visual question answering, and visual grounding. However, existing RS MLLMs lack the pixel-level dialogue capability, which involves responding to user instructions with segmentation masks for specific instances. In this paper, we propose GeoPix, a RS MLLM that extends image understanding capabilities to the pixel level. This is achieved by equipping the MLLM with a mask predictor, which transforms visual features from the vision encoder into masks conditioned on the LLM’s segmentation token embeddings. To facilitate the segmentation of multi-scale objects in RS imagery, a class-wise learnable memory module is integrated into the mask predictor to capture and store class-wise geo-context at the instance level across the entire dataset. In addition, to address the absence of large-scale datasets for training pixel-level RS MLLMs, we construct the GeoPixInstruct dataset, comprising 65,463 images and 140,412 instances, with each instance annotated with text descriptions, bounding boxes, and masks. Furthermore, we develop a two-stage training strategy to balance the distinct requirements of text generation and masks prediction in multi-modal multi-task optimization. Extensive experiments verify the effectiveness and superiority of GeoPix in pixel-level segmentation tasks, while also maintaining competitive performance in image- and region-level benchmarks. The models, dataset, and code are publicly available at https://***/Norman-Ou/GeoPix. © 2025, CC BY.

关键词： Image segmentation

Learning Invariant Visual Representations for Compositional Zero-Shot Learning

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhang, Tian Liang, Kongming Du, Ruoyi Sun, Xian Ma, Zhanyu Guo, Jun Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications China Aerospace Information Research Institute Chinese Academy of Sciences China

Compositional Zero-Shot Learning (CZSL) aims to recognize novel compositions using knowledge learned from seen attribute-object compositions in the training set. Previous works mainly project an image and a composition into a common embedding space to measure their compatibility score. However, both attributes and objects share the visual representations learned above, leading the model to exploit spurious correlations and bias towards seen pairs. Instead, we reconsider CZSL as an out-of-distribution generalization problem. If an object is treated as a domain, we can learn object-invariant features to recognize the attributes attached to any object reliably. Similarly, attribute-invariant features can also be learned when recognizing the objects with attributes as domains. Specifically, we propose an invariant feature learning framework to align different domains at the representation and gradient levels to capture the intrinsic characteristics associated with the tasks. Experiments on three CZSL benchmarks demonstrate that the proposed method significantly outperforms the previous state-of-the-art. Copyright © 2022, The Authors. All rights reserved.

关键词： Zero-shot learning

Unsupervised Structure-Texture Separation Network for Oracle Character recognition

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Wang, Mei Deng, Weihong Liu, Cheng-Lin The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China The National Laboratory of Pattern Recognition Institute of Automation Chinese Academy of Sciences Beijing100190 China The School of Artificial Intelligence University of Chinese Academy of Sciences Beijing100049 China

Oracle bone script is the earliest-known Chinese writing system of the Shang dynasty and is precious to archeology and philology. However, real-world scanned oracle data are rare and few experts are available for annotation which make the automatic recognition of scanned oracle characters become a challenging task. Therefore, we aim to explore unsupervised domain adaptation to transfer knowledge from handprinted oracle data, which are easy to acquire, to scanned domain. We propose a structure-texture separation network (STSN), which is an end-to-end learning framework for joint disentanglement, transformation, adaptation and recognition. First, STSN disentangles features into structure (glyph) and texture (noise) components by generative models, and then aligns handprinted and scanned data in structure feature space such that the negative influence caused by serious noises can be avoided when adapting. Second, transformation is achieved via swapping the learned textures across domains and a classifier for final classification is trained to predict the labels of the transformed scanned characters. This not only guarantees the absolute separation, but also enhances the discriminative ability of the learned features. Extensive experiments on Oracle-241 dataset show that STSN outperforms other adaptation methods and successfully improves recognition performance on scanned data even when they are contaminated by long burial and careless excavation. © 2022, CC BY.

关键词： Generative adversarial networks

Natural image matting based on image inpainting

学校读者我要写书评

暂无评论

Natural image matting based on image inpainting

Computer Graphics, Image and Virtualization (ICCGIV), International Conference on

作者： Yuan Zhang Mian Tan Zhulian Zhou Yuan Yang Yihui Liang Fujian Feng Guizhou Key Laboratory of Pattern Recognition and Intelligent System Guizhou Minzu University Guiyang China School of Computer Science University of Electronic Science and Technology China Zhongshan China

Deep image matting is a hot problem with applications in computer vision and image processing. It has been widely used in image composition, film production and video editing etc. The current matting method based on image inpainting uses a deep neural network to inpaint the foreground target and background region to further improve the accuracy of alpha matte. However, when the trimap contains large unknown regions, the excessive inpainting of foreground and background produce a lot of redundant information, which leads to a degradation of alpha matte quality. Therefore, to address this issue, a natural image matting method based on image inpainting is designed. This method involves the refinement process for trimap, which improves the quality of the trimap, enlarges the foreground and background regions providing additional information for image matting. Extensive experimental results on the composition-1k dataset demonstrate that the presented method provide high-quality alpha mattes not only in the case that the trimap contains small unknown regions, but also in the case that the trimap contains large unknown regions.

关键词： Degradation Deep learning Computer vision Image processing Refining Neural networks Production

Isoform Function Prediction Based on Heterogeneous Graph Attention Networks

学校读者我要写书评

暂无评论

Isoform Function Prediction Based on Heterogeneous Graph Att...

2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023

作者： Guo, Kuo Li, Yifan Chen, Hao Shen, Hong-Bin Yang, Yang Shanghai Jiao Tong University Key Lab. of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Department of Computer Science and Engineering Shanghai200240 China Shanghai Jiao Tong University Key Laboratory of System Control and Information Processing Ministry of Education of China Institute of Image Processing and Pattern Recognition Shanghai200240 China Carnegie Mellon University School of Computer Science Computational Biology Department PittsburghPA15213 United States

ISBN: (纸本)9798350337488

Isoforms refer to different mRNA molecules transcribed from the same gene, which can be translated into proteins with varying structures and functions. Predicting the functions of isoforms is an essential topic in bioinformatics as it can provide valuable insights into the intricate mechanisms of gene regulation and biological processes. Conventionally, gene function labels are standardized in Gene Ontology (GO) terms. However, traditional methods for predicting isoform function are largely limited by the absence of isoform-specific labels, sparse annotations, and the vast number of GO terms. To address these issues, we propose HANIso, a deep learning-based method for isoform function prediction. HANIso leverages a pretrained protein language model to extract features from protein sequences. It also integrates heterogeneous information, such as isoform sequence features, GO annotations, and isoform interaction data, using a Heterogeneous Graph Attention Network (HAN). This allows the model to learn the importance of different sources of information and their semantic relationships through the attention mechanism. Our method can predict function labels at both the gene level and isoform level. We conduct experiments on two species datasets, and the results demonstrate that our method outperforms existing methods on both AUROC and AUPRC. HANIso has the potential to overcome the limitations of traditional methods and provide a more accurate and comprehensive understanding of isoform function. © 2023 IEEE.

关键词： alternative splicing gene ontology heterogeneous graph attention network isoform function prediction protein language model

MEET: A Million-Scale Dataset for Fine-Grained Geospatial Scene Classification with Zoom-Free Remote Sensing Imagery

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Li, Yansheng Wu, Yuning Cheng, Gong Tao, Chao Dang, Bo Wang, Yu Zhang, Jiahao Zhang, Chuge Liu, Yiting Tang, Xu Ma, Jiayi Zhang, Yongjun Information and computing science from Shandong University Weihai China Pattern recognition and intelligent system from the Huazhong University of Science and Technology Wuhan China Computer science and technology from Wuhan University Wuhan China Xidian University Xi’an China Pattern recognition and intelligent systems from Northwestern Polytechnical University Xi’an China School of Mathematics and Statistics Huazhong University of Science and Technology Wuhan China Institute for Pattern Recognition and Artificial Intelligence Huazhong China University of Science and Technology Remote sensing science and technology from Wuhan University Wuhan China School of Remote Sensing and Information Engineering Wuhan University Wuhan China Electronic circuit and system from Xidian University Xi’an China University of Colorado at Boulder BoulderCO United States Control science and engineering from the Huazhong University of Science and Technology Wuhan China Geodesy and photography from Wuhan University Wuhan China

Accurate fine-grained geospatial scene classification using remote sensing imagery is essential for a wide range of applications. However, existing approaches often rely on manually zooming remote sensing images at different scales to create typical scene samples. This approach fails to adequately support the fixed-resolution image interpretation requirements in real-world scenarios. To address this limitation, we introduce the Million-scale finE-grained geospatial scEne classification dataseT (MEET), which contains over 1.03 million zoom-free remote sensing scene samples, manually annotated into 80 fine-grained categories. In MEET, each scene sample follows a scene-in-scene layout, where the central scene serves as the reference, and auxiliary scenes provide crucial spatial context for fine-grained classification. Moreover, to tackle the emerging challenge of scene-in-scene classification, we present the Context-Aware Transformer (CAT), a model specifically designed for this task, which adaptively fuses spatial context to accurately classify the scene samples. CAT adaptively fuses spatial context to accurately classify the scene samples by learning attentional features that capture the relationships between the center and auxiliary scenes. Based on MEET, we establish a comprehensive benchmark for fine-grained geospatial scene classification, evaluating CAT against 11 competitive baselines. The results demonstrate that CAT significantly outperforms these baselines, achieving a 1.88% higher balanced accuracy (BA) with the Swin-Large backbone, and a notable 7.87% improvement with the Swin-Huge backbone. Further experiments validate the effectiveness of each module in CAT and show the practical applicability of CAT in the urban functional zone mapping. The source code and dataset will be publicly available at https://***/project/***. Copyright © 2025, The Authors. All rights reserved.

关键词： Image classification