检索结果-内蒙古大学图书馆

Zero-Shot Audio Captioning Using Soft and Hard Prompts

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Yiming Xu, Xuenan Du, Ruoyi Liu, Haohe Dong, Yuan Tan, Zheng-Hua Wang, Wenwu Ma, Zhanyu The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China The Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai200240 China The Department of Electronic Systems Aalborg University Aalborg9220 Denmark The Centre for Vision Speech and Signal Processing University of Surrey GuildfordGU2 7XH United Kingdom

In traditional audio captioning methods, a model is usually trained in a fully supervised manner using a human-annotated dataset containing audio-text pairs and then evaluated on the test sets from the same dataset. Such methods have two limitations. First, these methods are often data-hungry and require time-consuming and expensive human annotations to obtain audio-text pairs. Second, these models often suffer from performance degradation in cross-domain scenarios, i.e., when the input audio comes from a different domain than the training set, which, however, has received little attention. We propose an effective audio captioning method based on the contrastive language-audio pre-training (CLAP) model to address these issues. Our proposed method requires only textual data for training, enabling the model to generate text from the textual feature in the cross-modal semantic space. In the inference stage, the model generates the descriptive text for the given audio from the audio feature by leveraging the audio-text alignment from CLAP. We devise two strategies to mitigate the discrepancy between text and audio embeddings: a mixed-augmentation-based soft prompt and a retrieval-based acoustic-aware hard prompt. These approaches are designed to enhance the generalization performance of our proposed model, facilitating the model to generate captions more robustly and accurately. Extensive experiments on AudioCaps and Clotho benchmarks show the effectiveness of our proposed method, which outperforms other zero-shot audio captioning approaches for in-domain scenarios and outperforms the compared methods for cross-domain scenarios, underscoring the generalization ability of our method. © 2024, CC BY.

关键词： Semantics

Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Meiqi Huang, Kaiqi Cai, Yuanqiang Hu, Shiyu Zhao, Yuzhong Wang, Weiqiang School of Computer Science and Technology University of Chinese Academy of Sciences Beijing100049 China Beijing University of Posts and Telecommunications Beijing100876 China School of Artificial Intelligence University of Chinese Academy of Sciences Beijing100049 China Center for Research on Intelligent System and Engineering Institute of Automation Chinese Academy of Sciences Beijing100190 China Center for Research on Intelligent System and Engineering National Laboratory of Pattern Recognition Institute of Automation Chinese Academy of Sciences Beijing100190 China University of Chinese Academy of Sciences Beijing100049 China CAS Center for Excellence in Brain Science and Intelligence Technology Shanghai200031 China

Air-writing is a challenging task that combines the fields of computer vision and natural language processing, offering an intuitive and natural approach for human-computer interaction. However, current air-writing solutions face two primary challenges: (1) their dependency on complex sensors (e.g., Radar, EEGs and others) for capturing precise handwritten trajectories, and (2) the absence of a video-based air-writing dataset that covers a comprehensive vocabulary range. These limitations impede their practicality in various real-world scenarios, including the use on devices like iPhones and laptops. To tackle these challenges, we present the groundbreaking air-writing Chinese character video dataset (AWCV-100K-UCAS2024), serving as a pioneering benchmark for video-based air-writing. This dataset captures handwritten trajectories in various real-world scenarios using commonly accessible RGB cameras, eliminating the need for complex sensors. AWCV-100K-UCAS2024 includes 8.8 million video frames, encompassing the complete set of 3,755 characters from the GB2312-80 level-1 set (GB1). Furthermore, we introduce our baseline approach, the video-based character recognizer (VCRec). VCRec adeptly extracts fingertip features from sparse visual cues and employs a spatio-temporal sequence module for analysis. Experimental results showcase the superior performance of VCRec compared to existing models in recognizing air-written characters, both quantitatively and qualitatively. This breakthrough paves the way for enhanced human-computer interaction in real-world contexts. Moreover, our approach leverages affordable RGB cameras, enabling its applicability in a diverse range of scenarios. The code and data examples will be made public at https://***/wmeiqi/AWCV. Copyright © 2024, The Authors. All rights reserved.

关键词： Natural language processing systems

Structured DropConnect for uncertainty inference in image classification

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Zheng, Wenqing Xie, Jiyang Liu, Weidong Ma, Zhanyu Pattern Recognition and Intelligent System Lab Beijing University of Posts and Telecommunications Beijing China China Mobile Research Institute Beijing China Beijing Academy of Artificial Intelligence

With the complexity of the network structure, uncertainty inference has become an important task to improve classification accuracy for artificial intelligence systems. For image classification tasks, we propose a structured DropConnect (SDC) framework to model the output of a deep neural network by a Dirichlet distribution. We introduce a DropConnect strategy on weights in the fully connected layers during training. In test, we split the network into several sub-networks, and then model the Dirichlet distribution by match its moments with the mean and variance of the outputs of these sub-networks. The entropy of the estimated Dirichlet distribution is finally utilized for uncertainty inference. In this paper, this framework is implemented on LeNet5 and VGG16 models for misclassification detection and out-of-distribution detection on MNIST and CIFAR-10 datasets. Experimental results show that the performance of the proposed SDC can be comparable to other uncertainty inference methods. Furthermore, the SDC is adapted well to different network structures with certain generalization capabilities and research *** Codes 14J60 (Primary) 14F05, 14J26 (Secondary) Copyright © 2021, The Authors. All rights reserved.

关键词： Image classification

Clue me in: Semi-supervised FGVC with out-of-distribution data

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Du, Ruoyi Chang, Dongliang Ma, Zhanyu Song, Yi-Zhe Guo, Jun The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China SketchX CVSSP University of Surrey London United Kingdom

Despite great strides made on fine-grained visual classification (FGVC), current methods are still heavily reliant on fully-supervised paradigms where ample expert labels are called for. Semi-supervised learning (SSL) techniques, acquiring knowledge from unlabeled data, provide a considerable means forward and have shown great promise for coarse-grained problems. However, exiting SSL paradigms mostly assume in-distribution (i.e., category-aligned) unlabeled data, which hinders their effectiveness when re-proposed on FGVC. In this paper, we put forward a novel design specifically aimed at making out-of-distribution data work for semi-supervised FGVC, i.e., to "clue them in". We work off an important assumption that all fine-grained categories naturally follow a hierarchical structure (e.g., the phylogenetic tree of "Aves" that covers all bird species). It follows that, instead of operating on individual samples, we can instead predict sample relations within this tree structure as the optimization goal of SSL. Beyond this, we further introduced two strategies uniquely brought by these tree structures to achieve inter-sample consistency regularization and reliable pseudo-relation. Our experimental results reveal that (i) the proposed method yields good robustness against out-of-distribution data, and (ii) it can be equipped with prior arts, boosting their performance thus yielding state-of-the-art results. Code is available at https://***/PRIS-CV/RelMatch. © 2021, CC BY.

关键词： Trees (mathematics)

TLRM: Task-level Relation Module for GNN-based Few-Shot Learning

学校读者我要写书评

暂无评论

TLRM: Task-level Relation Module for GNN-based Few-Shot Lear...

IEEE Visual Communications and Image Processing (VCIP)

作者： Yurong Guo Zhanyu Ma Xiaoxu Li Yuan Dong Pattern Recognition and Intelligent System Lab. Beijing University of Posts and Telecommunications Beijing China Beijing Academy of Artificial Intelligence Beijing China Lanzhou University of Technology Lanzhou China

ISBN: (纸本)9781728173221

Recently, graph neural networks (GNNs) have shown powerful ability to handle few-shot classification problem, which aims at classifying unseen samples when trained with limited labeled samples per class. GNN-based few-shot learning architectures mostly replace traditional metric with a learnable GNN. In the GNN, the nodes are set as the samples' embedding, and the relationship between two connected nodes can be obtained by a network, the input of which is the difference of their embedding features. We consider this method of measuring relation of samples only models the sample-to-sample relation, while neglects the specificity of different tasks. That is, this method of measuring relation does not take the task-level information into account. To this end, we propose a new relation measure method, namely the task-level relation module (TLRM), to explicitly model the task-level relation of one sample to all the others. The proposed module captures the relation representations between nodes by considering the sample-to-task instead of sample-to-sample embedding features. We conducted extensive experiments on four benchmark datasets: mini-ImageNet, tiered-ImageNet, CUB-200-2011, and CIFAR-FS. Experimental results demonstrate that the proposed module is effective for GNN-based few-shot learning.

关键词： Measurement Visual communication Image processing Benchmark testing Graph neural networks Task analysis

Fgsd: A dataset for fine-grained ship detection in high resolution satellite images

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Chen, Kaiyan Wu, Ming Liu, Jiaming Zhang, Chuang Pattern Recognition and Intelligent System Lab Beijing University of Posts and Telecommunications Beijing China

Ship detection using high-resolution remote sensing images is an important task, which contribute to sea surface regulation. The complex background and special visual angle make ship detection relies in high quality datasets to a certain extent. However, there is few works on giving both precise classification and accurate location of ships in existing ship detection datasets. To further promote the research of ship detection, we introduced a new fine-grained ship detection datasets, which is named as FGSD. The dataset collects high-resolution remote sensing images that containing ship samples from multiple large ports around the world. Ship samples were fine categorized and annotated with both horizontal and rotating bounding boxes. To further detailed the information of the dataset, we put forward a new representation method of ships orientation. For future research, the dock as a new class was annotated in the dataset. Besides, rich information of images were provided in FGSD, including the source port, resolution and corresponding GoogleEarth's resolution level of each image. As far as we know, FGSD is the most comprehensive ship detection dataset currently and it'll be available soon. Some baselines for FGSD are also provided in this paper. You are also welcome to contact us at ckyan@*** to get the dataset. Copyright © 2020, The Authors. All rights reserved.

关键词： Remote sensing

C-DLinkNet: Considering Multi-level semantic features for human parsing

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Lu, Yu Feng, Muyan Wu, Ming Zhang, Chuang Pattern Recognition and Intelligent System Lab Beijing University of Posts and Telecommunications Beijing China

Human parsing is an essential branch of semantic segmentation, which is a fine-grained semantic segmentation task to identify the constituent parts of human. The challenge of human parsing is to extract effective semantic features to resolve deformation and multi-scale variations. In this work, we proposed an end-to-end model called C-DLinkNet based on LinkNet, which contains a new module named Smooth Module to combine the multi-level features in Decoder part. C-DLinkNet is capable of producing competitive parsing performance compared with the state-of-the-art methods with smaller input sizes and no additional information, i.e., achiving mIoU=53.05 on the validation set of LIP dataset. Copyright © 2020, The Authors. All rights reserved.

关键词： Semantics

IU-Module: Intersection and Union Module for Fine-Grained Visual Classification

学校读者我要写书评

暂无评论

IU-Module: Intersection and Union Module for Fine-Grained Vi...

IEEE International Conference on Multimedia and Expo (ICME)

作者： Yixiao Zheng Dongliang Chang Jiyang Xie Zhanyu Ma Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications

ISBN: (数字)9781728113319

ISBN: (纸本)9781728113326

A predominant viewpoint in previous works of fine-grained visual classification (FGVC) is to the localize discriminative parts by auxiliary networks and extract the part-based finegrained features for classification. In this paper, we propose a simple yet effective approach by introducing an intersection and union module (IU-Module). The IU-Module aims to capture more discriminative features by 1) dividing features into distinct groups, 2) sharing parts of interests within each group, and 3) adding a differentiation loss to reduce the similarity among those grouped feature channels. Without adding any new learnable parameters, the proposed approach imposes two straightforward operations, namely channel intersection (CI) and channel union (CU) operations, on the convolutional features and achieves competitive results compared with the state-of-the-art methods. Experimental results on three publicly available FGVC datasets show the effectiveness of the IU-Module. Ablation studies and visualizations are also provided to make further demonstrations.

关键词： Feature extraction Visualization Task analysis Automobiles Semantics Toy manufacturing industry Atmospheric modeling

Survey on Deep Face Restoration: From Non-blind to Blind and Beyond

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Li, Wenjie Wang, Mei Zhang, Kai Li, Juncheng Li, Xiaoming Zhang, Yuhang Gao, Guangwei Deng, Weihong Lin, Chia-Wen The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing China The Computer Vision Lab ETH Zürich Zürich Switzerland The School of Communication and Information Engineering Shanghai University Shanghai China The Nanyang Technological University Singapore The Intelligent Visual Information Perception Laboratory Institute of Advanced Technology Nanjing University of Posts and Telecommunications Nanjing China The Department of Electrical Engineering National Tsing Hua University Hsinchu Taiwan

Face restoration (FR) is a specialized field within image restoration that aims to recover low-quality (LQ) face images into high-quality (HQ) face images. Recent advances in deep learning technology have led to significant progress in FR methods. In this paper, we begin by examining the prevalent factors responsible for real-world LQ images and introduce degradation techniques used to synthesize LQ images. We also discuss notable benchmarks commonly utilized in the field. Next, we categorize FR methods based on different tasks and explain their evolution over time. Furthermore, we explore the various facial priors commonly utilized in the restoration process and discuss strategies to enhance their effectiveness. In the experimental section, we thoroughly evaluate the performance of state-of-the-art FR methods across various tasks using a unified benchmark. We analyze their performance from different perspectives. Finally, we discuss the challenges faced in the field of FR and propose potential directions for future advancements. The open-source repository corresponding to this work can be found at https://***/24wenjie-li/Awesome-Face-Restoration. Copyright © 2023, The Authors. All rights reserved.

关键词： Restoration