检索结果-内蒙古大学图书馆

Zero-Shot Audio Captioning Using Soft and Hard Prompts

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Yiming Xu, Xuenan Du, Ruoyi Liu, Haohe Dong, Yuan Tan, Zheng-Hua Wang, Wenwu Ma, Zhanyu The Pattern Recognition and Intelligent System Laboratory School of Artificial Intelligence Beijing University of Posts and Telecommunications Beijing100876 China The Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai200240 China The Department of Electronic Systems Aalborg University Aalborg9220 Denmark The Centre for Vision Speech and Signal Processing University of Surrey GuildfordGU2 7XH United Kingdom

In traditional audio captioning methods, a model is usually trained in a fully supervised manner using a human-annotated dataset containing audio-text pairs and then evaluated on the test sets from the same dataset. Such methods have two limitations. First, these methods are often data-hungry and require time-consuming and expensive human annotations to obtain audio-text pairs. Second, these models often suffer from performance degradation in cross-domain scenarios, i.e., when the input audio comes from a different domain than the training set, which, however, has received little attention. We propose an effective audio captioning method based on the contrastive language-audio pre-training (CLAP) model to address these issues. Our proposed method requires only textual data for training, enabling the model to generate text from the textual feature in the cross-modal semantic space. In the inference stage, the model generates the descriptive text for the given audio from the audio feature by leveraging the audio-text alignment from CLAP. We devise two strategies to mitigate the discrepancy between text and audio embeddings: a mixed-augmentation-based soft prompt and a retrieval-based acoustic-aware hard prompt. These approaches are designed to enhance the generalization performance of our proposed model, facilitating the model to generate captions more robustly and accurately. Extensive experiments on AudioCaps and Clotho benchmarks show the effectiveness of our proposed method, which outperforms other zero-shot audio captioning approaches for in-domain scenarios and outperforms the compared methods for cross-domain scenarios, underscoring the generalization ability of our method. © 2024, CC BY.

关键词： Semantics

Geometry Constrained Weakly Supervised Object Localization 16th

学校读者我要写书评

暂无评论

Geometry Constrained Weakly Supervised Object Localization

16th European Conference on computer vision, ECCV 2020

作者： Lu, Weizeng Jia, Xi Xie, Weicheng Shen, Linlin Zhou, Yicong Duan, Jinming Computer Vision Institute School of Computer Science and Software Engineering Shenzhen University Shenzhen China Shenzhen Institute of Artificial Intelligence and Robotics for Society Shenzhen China School of Computer Science University of Birmingham Birmingham United Kingdom Department of Computer and Information Science University of Macau Zhuhai China

ISBN: (纸本)9783030585730

We propose a geometry constrained network, termed GC-Net, for weakly supervised object localization (WSOL). GC-Net consists of three modules: a detector, a generator and a classifier. The detector predicts the object location defined by a set of coefficients describing a geometric shape (i.e. ellipse or rectangle), which is geometrically constrained by the mask produced by the generator. The classifier takes the resulting masked images as input and performs two complementary classification tasks for the object and background. To make the mask more compact and more complete, we propose a novel multi-task loss function that takes into account area of the geometric shape, the categorical cross-entropy and the negative entropy. In contrast to previous approaches, GC-Net is trained end-to-end and predict object location without any post-processing (e.g. thresholding) that may require additional tuning. Extensive experiments on the CUB-200-2011 and ILSVRC2012 datasets show that GC-Net outperforms state-of-the-art methods by a large margin. Our source code is available at https://***/lwzeng/GC-Net. © 2020, Springer Nature Switzerland AG.

关键词： Geometry

Efficient Multi-Query Oriented Continuous Subgraph Matching

学校读者我要写书评

暂无评论

Efficient Multi-Query Oriented Continuous Subgraph Matching

International Conference on Data Engineering

作者： Ziyi Ma Jianye Yang Xu Zhou Guoqing Xiao Jianhua Wang Liang Yang Kenli Li Xuemin Lin School of Artificial Intelligence Hebei University of Technology China Guangxi Key Laboratory of Machine Vision and Intelligent Control Wuzhou University China Cyberspace Institute of Advanced Technology Guangzhou University China Department of New Networks PengCheng Laboratory China College of Computer Science and Electronic Engineering Hunan University China Shenzhen Research Institute Hunan University China Antai College of Economics and Management Shanghai Jiao Tong University China

ISBN: (数字)9798350317152

ISBN: (纸本)9798350317169

Continuous subgraph matching (CSM) is a critical task for analyzing dynamic graphs and has a wide range of applications, such as merchant fraud detection, cyber-attack hunting, and rumor detection. Although many efficient CSM algorithms have been recently proposed, they are mainly designed to process a single query. However, in some application scenarios, multi-query oriented continuous subgraph matching (MQCSM) may be of more interest. To our knowledge, the two existing solutions to MQCSM are outdated due to unsatisfactory performance. In this paper, we propose MQ-Match, an efficient approach to MQCSM. First, we design a compact yet effective index structure CCG, which maintains the local matching result of vertices in the data graph using a directed graph. The directed edges in CCG can be utilized as an effective pruning rule for the subsequent incremental matching algorithm when expanding a partial match. Then, we develop a computation sharing incremental matching algorithm. In specific, a set of matching trees is constructed based on the depth-first search trees of the query graphs. By utilizing CCG, we conduct subgraph matching for the matching tree to collect the incremental matches for the query graphs, where the common structures of query graphs are matched only once. Extensive experiments show that MQ-Match can achieve 3.1x-7071.4x speedup over the competitors, and consumes much less memory under the majority of the experiment settings.

关键词： Heuristic algorithms Memory management Directed graphs Data engineering Fraud Computational efficiency Indexes

D2NT: A High-Performing Depth-to-Normal Translator

学校读者我要写书评

暂无评论

D2NT: A High-Performing Depth-to-Normal Translator

IEEE International Conference on robotics and Automation (ICRA)

作者： Yi Feng Bohuan Xue Ming Liu Qijun Chen Rui Fan Robotics & Artificial Intelligence Laboratory (RAIL) the State Key Laboratory of Intelligent Autonomous Systems and Frontiers Science Center for Intelligent Autonomous Systems the College of Electronic & Information Engineering Tongji University Shanghai P. R. China Department of Computer Science & Engineering the Hong Kong University of Science and Technology (Guangzhou) Hong Kong SAR P. R. China Robotics & Autonomous Systems Thrust of the Systems Hub the Hong Kong University of Science and Technol-ogy (Guangzhou) Nansha Guangzhou P. R. China

Surface normal holds significant importance in visual environmental perception, serving as a source of rich geometric information. However, the state-of-the-art (SoTA) surface normal estimators (SNEs) generally suffer from an unsatisfactory trade-off between efficiency and accuracy. To resolve this dilemma, this paper first presents a superfast depth-to-normal translator (D2NT), which can directly translate depth images into surface normal maps without calculating 3D coordinates. We then propose a discontinuity-aware gradient (DAG) filter, which adaptively generates gradient convolution kernels to improve depth gradient estimation. Finally, we propose a surface normal refinement module that can easily be integrated into any depth-to-normal SNEs, substantially improving the surface normal estimation accuracy. Our proposed algorithm demonstrates the best accuracy among all other existing real-time SNEs and achieves the SoTA trade-off between efficiency and accuracy.

关键词：

I Can’t Believe It’s Not Better: In-air Movement For Alzheimer Handwriting Synthetic Generation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Bensalah, Asma Parziale, Antonio De Gregorio, Giuseppe Marcelli, Angelo Fornés, Alicia Lladós, Josep Computer Vision Center Universitat Autònoma de Barcelona Spain Computer Science Department Universitat Autònoma de Barcelona Spain DIEM University of Salerno Via Giovanni Paolo II 132 SA Fisciano84084 Italy AI3S Unit CINI National Laboratory of Artificial Intelligence and Intelligent Systems University of Salerno SA Fisciano Italy

During recent years, there here has been a boom in terms of deep learning use for handwriting analysis and recognition. One main application for handwriting analysis is early detection and diagnosis in the health field. Unfortunately, most real case problems still suffer a scarcity of data, which makes difficult the use of deep learning-based models. To alleviate this problem, some works resort to synthetic data generation. Lately, more works are directed towards guided data synthetic generation, a generation that uses the domain and data knowledge to generate realistic data that can be useful to train deep learning models. In this work, we combine the domain knowledge about the Alzheimer’s disease for handwriting and use it for a more guided data generation. Concretely, we have explored the use of in-air movements for synthetic data generation. © 2023, CC BY.

关键词： Recurrent neural networks

Scalable Multi-view Regression Clustering for Large-scale Data

学校读者我要写书评

暂无评论

IEEE Transactions on Circuits and Systems for Video Technology 2025年

作者： Zhao, Xiaowei Fan, Jie Chang, Xiaojun Nie, Feiping Zhang, Qiang Guo, Jun Xidian University State Key Laboratory of Electromechanical Integrated Manufacturing of High-Performance Electronic Equipment Shaanxi Xi’an710071 China Xidian University Center for Complex Systems School of Mechano-Electronic Engineering Shaanxi Xi’an710071 China Northwest University School of Information Science and Technology Shaanxi Xi’an710127 China University of Science and Technology of China School of Information Science and Technology Hefei230026 China Mohamed bin Zayed University of Artificial Intelligence Department of Computer Vision MBZUAI Abu Dhabi United Arab Emirates Xi’an710072 China

In recent years, unsupervised linear regression has attracted attention for its ability to directly capture the mapping relationship between samples and targets. However, existing algorithms can only utilize limited information from a single view, which often leads to unsatisfactory results. To address this problem, we propose a regression clustering model based on multi-view information fusion, called Scalable Multi-view Regression Clustering. This model consists of two parts: intra-view information fusion and inter-view information fusion. In the first part, to capture the local correlations among samples, we propose constructing view-specific bipartite graphs. Unlike traditional single-view and multi-view clustering algorithms, we treat the weights of the bipartite graph as additional features of the samples, thereby directly incorporating the local manifold structure of the samples at the feature level. Furthermore, since the original features of the samples also contain valuable information, we perform unsupervised linear regression separately on the samples represented by the original features and those represented by the bipartite graph weights in each view. The results are then integrated in a weighted manner. In the second part, we propose adaptively weighting the clustering results from each view to capture complementary information across views, thereby enhancing clustering performance. This strategy not only avoids the bipartite graph alignment issue in multi-view clustering but also enables clustering with linear time complexity, making it effective for handling large-scale data. An iterative optimization algorithm is developed to update all variables alternately. Experiments conducted on benchmark datasets demonstrate the superiority of our proposed model. © 2025 IEEE. All rights reserved.

关键词： Multiple linear regression

MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Yaning Wang, Tianyi Yu, Zitong Gao, Zan Shen, Linlin Chen, Shengyong Jinan250353 China Nanyang Technological University 50 Nanyang Ave Block N 4 639798 Singapore School of Computing and Information Technology Great Bay University Dongguan523000 China National Engineering Laboratory for Big Data System Computing Technology Shenzhen University Shenzhen518060 China Jinan250014 China The Key Laboratory of Computer Vision and System Ministry of Education Tianjin University of Technology Tianjin300384 China Computer Vision Institute College of Computer Science and Software Engineering Shenzhen University Shenzhen518060 China National Engineering Laboratory for Big Data System Computing Technology Shenzhen University Shenzhen China Shenzhen Institute of Artificial Intelligence and Robotics for Society Shenzhen518129 China Guangdong Key Laboratory of Intelligent Information Processing Shenzhen University Shenzhen518060 China

The rapid development of photo-realistic face generation methods has raised significant concerns in society and academia, highlighting the urgent need for robust and generalizable face forgery detection (FFD) techniques. Although existing approaches mainly capture face forgery patterns using image modality, other modalities like fine-grained noises and texts are not fully explored, which limits the generalization capability of the model. In addition, most FFD methods tend to identify facial images generated by GAN, but struggle to detect unseen diffusion-synthesized ones. To address the limitations, we aim to leverage the cutting-edge foundation model, contrastive language-image pre-training (CLIP), to achieve generalizable diffusion face forgery detection (DFFD). In this paper, we propose a novel multi-modal fine-grained CLIP (MFCLIP) model, which mines comprehensive and fine-grained forgery traces across image-noise modalities via language-guided face forgery representation learning, to facilitate the advancement of DFFD. Specifically, we devise a fine-grained language encoder (FLE) that extracts fine global language features from hierarchical text prompts. We design a multi-modal vision encoder (MVE) to capture global image forgery embeddings as well as fine-grained noise forgery patterns extracted from the richest patch, and integrate them to mine general visual forgery traces. Moreover, we build an innovative plug-and-play sample pair attention (SPA) method to emphasize relevant negative pairs and suppress irrelevant ones, allowing cross-modality sample pairs to conduct more flexible alignment. Extensive experiments and visualizations show that our model outperforms the state of the arts on different settings like cross-generator, cross-forgery, and cross-dataset evaluations. Copyright © 2024, The Authors. All rights reserved.

关键词： Alignment

Cross Domain Object Detection by Target-Perceived Dual Branch Distillation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： He, Mengzhe Wang, Yali Wu, Jiaxi Wang, Yiru Li, Hanqing Li, Bo Gan, Weihao Wu, Wei Qiao, Yu ShenZhen Key Lab of Computer Vision and Pattern Recognition Shenzhen Institute of Advanced Technology Chinese Academy of Sciences China SenseTime Research University of Chinese Academy of Science China Shanghai AI Laboratory Shanghai China Beihang University China SIAT Branch Shenzhen Institute of Artificial Intelligence and Robotics for Society China

Cross domain object detection is a realistic and challenging task in the wild. It suffers from performance degradation due to large shift of data distributions and lack of instance-level annotations in the target domain. Existing approaches mainly focus on either of these two difficulties, even though they are closely coupled in cross domain object detection. To solve this problem, we propose a novel Target-perceived Dual-branch Distillation (TDD) framework. By integrating detection branches of both source and target domains in a unified teacher-student learning scheme, it can reduce domain shift and generate reliable supervision effectively. In particular, we first introduce a distinct Target Proposal Perceiver between two domains. It can adaptively enhance source detector to perceive objects in a target image, by leveraging target proposal contexts from iterative cross-attention. Afterwards, we design a concise Dual Branch Self Distillation strategy for model training, which can progressively integrate complementary object knowledge from different domains via self-distillation in two branches. Finally, we conduct extensive experiments on a number of widely-used scenarios in cross domain object detection. The results show that our TDD significantly outperforms the state-of-the-art methods on all the benchmarks. Our code and model will be available at here. Copyright © 2022, The Authors. All rights reserved.

关键词： Distillation

NTIRE 2023 Image Shadow Removal Challenge Report

学校读者我要写书评

暂无评论

NTIRE 2023 Image Shadow Removal Challenge Report

2023 IEEE/CVF Conference on computer vision and Pattern Recognition Workshops, CVPRW 2023

作者： Vasluianu, Florin-Alexandru Seizinger, Tim Timofte, Radu Cui, Shuhao Huang, Junshi Tian, Shuman Fan, Mingyuan Zhang, Jiaqi Zhu, Li Wei, Xiaoming Wei, Xiaolin Luo, Ziwei Gustafsson, Fredrik K. Zhao, Zheng Sjölund, Jens Schön, Thomas B. Dong, Xiaoyi Zhang, Xi Sheryl Li, Chenghua Leng, Cong Yeo, Woon-Ha Oh, Wang-Taek Lee, Yeo-Reum Ryu, Han-Cheol Luo, Jinting Jiang, Chengzhi Han, Mingyan Wu, Qi Lin, Wenjie Yu, Lei Li, Xinpeng Jiang, Ting Fan, Haoqiang Liu, Shuaicheng Xu, Shuning Song, Binbin Chen, Xiangyu Zhang, Shile Zhou, Jiantao Zhang, Zhao Zhao, Suiyi Zheng, Huan Gao, Yangcheng Wei, Yanyan Wang, Bo Ren, Jiahuan Luo, Yan Kondo, Yuki Miyata, Riku Yasue, Fuma Naruki, Taito Ukita, Norimichi Chang, Hua-En Yang, Hao-Hsiang Chen, Yi-Chung Chiang, Yuan-Chun Huang, Zhi-Kai Chen, Wei-Ting Chen, I-Hsiang Hsieh, Chia-Hsuan Kuo, Sy-Yen Xianwei, Li Fu, Huiyuan Liu, Chunlin Ma, Huadong Fu, Binglan He, Huiming Wang, Mengjia She, Wenxuan Liu, Yu Nathan, Sabari Kansal, Priya Zhang, Zhongjian Yang, Huabin Wang, Yan Zhang, Yanru Phutke, Shruti S. Kulkarni, Ashutosh Khan, Md Raqib Murala, Subrahmanyam Vipparthi, Santosh Kumar Ye, Heng Liu, Zixi Yang, Xingyi Liu, Songhua Wu, Yinwei Jing, Yongcheng Yu, Qianhao Zheng, Naishan Huang, Jie Long, Yuhang Yao, Mingde Zhao, Feng Zhao, Bowen Ye, Nan Shen, Ning Cao, Yanpeng Xiong, Tong Xia, Weiran Li, Dingwen Xia, Shuchen Computer Vision Lab Ifi Caidas University of Würzburg Germany Computer Vision Lab Eth Zürich Switzerland Meituan Group China Department of Information Technology Uppsala University Sweden Institute of Automation Chinese Academy of Sciences Beijing China Nanjing China Maicro Nanjing China Department of Artificial Intelligence Convergence Sahmyook University Seoul Korea Republic of Megvii Technology China University of Electronic Science and Technology of China China University of Macau China China Toyota Technological Institute Japan Graduate Institute of Electronics Engineering National Taiwan University Taiwan Department of Electrical Engineering National Taiwan University Taiwan Graduate Institute of Communication Engineering National Taiwan University Taiwan ServiceNow United States Beijing University of Post and Teleconmunication Beijing China Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education China Couger Inc. Computer Vision and Pattern Recognition Lab Indian Institute of Technology Ropar Punjab Rupnagar India Research Institute Singapore National University of Singapore Singapore Research Institute Singapore University of Sydney Australia Brain-Inspired Vision Laboratory Information Science and Technology Institution University of Science and Technology of China China State Key Laboratory of Fluid Power and Mechatronic Systems School of Mechanical Engineering Zhejiang University Hangzhou310027 China Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province School of Mechanical Engineering Zhejiang University Hangzhou310027 China South China University of Technology China

ISBN: (纸本)9798350302493

This work reviews the results of the NTIRE 2023 Challenge on Image Shadow Removal. The described set of solutions were proposed for a novel dataset, which captures a wide range of object-light interactions. It consists of 1200 roughly pixel aligned pairs of real shadow free and shadow affected images, captured in a controlled environment. The data was captured in a white-box setup, using professional equipment for lights and data acquisition sensors. The challenge had a number of 144 participants registered, out of which 19 teams were compared in the final ranking. The proposed solutions extend the work on shadow removal, improving over the performance level describing state-of-the-art methods. © 2023 IEEE.

关键词： Data acquisition