检索结果-内蒙古大学图书馆

Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Meng, Benyuan Xu, Qianqian Wang, Zitai Cao, Xiaochun Huang, Qingming Institute of Information Engineering CAS China School of Cyber Security University of Chinese Academy of Sciences China Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS China Peng Cheng Laboratory China School of Cyber Science and Tech. Shenzhen Campus of Sun Yat-sen University China School of Computer Science and Tech. University of Chinese Academy of Sciences China Key Laboratory of Big Data Mining and Knowledge Management CAS China

Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at this url. © 2024, CC BY.

关键词： Semantics

Understanding the Robustness of 3D Object Detection with Bird'View Representations in Autonomous Driving

学校读者我要写书评

暂无评论

Understanding the Robustness of 3D Object Detection with Bir...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Zijian Zhu Yichi Zhang Hai Chen Yinpeng Dong Shu Zhao Wenbo Ding Jiachen Zhong Shibao Zheng Institute of Image Communication and Network Engineering Shanghai Jiao Tong University Dept. of Comp. Sci. and Tech. THBI Lab Institute for AI Tsinghua University BNRist Center Zhongguancun Laboratory Key Laboratory of Intelligent Computing and Signal Processing Ministry of Education Information Materials and Intelligent Sensing Laboratory of Anhui Province School of Computer Science and Technology Anhui University SAIC Motor AI Lab

3D object detection is an essential perception task in autonomous driving to understand the environments. The Bird's-Eye-View (BEV) representations have significantly improved the performance of 3D detectors with camera inputs on popular benchmarks. However, there still lacks a systematic understanding of the robustness of these vision-dependent BEV models, which is closely related to the safety of autonomous driving systems. In this paper, we evaluate the natural and adversarial robustness of various representative models under extensive settings, to fully understand their behaviors influenced by explicit BEV features compared with those without BEV. In addition to the classic settings, we propose a 3D consistent patch attack by applying adversarial patches in the 3D space to guarantee the spatiotemporal consistency, which is more realistic for the scenario of autonomous driving. With substantial experiments, we draw several findings: 1) BEV models tend to be more stable than previous methods under different natural conditions and common corruptions due to the expressive spatial representations; 2) BEV models are more vulnerable to adversarial noises, mainly caused by the redundant BEV features; 3) Camera-LiDARfusion models have superior performance under different settings with multi-modal inputs, but BEV fusion model is still vulnerable to adversarial noises of both point cloud and image. These findings alert the safety issue in the applications of BEV detectors and could facilitate the development of more robust models.

关键词：

A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hei, Nailei Guo, Qianyu Wang, Zihao Wang, Yan Wang, Haofen Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University China Engineering Research Center of Robotics Ministry of Education Academy for Engineering & Technology Fudan University China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China Tongji University China College of Design and Innovation Tongji University China

Well-designed prompts have demonstrated the potential to guide text-to-image models in generating amazing images. Although existing prompt engineering methods can provide high-level guidance, it is challenging for novice users to achieve the desired results by manually entering prompts due to a discrepancy between novice-user-input prompts and the model-preferred prompts. To bridge the distribution gap between user input behavior and model training datasets, we first construct a novel Coarse-Fine Granularity Prompts dataset (CFP) and propose a novel User-Friendly Fine-Grained Text Generation framework (UF-FGTG) for automated prompt optimization. For CFP, we construct a novel dataset for text-to-image tasks that combines coarse and fine-grained prompts to facilitate the development of automated prompt generation methods. For UF-FGTG, we propose a novel framework that automatically translates user-input prompts into model-preferred prompts. Specifically, we propose a prompt refiner that continually rewrites prompts to empower users to select results that align with their unique needs. Meanwhile, we integrate image-related loss functions from the text-to-image model into the training process of text generation to generate model-preferred prompts. Additionally, we propose an adaptive feature extraction module to ensure diversity in the generated results. Experiments demonstrate that our approach is capable of generating more visually appealing and diverse images than previous state-of-the-art methods, achieving an average improvement of 5% across six quality and aesthetic metrics. Data and code are available at https://***/Naylenv/UF-FGTG. Copyright © 2024, The Authors. All rights reserved.

关键词： Image enhancement

Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Meng, Benyuan Xu, Qianqian Wang, Zitai Yang, Zhiyong Cao, Xiaochun Huang, Qingming Institute of Information Engineering CAS China School of Cyber Security University of Chinese Academy of Sciences China Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS China Peng Cheng Laboratory China School of Computer Science and Tech. University of Chinese Academy of Sciences China Key Laboratory of Big Data Mining and Knowledge Management CAS China School of Cyber Science and Tech. Sun Yat-sen University Shenzhen Campus China

Diffusion models are powerful generative models, and this capability can also be applied to discrimination. The inner activations of a pre-trained diffusion model can serve as features for discriminative tasks, namely, diffusion feature. We discover that diffusion feature has been hindered by a hidden yet universal phenomenon that we call content shift. To be specific, there are content differences between features and the input image, such as the exact shape of a certain object. We locate the cause of content shift as one inherent characteristic of diffusion models, which suggests the broad existence of this phenomenon in diffusion feature. Further empirical study also indicates that its negative impact is not negligible even when content shift is not visually perceivable. Hence, we propose to suppress content shift to enhance the overall quality of diffusion features. Specifically, content shift is related to the information drift during the process of recovering an image from the noisy input, pointing out the possibility of turning off-the-shelf generation techniques into tools for content shift suppression. We further propose a practical guideline named GATE to efficiently evaluate the potential benefit of a technique and provide an implementation of our methodology. Despite the simplicity, the proposed approach has achieved superior results on various tasks and datasets, validating its potential as a generic booster for diffusion features. Our code is available at this url. © 2024, CC BY.

关键词：

D2SP: Dynamic Dual-Stage Purification Framework for Dual Noise Mitigation in Vision-based Affective Recognition.

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Haoran Mai, Xinji Tao, Zeng Tong, Xuan Lin, Junxiong Wang, Yan Yu, Jiawen Wang, Boyang Yan, Shaoqi Zhao, Qing Zhou, Ziheng Gao, Shuyong Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China School of Information Science and Technology Fudan University Shanghai China Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

The contemporary state-of-the-art of Dynamic Facial Expression Recognition (DFER) technology facilitates remarkable progress by deriving emotional mappings of facial expressions from video content, underpinned by training on voluminous datasets. Yet, the DFER datasets encompass a substantial volume of noise data. Noise arises from low-quality captures that defy logical labeling, and instances that suffer from mislabeling due to annotation bias, engendering two principal types of uncertainty: the uncertainty regarding data usability and the uncertainty concerning label reliability. Addressing the two types of uncertainty, we have meticulously crafted a two-stage framework aiming at Seeking Certain data In extensive Uncertain data (SCIU). This initiative aims to purge the DFER datasets of these uncertainties, thereby ensuring that only clean, verified data is employed in training processes. To mitigate the issue of low-quality samples, we introduce the Coarse-Grained Pruning (CGP) stage, which assesses sample weights and prunes those deemed unusable due to their low weight. For samples with incorrect annotations, the Fine-Grained Correction (FGC) stage evaluates prediction stability to rectify mislabeled data. Moreover, SCIU is conceived as a universally compatible, plug-and-play framework, tailored to integrate seamlessly with prevailing DFER methodologies. Rigorous experiments across prevalent DFER datasets and against numerous benchmark methods substantiates SCIU’s capacity to markedly elevate performance metrics. Copyright © 2024, The Authors. All rights reserved.

关键词： Face recognition

The minority matters: a diversity-promoting collaborative metric learning algorithm 22

学校读者我要写书评

暂无评论

The minority matters: a diversity-promoting collaborative me...

Proceedings of the 36th International Conference on Neural information processing Systems

作者： Shilong Bao Qianqian Xu Zhiyong Yang Yuan He Xiaochun Cao Qingming Huang State Key Laboratory of Information Security Institute of Information Engineering CAS and School of Cyber Security University of Chinese Academy of Sciences Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS School of Computer Science and Tech. University of Chinese Academy of Sciences Alibaba Group School of Cyber Science and Technology Shenzhen Campus Sun Yat-sen University Key Lab. of Intelligent Information Processing Institute of Computing Technology CAS and School of Computer Science and Tech. University of Chinese Academy of Sciences and Key Laboratory of Big Data Mining and Knowledge Management CAS and Peng Cheng Laboratory

ISBN: (纸本)9781713871088

Collaborative Metric Learning (CML) has recently emerged as a popular method in recommendation systems (RS), closing the gap between metric learning and Collaborative Filtering. Following the convention of RS, existing methods exploit unique user representation in their model design. This paper focuses on a challenging scenario where a user has multiple categories of interests. Under this setting, we argue that the unique user representation might induce preference bias, especially when the item category distribution is imbalanced. To address this issue, we propose a novel method called Diversity-Promoting Collaborative Metric Learning (DPCML), with the hope of considering the commonly ignored minority interest of the user. The key idea behind DPCML is to include a multiple set of representations for each user in the system. Based on this embedding paradigm, user preference toward an item is aggregated from different embeddings by taking the minimum item-user distance among the user embedding set. Furthermore, we observe that the diversity of the embeddings for the same user also plays an essential role in the model. To this end, we propose a Diversity Control Regularization Scheme (DCRS) to accommodate the multi-vector representation strategy better. Theoretically, we show that DPCML could generalize well to unseen test data by tackling the challenge of the annoying operation that comes from the minimum value. Experiments over a range of benchmark datasets speak to the efficacy of DPCML.

关键词：

Two-photon imaging of rabbit mammary tissue with 3D reconstruction

学校读者我要写书评

暂无评论

Optik 2024年 309卷

作者： Liu, Meng Hou, Guozhong Li, Yuan Deng, Yanyan Zhang, Yuan Wang, Xiaofeng Xu, Chang Song, Kaiyuan Liu, Xinlong Xia, Yuanqin Zhang, Yong Hebei University of Technology Hebei Key Laboratory of Advanced Laser Technology and Equipment School of Science Tianjin300401 China Research Center for Humanoid Sensing Research Institute of Intelligent Sensing Zhejiang Lab Hangzhou311100 China Cangzhou Normal University School of Physics and Information Engineering Cangzhou061001 China Technical Institute of Physics and Chemistry Chinese Academy of Sciences Beijing100190 China

A broadband two-photon fluorescence (TPF) imaging system, based on a femtosecond oscillator, has been constructed. This system enables TPF imaging of rabbit mammary gland tissue samples by detecting TPF signals generated from the tissue upon excitation by broadband femtosecond pulses. Specifically, TPF imaging utilizes a detection window of 565–615 nm to capture TPF signals from the fluorescent adenine dinucleotide (FAD) present in lactating cells within breast tissue. Moreover, the 3D reconstruction algorithm has been optimized to visualize the 3D structure of the rabbit mammary gland acinar *** optimized three-dimensional reconstruction algorithm can reduce ambiguous signal regions, making the reconstructed three-dimensional structure more reliable and accurate. This is of significant importance for studying the microscopic structure and cell distribution of breast tissue, contributing to a deeper understanding of the tissue's physiological status. © 2024 Elsevier GmbH

关键词： Tissue

Text recognition in real scenarios with a few labeled samples 25

学校读者我要写书评

暂无评论

Text recognition in real scenarios with a few labeled sample...

25th International Conference on Pattern Recognition, ICPR 2020

作者： Lin, Jinghuang Cheng, Zhanzhan Bai, Fan Niu, Yi Pu, Shiliang Zhou, Shuigeng Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai200433 China Hikvision Research Institute China

ISBN: (纸本)9781728188089

Scene text recognition (STR) is still a hot research topic in computer vision field due to its various applications. Existing works mainly focus on learning a general model with a huge number of synthetic text images to recognize unconstrained scene texts, and have achieved substantial progress. However, these methods are not quite applicable in many real-world scenarios where 1) high recognition accuracy is required, while 2) labeled samples are lacked. To tackle this challenging problem, this paper proposes a few-shot adversarial sequence domain adaptation (FASDA) approach to build sequence adaptation between the synthetic source domain (with many synthetic labeled samples) and a specific target domain (with only some or a few real labeled samples). This is done by simultaneously learning each character's feature representation with an attention mechanism and establishing the corresponding character-level latent subspace with adversarial learning. Our approach can maximize the character-level confusion between the source domain and the target domain, thus achieves the sequence-level adaptation with even a small number of labeled samples in the target domain. Extensive experiments on various datasets show that our method significantly outperforms the finetuning scheme, and obtains comparable performance to the state-of-the-art STR methods. © 2020 IEEE

关键词： Computer vision

MVC-VPR: Mutual Learning of Viewpoint Classification and Visual Place Recognition

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gu, Qiwen Wang, Xufei Zhang, Fenglin Zhao, Junqiao Tao, Siyue Ye, Chen Feng, Tiantian Jiang, Changjun Department of Computer Science and Technology School of Electronics and Information Engineering Tongji University Shanghai China The Shanghai Research Institute for Intelligent Autonomous System Tongji University Shanghai China The MOE Key Lab of Embedded System and Service Computing Tongji University Shanghai China School of Surveying and Geo-Informatics Tongji University Shanghai China

Visual Place Recognition (VPR) aims to robustly identify locations by leveraging image retrieval based on descriptors encoded from environmental images. However, drastic appearance changes of images captured from different viewpoints at the same location pose incoherent supervision signals for descriptor learning, which severely hinder the performance of VPR. Previous work proposes classifying images based on manually defined rules or ground truth labels for viewpoints, followed by descriptor training based on the classification results. However, not all datasets have ground truth labels of viewpoints and manually defined rules may be suboptimal, leading to degraded descriptor performance. To address these challenges, we introduce the mutual learning of viewpoint self-classification and VPR. Starting from coarse classification based on geographical coordinates, we progress to finer classification of viewpoints using simple clustering techniques. The dataset is partitioned in an unsupervised manner while simultaneously training a descriptor extractor for place recognition. Experimental results show that this approach almost perfectly partitions the dataset based on viewpoints, thus achieving mutually reinforcing effects. Our method even excels state-of-the-art (SOTA) methods that partition datasets using ground truth labels. Copyright © 2024, The Authors. All rights reserved.

关键词： Self-supervised learning