检索结果-内蒙古大学图书馆

JarvisIR: Elevating Autonomous Driving perception with Intelligent Image Restoration

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Lin, Yunlong Lin, Zixu Chen, Haoyu Pan, Panwang Li, Chenxin Chen, Sixiang Jin, Yeying Li, Wenbo Ding, Xinghao Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Fujian Xiamen China The Hong Kong University of Science and Technology Guangzhou China Bytedance’s Pico China Tencent China Huawei Noah’s Ark Lab Hong Kong The Chinese University of Hong Kong Hong Kong

Vision-centric perception systems struggle with unpredictable and coupled weather degradations in the wild. Current solutions are often limited, as they either depend on specific degradation priors or suffer from significant domain gaps. To enable robust and autonomous operation in real-world conditions, we propose JarvisIR, a VLM-powered agent that leverages the VLM as a controller to manage multiple expert restoration models. To further enhance system robustness, reduce hallucinations, and improve generalizability in real-world adverse weather, JarvisIR employs a novel two-stage framework consisting of supervised fine-tuning and human feedback alignment. Specifically, to address the lack of paired data in real-world scenarios, the human feedback alignment enables the VLM to be fine-tuned effectively on large-scale real-world data in an unsupervised manner. To support the training and evaluation of JarvisIR, we introduce CleanBench, a comprehensive dataset consisting of high-quality and large-scale instruction-responses pairs, including 150K synthetic entries and 80K real entries. Extensive experiments demonstrate that JarvisIR exhibits superior decision-making and restoration capabilities. Compared with existing methods, it achieves a 50% improvement in the average of all perception metrics on CleanBench-Real. Project page: https://***/. Copyright © 2025, The Authors. All rights reserved.

关键词： Risk perception

DiffRate: Differentiable Compression Rate for efficient Vision Transformers

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Chen, Mengzhao Shao, Wenqi Xu, Peng Lin, Mingbao Zhang, Kaipeng Chao, Fei Ji, Rongrong Qiao, Yu Luo, Ping Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Institute of Artificial Intelligence Xiamen University China OpenGVLab Shanghai AI Laboratory China The University of Hong Kong Hong Kong Tencent Holdings Ltd. China

Token compression aims to speed up large-scale vision transformers (e.g. ViTs) by pruning (dropping) or merging tokens. It is an important but challenging task. Although recent advanced approaches achieved great success, they need to carefully handcraft a compression rate (i.e. number of tokens to remove), which is tedious and leads to sub-optimal performance. To tackle this problem, we propose Differentiable Compression Rate (DiffRate), a novel token compression method that has several appealing properties prior arts do not have. First, DiffRate enables propagating the loss function's gradient onto the compression ratio, which is considered as a non-differentiable hyperparameter in previous work. In this case, different layers can automatically learn different compression rates layer-wisely without extra overhead. Second, token pruning and merging can be naturally performed simultaneously in DiffRate, while they were isolated in previous works. Third, extensive experiments demonstrate that DiffRate achieves state-of-the-art performance. For example, by applying the learned layer-wise compression rates to an off-the-shelf ViT-H (MAE) model, we achieve a 40% FLOPs reduction and a 1.5× throughput improvement, with a minor accuracy drop of 0.16% on ImageNet without fine-tuning, even outperforming previous methods with fine-tuning. Codes and models are available at https://***/OpenGVLab/DiffRate. © 2023, CC BY.

关键词： Merging

One-for-More: Continual Diffusion Model for Anomaly Detection

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Li, Xiaofan Tan, Xin Chen, Zhuo Zhang, Zhizhong Zhang, Ruixin Guo, Rizen Jiang, Guanna Chen, Yulong Qu, Yanyun Ma, Lizhuang Xie, Yuan East China Normal University China Shanghai Innovation Institute China Xiamen University China Shanghai Jiao Tong University China Tencent YouTu Lab China Tencent WeChatPay Lab33 China CATL China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China China

With the rise of generative models, there is a growing interest in unifying all tasks within a generative framework. Anomaly detection methods also fall into this scope and utilize diffusion models to generate or reconstruct normal samples when given arbitrary anomaly images. However, our study found that the diffusion model suffers from severe "faithfulness hallucination" and "catastrophic forgetting", which can’t meet the unpredictable pattern increments. To mitigate the above problems, we propose a continual diffusion model that uses gradient projection to achieve stable continual learning. Gradient projection deploys a regularization on the model updating by modifying the gradient towards the direction protecting the learned knowledge. But as a double-edged sword, it also requires huge memory costs brought by the Markov process. Hence, we propose an iterative singular value decomposition method based on the transitive property of linear representation, which consumes tiny memory and incurs almost no performance loss. Finally, considering the risk of "over-fitting" to normal images of the diffusion model, we propose an anomaly-masked network to enhance the condition mechanism of the diffusion model. For continual anomaly detection, ours achieves first place in 17/18 settings on MVTec and VisA. Code is available at https://***/FuNz-0/One-for-More Copyright © 2025, The Authors. All rights reserved.

关键词： Markov processes

An efficient Dynamic Resource Allocation Framework for Evolutionary Bilevel Optimization

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Xu, Dejun Ye, Kai Zheng, Zimo Zhou, Tao Yen, Gary G. Jiang, Min Department of Artificial Intelligence School of Informatics Xiamen University Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan Ministry of Culture and Tourism Xiamen361005 China School of Electrical and Computer Engineering Oklahoma State University StillwaterOK74075 United States

Bilevel optimization problems are characterized by an interactive hierarchical structure, where the upper level seeks to optimize its strategy while simultaneously considering the response of the lower level. Evolutionary algorithms are commonly used to solve complex bilevel problems in practical scenarios, but they face significant resource consumption challenges due to the nested structure imposed by the implicit lower-level optimality condition. This challenge becomes even more pronounced as problem dimensions increase. Although recent methods have enhanced bilevel convergence through task-level knowledge sharing, further efficiency improvements are still hindered by redundant lower-level iterations that consume excessive resources while generating unpromising solutions. To overcome this challenge, this paper proposes an efficient dynamic resource allocation framework for evolutionary bilevel optimization, named DRCBLEA. Compared to existing approaches, DRC-BLEA introduces a novel competitive quasi-parallel paradigm, in which multiple lower-level optimization tasks, derived from different upperlevel individuals, compete for resources. A continuously updated selection probability is used to prioritize execution opportunities to promising tasks. Additionally, a cooperation mechanism is integrated within the competitive framework to further enhance efficiency and prevent premature convergence. Experimental results compared with chosen state-of-the-art algorithms demonstrate the effectiveness of the proposed method. Specifically, DRCBLEA achieves competitive accuracy across diverse problem sets and real-world scenarios, while significantly reducing the number of function evaluations and overall running time. © 2024, CC BY-NC-ND.

关键词： Structural optimization

E2PNet: event to point cloud registration with spatio-temporal representation learning 23

学校读者我要写书评

暂无评论

E2PNet: event to point cloud registration with spatio-tempor...

Proceedings of the 37th International Conference on Neural Information Processing Systems

作者： Xiuhong Lin Changjie Qiu Zhipeng Cai Siqi Shen Yu Zang Weiquan Liu Xuesheng Bian Matthias Müller Cheng Wang Fujian Key Lab of Sensing and Computing for Smart Cities School of Informatics Xiamen University (XMU) China and Key Laboratory of Multimedia Trusted Perception and Efficient Computing XMU China Intel Labs. Fujian Key Lab of Sensing and Computing for Smart Cities School of Informatics Xiamen University (XMU) China and Yancheng Institute Of Technology China Apple Inc.

Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: https://***/Xmu-qcj/E2PNet.

关键词：

Adaptive feature selection for no-reference image quality assessment by mitigating semantic noise sensitivity 24

学校读者我要写书评

暂无评论

Adaptive feature selection for no-reference image quality as...

Proceedings of the 41st International Conference on Machine Learning

作者： Xudong Li Timin Gao Runze Hu Yan Zhang Shengchuan Zhang Xiawu Zheng Jingyuan Zheng Yunhang Shen Ke Li Yutao Liu Pingyang Dai Rongrong Ji Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University P.R. China School of Information and Electronics Beijing Institute of Technology Beijing China School of Medicine Xiamen University Tencent Youtu Lab. School of Computer Science and Technology Ocean University of China

The current state-of-the-art No-Reference Image Quality Assessment (NR-IQA) methods typically rely on feature extraction from upstream semantic backbone networks, assuming that all extracted features are relevant. However, we make a key observation that not all features are beneficial, and some may even be harmful, necessitating careful selection. Empirically, we find that many image pairs with small feature spatial distances can have vastly different quality scores, indicating that the extracted features may contain quality-irrelevant noise. To address this issue, we propose a Quality-Aware Feature Matching IQA Metric (QFM-IQM) that employs an adversarial perspective to remove harmful semantic noise features from the upstream task. Specifically, QFM-IQM enhances the semantic noise distinguish capabilities by matching image pairs with similar quality scores but varying semantic features as adversarial semantic noise and adaptively adjusting the upstream task's features by reducing sensitivity to adversarial noise perturbation. Furthermore, we utilize a distillation framework to expand the dataset and improve the model's generalization ability. Extensive experiments conducted on eight standard IQA datasets have demonstrated the effectiveness of our proposed QFM-IQM.

关键词：

Shadow Removal by High-Quality Shadow Synthesis

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhong, Yunshan You, Lizhou Zhang, Yuxin Chao, Fei Tian, Yonghong Ji, Rongrong Institute of Artificial Intelligence Department of Artificial Intelligence School of Informatics Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Xiamen361005 China Department of Artificial Intelligence School of Informatics Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Xiamen361005 China The Peng Cheng Laboratory Shenzhen518066 China School of Electronics Engineering and Computer Science Peking University Beijing100871 China Institute of Artificial Intelligence Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China China Xiamen University Xiamen361005 China The Peng Cheng Laboratory Shenzhen518000 China

Most shadow removal methods rely on the invasion of training images associated with laborious and lavish shadow region annotations, leading to the increasing popularity of shadow image synthesis. However, the poor performance also stems from these synthesized images since they are often shadow-inauthentic and details-impaired. In this paper, we present a novel generation framework, referred to as HQSS, for high-quality pseudo shadow image synthesis. The given image is first decoupled into a shadow region identity and a non-shadow region identity. HQSS employs a shadow feature encoder and a generator to synthesize pseudo images. Specifically, the encoder extracts the shadow feature of a region identity which is then paired with another region identity to serve as the generator input to synthesize a pseudo image. The pseudo image is expected to have the shadow feature as its input shadow feature and as well as a real-like image detail as its input region identity. To fulfill this goal, we design three learning objectives. When the shadow feature and input region identity are from the same region identity, we propose a self-reconstruction loss that guides the generator to reconstruct an identical pseudo image as its input. When the shadow feature and input region identity are from different identities, we introduce an inter-reconstruction loss and a cycle-reconstruction loss to make sure that shadow characteristics and detail information can be well retained in the synthesized images. Our HQSS is observed to outperform the state-of-the-art methods on ISTD dataset, Video Shadow Removal dataset, and SRD dataset. The code is available at https://***/zysxmu/HQSS. Copyright © 2022, The Authors. All rights reserved.

关键词： Image reconstruction

Automatic Network Pruning via Hilbert-Schmidt Independence Criterion Lasso under Information Bottleneck Principle

学校读者我要写书评

暂无评论

Automatic Network Pruning via Hilbert-Schmidt Independence C...

International Conference on Computer Vision (ICCV)

作者： Song Guo Lei Zhang Xiawu Zheng Yan Wang Yuchao Li Fei Chao Chenglin Wu Shengchuan Zhang Rongrong Ji Department of Artificial Intelligence School of Informatics Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Peng Cheng Laboratory Samsara Inc Alibaba Group Deep Wisdom Inc Institute of Artificial Intelligence Xiamen University Fujian Engineering Research Center of Trusted Artificial Intelligence Analysis and Application Xiamen University

Most existing neural network pruning methods hand-crafted their importance criteria and structures to prune. This constructs heavy and unintended dependencies on heuristics and expert experience for both the objective and the parameters of the pruning approach. In this paper, we try to solve this problem by introducing a principled and unified framework based on Information Bottleneck (IB) theory, which further guides us to an automatic pruning approach. Specifically, we first formulate the channel pruning problem from an IB perspective, and then implement the IB principle by solving a Hilbert-Schmidt Independence Criterion (HSIC) Lasso problem under certain conditions. Based on the theoretical guidance, we then provide an automatic pruning scheme by searching for global penalty coefficients. Verified by extensive experiments, our method yields state-of-the-art performance on various benchmark networks and datasets. For example, with VGG-16, we achieve a 60%-FLOPs reduction by removing 76% of the parameters, with an improvement of 0.40% in top-1 accuracy on CIFAR-10. With ResNet-50, we achieve a 56%-FLOPs reduction by removing 50% of the parameters, with a small loss of 0.08% in the top-1 accuracy on ImageNet. The code is available at https://***/sunggo/APIB.

关键词：

AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Li, Lijiang Li, Huixia Zheng, Xiawu Wu, Jie Xiao, Xuefeng Wang, Rui Zheng, Min Pan, Xin Chao, Fei Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Department of Artificial Intelligence School of Informatics Xiamen University China ByteDance Inc China Peng Cheng Laboratory China Institute of Artificial Intelligence Xiamen University China Fujian Engineering Research Center of Trusted Artificial Intelligence Analysis and Application Xiamen University China

Diffusion models are emerging expressive generative models, in which a large number of time steps (inference steps) are required for a single image generation. To accelerate such tedious process, reducing steps uniformly is considered as an undisputed principle of diffusion models. We consider that such a uniform assumption is not the optimal solution in practice;i.e., we can find different optimal time steps for different models. Therefore, we propose to search the optimal time steps sequence and compressed model architecture in a unified framework to achieve effective image generation for diffusion models without any further training. Specifically, we first design a unified search space that consists of all possible time steps and various architectures. Then, a two stage evolutionary algorithm is introduced to find the optimal solution in the designed search space. To further accelerate the search process, we employ FID score between generated and real samples to estimate the performance of the sampled examples. As a result, the proposed method is (i).training-free, obtaining the optimal time steps and model architecture without any training process;(ii). orthogonal to most advanced diffusion samplers and can be integrated to gain better sample quality. (iii). generalized, where the searched time steps and architectures can be directly applied on different diffusion models with the same guidance scale. Experimental results show that our method achieves excellent performance by using only a few time steps, e.g. 17.86 FID score on ImageNet 64 × 64 with only four steps, compared to 138.66 with DDIM. The code is available at https://***/lilijiangg/AutoDiffusion. Copyright © 2023, The Authors. All rights reserved.

关键词： Diffusion