检索结果-内蒙古大学图书馆

Pathformer3D: A 3D Scanpath Transformer for 360° Images

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Quan, Rong Lai, Yantao Qiu, Mengyu Liang, Dong College of Artificial Intelligence Nanjing University of Aeronautics and Astronautics The Key Laboratory of Brain-Machine Intelligence Technology Ministry of Education Nanjing211106 China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics Nanjing China Shenzhen Research Institute Nanjing University of Aeronautics and Astronautics Shenzhen China

Scanpath prediction in 360° images can help realize rapid rendering and better user interaction in Virtual/Augmented Reality applications. However, existing scanpath prediction models for 360° images execute scanpath prediction on 2D equirectangular projection plane, which always result in big computation error owing to the 2D plane’s distortion and coordinate discontinuity. In this work, we perform scanpath prediction for 360° images in 3D spherical coordinate system and proposed a novel 3D scanpath Transformer named Pathformer3D. Specifically, a 3D Transformer encoder is first used to extract 3D contextual feature representation for the 360° image. Then, the contextual feature representation and historical fixation information are input into a Transformer decoder to output current time step’s fixation embedding, where the self-attention module is used to imitate the visual working memory mechanism of human visual system and directly model the time dependencies among the fixations. Finally, a 3D Gaussian distribution is learned from each fixation embedding, from which the fixation position can be sampled. Evaluation on four panoramic eye-tracking datasets demonstrates that Pathformer3D outperforms the current state-of-the-art methods. Code is available at https://***/lsztzp/ Pathformer3D. Copyright © 2024, The Authors. All rights reserved.

关键词： Forecasting

Recovering from out-of-sample states via inverse dynamics in offline reinforcement learning 23

学校读者我要写书评

暂无评论

Recovering from out-of-sample states via inverse dynamics in...

Proceedings of the 37th International Conference on Neural Information Processing Systems

作者： Ke Jiang Jia-yu Yao Xiaoyang Tan College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics and MIIT Key Laboratory of Pattern Analysis and Machine Intelligence School of Electronic and Computer Engineering Peking University

We deal with the state distributional shift problem commonly encountered in offline reinforcement learning during test, where the agent tends to take unreliable actions at out-of-sample (unseen) states. Our idea is to encourage the agent to follow the so called state recovery principle when taking actions, i.e., besides long-term return, the immediate consequences of the current action should also be taken into account and those capable of recovering the state distribution of the behavior policy are preferred. For this purpose, an inverse dynamics model is learned and employed to guide the state recovery behavior of the new policy. Theoretically, we show that the proposed method helps aligning the transited state distribution of the new policy with the offline dataset at out-of-sample states, without the need of explicitly predicting the transited state distribution, which is usually difficult in high-dimensional and complicated environments. The effectiveness and feasibility of the proposed method is demonstrated with the state-of-the-art performance on the general offline RL benchmarks.

关键词：

XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ni, Xuanfan Cai, Hengyi Wei, Xiaochi Wang, Shuaiqiang Yin, Dawei Li, Piji Nanjing University of Aeronautics and Astronautics Nanjing China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing China Institute of Computing Technology CAS Beijing China Baidu Inc. Beijing China

Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly cater to long-text comprehension by expanding the input of traditional tasks, which falls short to exhibit the unique characteristics of long-text understanding, including long dependency tasks and longer text length compatible with modern LLMs’ context window size. In this paper, we introduce a benchmark for eXtremely Long context understanding with Long-range dependencies, XL2Bench, which includes three scenarios—Fiction Reading, Paper Reading, and Law Reading—and four tasks of increasing complexity: Memory Retrieval, Detailed Understanding, Overall Understanding, and Open-ended Generation, covering 27 subtasks in English and Chinese. It has an average length of 100K+ words (English) and 200K+ characters (Chinese). Evaluating six leading LLMs on XL2Bench, we find that their performance significantly lags behind human levels. Moreover, the observed decline in performance across both the original and enhanced datasets underscores the efficacy of our approach to mitigating data contamination. © 2024, CC BY.

关键词： Benchmarking

Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Xinrui Geng, Chuanxing Wan, Wenhai Li, Shao-Yuan Chen, Songcan College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence China School of Computer Science and Technology Huazhong University of Science and Technology China

Online continual learning (OCL) requires the models to learn from constant, endless streams of data. While significant efforts have been made in this field, most were focused on mitigating the catastrophic forgetting issue to achieve better classification ability, at the cost of a much heavier training workload. They overlooked that in real-world scenarios, e.g., in high-speed data stream environments, data do not pause to accommodate slow models. In this paper, we emphasize that model throughput– defined as the maximum number of training samples that a model can process within a unit of time – is equally important. It directly limits how much data a model can utilize and presents a challenging dilemma for current methods. With this understanding, we revisit key challenges in OCL from both empirical and theoretical perspectives, highlighting two critical issues beyond the well-documented catastrophic forgetting: (i) Model’s ignorance: the single-pass nature of OCL challenges models to learn effective features within constrained training time and storage capacity, leading to a trade-off between effective learning and model throughput;(ii) Model’s myopia: the local learning nature of OCL on the current task leads the model to adopt overly simplified, task-specific features and excessively sparse classifier, resulting in the gap between the optimal solution for the current task and the global objective. To tackle these issues, we propose the Non-sparse Classifier Evolution framework (NsCE) to facilitate effective global discriminative feature learning with minimal time cost. NsCE integrates non-sparse maximum separation regularization and targeted experience replay techniques with the help of pre-trained models, enabling rapid acquisition of new globally discriminative features. Extensive experiments demonstrate the substantial improvements of our framework in performance, throughput and real-world practicality. © 2024, CC BY.

关键词： Contrastive Learning

Relative Difficulty Distillation for Semantic Segmentation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Liang, Dong Sun, Yue Du, Yun Chen, Songcan Huang, Sheng-Jun MIIT Key Laboratory of Pattern Analysis and Machine Intelligence College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics Nanjing211106 China Nanjing University of Aeronautics Astronautics Shenzhen Research Institute China

Current knowledge distillation (KD) methods primarily focus on transferring various structured knowledge and designing corresponding optimization goals to encourage the student network to imitate the output of the teacher network. However, introducing too many additional optimization objectives may lead to unstable training, such as gradient conflicts. Moreover, these methods ignored the guidelines of relative learning difficulty between the teacher and student networks. Inspired by human cognitive science, in this paper, we redefine knowledge from a new perspective – the student and teacher networks’ relative difficulty of samples, and propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD). We propose a two-stage RDD framework: Teacher-Full Evaluated RDD (TFE-RDD) and Teacher-Student Evaluated RDD (TSE-RDD). RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals, thus avoiding adjusting learning weights for multiple losses. Extensive experimental evaluations using a general distillation loss function on popular datasets such as Cityscapes, CamVid, Pascal VOC, and ADE20k demonstrate the effectiveness of RDD against state-of-the-art KD methods. Additionally, our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound. Codes are available at https://***/sunyueue/***. Copyright © 2024, The Authors. All rights reserved.

关键词： Distillation

Improving Generalization of Deep Neural Networks by Optimum Shifting 39

学校读者我要写书评

暂无评论

Improving Generalization of Deep Neural Networks by Optimum ...

39th Annual AAAI Conference on Artificial intelligence, AAAI 2025

作者： Zhou, Yuyan Li, Ye Feng, Lei Huang, Sheng-Jun MIIT Key Laboratory of Pattern Analysis and Machine Intelligence College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics Nanjing211106 China Information Systems Technology and Design Pillar Singapore University of Technology and Design Singapore

ISBN: (纸本)157735897X

Recent studies showed that the generalization of neural networks is correlated with the sharpness of the loss landscape, and flat minima suggests a better generalization ability than sharp minima. In this paper, we propose a novel method called optimum shifting, which changes the parameters of a neural network from a sharp minimum to a flatter one while maintaining the same training loss value. Our method is based on the observation that when the input and output of a neural network are fixed, the matrix multiplications within the network can be treated as systems of under-determined linear equations, enabling adjustment of parameters in the solution space, which can be simply accomplished by solving a constrained optimization problem. Furthermore, we introduce a practical stochastic optimum shifting technique utilizing the Neural Collapse theory to reduce computational costs and provide more degrees of freedom for optimum shifting. Extensive experiments (including classification and detection) with various deep neural network architectures on benchmark datasets demonstrate the effectiveness of our method. Copyright © 2025, Association for the Advancement of Artificial intelligence (***). All rights reserved.

关键词： Deep neural networks

Topology Reorganized Graph Contrastive Learning with Mitigating Semantic Drift

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Jiaqiang Chen, Songcan College of Computer Science & Technology Nanjing University of Aeronautics & Astronautics Jiangsu Nanjing211106 China MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Nanjing University of Aeronautics & Astronautics Jiangsu Nanjing211106 China

Graph contrastive learning (GCL) is an effective paradigm for node representation learning in graphs. The key components hidden behind GCL are data augmentation and positive-negative pair selection. Typical data augmentations in GCL, such as uniform deletion of edges, are generally blind and resort to local perturbation, which is prone to producing under-diversity views. Additionally, there is a risk of making the augmented data traverse to other classes. Moreover, most methods always treat all other samples as negatives. Such a negative pairing naturally results in sampling bias and likewise may make the learned representation suffer from semantic drift. Therefore, to increase the diversity of the contrastive view, we propose two simple and effective global topological augmentations to compensate current GCL. One is to mine the semantic correlation between nodes in the feature space. The other is to utilize the algebraic properties of the adjacency matrix to characterize the topology by eigen-decomposition. With the help of both, we can retain important edges to build a better view. To reduce the risk of semantic drift, a prototype-based negative pair selection is further designed which can filter false negative samples. Extensive experiments on various tasks demonstrate the advantages of the model compared to the state-of-the-art methods. © 2024, CC BY.

关键词： Graph neural networks

StructSR: Refuse Spurious Details in Real-World Image Super-Resolution 39

学校读者我要写书评

暂无评论

StructSR: Refuse Spurious Details in Real-World Image Super-...

39th Annual AAAI Conference on Artificial intelligence, AAAI 2025

作者： Li, Yachao Liang, Dong Ding, Tianyu Huang, Sheng-Jun College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence Collaborative Innovation Center of Novel Software Technology and Industrialization Nanjing211106 China Microsoft Washington United States

ISBN: (纸本)157735897X

Diffusion-based models have shown great promise in real-world image super-resolution (Real-ISR), but often generate content with structural errors and spurious texture details due to the empirical priors and illusions of these models. To address this issue, we introduce StructSR, a simple, effective, and plug-and-play method that enhances structural fidelity and suppresses spurious details for diffusion-based Real-ISR. StructSR operates without the need for additional fine-tuning, external model priors, or high-level semantic knowledge. At its core is the Structure-Aware Screening (SAS) mechanism, which identifies the image with the highest structural similarity to the low-resolution (LR) input in the early inference stage, allowing us to leverage it as a historical structure knowledge to suppress the generation of spurious details. By intervening in the diffusion inference process, StructSR seamlessly integrates with existing diffusion-based Real-ISR models. Our experimental results demonstrate that StructSR significantly improves the fidelity of structure and texture, improving the PSNR and SSIM metrics by an average of 5.27% and 9.36% on a synthetic dataset (DIV2KVal) and 4.13% and 8.64% on two real-world datasets (RealSR and DRealSR) when integrated with four state-of-the-art diffusion-based Real-ISR methods. © 2025, Association for the Advancement of Artificial intelligence (***). All rights reserved.

关键词：

StructSR: Refuse Spurious Details in Real-World Image Super-Resolution

学校读者我要写书评

暂无评论

arXiv 2025年

Diffusion-based models have shown great promise in real-world image super-resolution (Real-ISR), but often generate content with structural errors and spurious texture details due to the empirical priors and illusions of these models. To address this issue, we introduce StructSR, a simple, effective, and plug-and-play method that enhances structural fidelity and suppresses spurious details for diffusion-based Real-ISR. StructSR operates without the need for additional fine-tuning, external model priors, or high-level semantic knowledge. At its core is the Structure-Aware Screening (SAS) mechanism, which identifies the image with the highest structural similarity to the low-resolution (LR) input in the early inference stage, allowing us to leverage it as a historical structure knowledge to suppress the generation of spurious details. By intervening in the diffusion inference process, StructSR seamlessly integrates with existing diffusion-based Real-ISR models. Our experimental results demonstrate that StructSR significantly improves the fidelity of structure and texture, improving the PSNR and SSIM metrics by an average of 5.27% and 9.36% on a synthetic dataset (DIV2KVal) and 4.13% and 8.64% on two real-world datasets (RealSR and DRealSR) when integrated with four state-of-the-art diffusion-based Real-ISR methods. Code - https://***/LYCEXE/StructSR. Copyright © 2025, The Authors. All rights reserved.

关键词： Inference engines