检索结果-内蒙古大学图书馆

Your Transformer May Not be as Powerful as You Expect 36

学校读者我要写书评

暂无评论

Your Transformer May Not be as Powerful as You Expect

36th Conference on Neural Information Processing Systems, NeurIPS 2022

作者： Luo, Shengjie Li, Shanda Zheng, Shuxin Liu, Tie-Yan Wang, Liwei He, Di National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University China Machine Learning Department School of Computer Science Carnegie Mellon University United States Microsoft Research United States Center for Data Science Peking University China Zhejiang Lab China

ISBN: (纸本)9781713871088

Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far as we know, theoretical understanding of the RPE-based Transformers is largely unexplored. In this work, we mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. One may naturally assume the answer is in the affirmative-RPE-based Transformers are universal function approximators. However, we present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. One key reason lies in that most RPEs are placed in the softmax attention that always generates a right stochastic matrix. This restricts the network from capturing positional information in the RPEs and limits its capacity. To overcome the problem and make the model more powerful, we first present sufficient conditions for RPE-based Transformers to achieve universal function approximation. With the theoretical guidance, we develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions. Therefore, the corresponding URPE-based Transformers become universal function approximators. Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications. The code will be made publicly available at https://***/lsj2408/URPE. © 2022 Neural information processing systems foundation. All rights reserved.

关键词： Encoding (symbols)

RandoMix: A Mixed Sample Data Augmentation Method with Multiple Mixed Modes

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Liu, Xiaoliang Shen, Furao Zhao, Jian Nie, Changhai State Key Laboratory for Novel Software Technology Nanjing University China Department of Computer Science and Technology Nanjing University China School of Artificial Intelligence Nanjing University China School of Electronic Science and Engineering Nanjing University China

Data augmentation plays a crucial role in enhancing the robustness and performance of machine learning models across various domains. In this study, we introduce a novel mixed-sample data augmentation method called RandoMix. RandoMix is specifically designed to simultaneously address robustness and diversity challenges. It leverages a combination of linear and mask mixed modes, introducing flexibility in candidate selection and weight adjustments. We evaluate the effectiveness of RandoMix on diverse datasets, including CIFAR-10/100, Tiny-ImageNet, ImageNet, and Google Speech Commands. Our results demonstrate its superior performance compared to existing techniques such as Mixup, CutMix, Fmix, and ResizeMix. Notably, RandoMix excels in enhancing model robustness against adversarial noise, natural noise, and sample occlusion. The comprehensive experimental results and insights into parameter tuning underscore the potential of RandoMix as a versatile and effective data augmentation method. Moreover, it seamlessly integrates into the training pipeline. Copyright © 2022, The Authors. All rights reserved.

关键词： Machine learning

Newton-Cotes Graph Neural Networks: On the Time Evolution of Dynamic Systems

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Guo, Lingbing Wang, Weiqing Chen, Zhuo Zhang, Ningyu Sun, Zequn Lai, Yixuan Zhang, Qiang Chen, Huajun College of Computer Science and Technology Zhejiang University China Zhejiang University Ant Group Joint Laboratory of Knowledge Graph China ZJU-Hangzhou Global Scientific and Technological Innovation Center China Department of Data Science & AI Monash University Australia State Key Laboratory for Novel Software Technology Nanjing University China

Reasoning system dynamics is one of the most important analytical approaches for many scientific studies. With the initial state of a system as input, the recent graph neural networks (GNNs)-based methods are capable of predicting the future state distant in time with high accuracy. Although these methods have diverse designs in modeling the coordinates and interacting forces of the system, we show that they actually share a common paradigm that learns the integration of the velocity over the interval between the initial and terminal coordinates. However, their integrand is constant w.r.t. time. Inspired by this observation, we propose a new approach to predict the integration based on several velocity estimations with Newton-Cotes formulas and prove its effectiveness theoretically. Extensive experiments on several benchmarks empirically demonstrate consistent and significant improvement compared with the state-of-the-art methods. Copyright © 2023, The Authors. All rights reserved.

关键词： Graph neural networks

Image Data Augmentation for Deep Learning: A Survey

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Yang, Suorong Xiao, Weikang Zhang, Mengchen Guo, Suhan Zhao, Jian Shen, Furao State Key Laboratory for Novel Software Technology Nanjing University China Department of Computer Science and Technology Nanjing University China School of Artificial Intelligence Nanjing University China School of Electronic Science and Engineering Nanjing University China

Deep learning has achieved remarkable results in many computer vision tasks. Deep neural networks typically rely on large amounts of training data to avoid overfitting. However, labeled data for real-world applications may be limited. By improving the quantity and diversity of training data, data augmentation has become an inevitable part of deep learning model training with image data. As an effective way to improve the sufficiency and diversity of training data, data augmentation has become a necessary part of successful application of deep learning models on image data. In this paper, we systematically review different image data augmentation methods. We propose a taxonomy of reviewed methods and present the strengths and limitations of these methods. We also conduct extensive experiments with various data augmentation methods on three typical computer vision tasks, including semantic segmentation, image classification and object detection. Finally, we discuss current challenges faced by data augmentation and future research directions to put forward some useful research guidance. © 2022, CC BY.

关键词： computer vision

Sensitivity Pruner: Filter-Level Deep Neural Network Compression

学校读者我要写书评

暂无评论

SSRN

SSRN 2022年

作者： Guo, Suhan Lai, Bilan Yang, Suorong Zhao, Jian Shen, Furao State Key Laboratory for Novel Software Technology Nanjing University China School of Artificial Intelligence Nanjing University China Department of Computer Science and Technology Nanjing University China School of Electronic Science and Engineering Nanjing University China

As neural networks get deeper to better higher performance, the demand for deployable models on resource-constrained devices also grows. In this work, we propose to achieve model compression using filter-level pruning, which eliminates less sensitive filters. The previous method evaluates neuron importance using the connection matrix gradient in a single shot, which is vulnerable to the sampling bias. Thus, we integrate this measure with the previously proposed "pruning while fine-tuning" framework. Besides classification errors, we introduce the difference between the learned pruning strategy and the single-shot strategy as the second loss component with a self-adjustive hyper-parameter that balances the training goal between improving accuracy and pruning more filters. Experimental results demonstrate the effectiveness of our Sensitivity Pruner (SP), which significantly reduces the computational cost and makes pruned models achieve comparable if not better performance on CIFAR10, CIFAR100, and ILSVRC-12 datasets. © 2022, The Authors. All rights reserved.

关键词： Deep neural networks

In2NeCT: Inter-class and Intra-class Neural Collapse Tuning for Semantic Segmentation of Imbalanced Remote Sensing Images 39

学校读者我要写书评

暂无评论

In2NeCT: Inter-class and Intra-class Neural Collapse Tuning ...

39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025

作者： Shen, Junao Hu, Qiyun Feng, Tian Wang, Xinyu Cui, Hui Wu, Sensen Zhang, Wei School of Software Technology Zhejiang University China State Key Lab of CAD&CG Zhejiang University China Department of Computer Science and Information Technology La Trobe University Australia School of Earth Sciences Zhejiang University China Innovation Center of Yangtze River Delta Zhejiang University China

ISBN: (纸本)157735897X

Remote sensing images (RSIs) are frequently characterized by multi-scale inter-class objects and inconsistently distributed objects due to scene limitations, which would cause a significant data imbalance challenging the corresponding semantic segmentation. Recent methods have leveraged various deep learning techniques to capture high-quality representations for RSI semantic segmentation, but are hardly capable of addressing the afore-mentioned challenge given their limited explorations toward the mechanisms behind the representations. The recently discovered Neural Collapse (NC) phenomenon in computer vision models suggests the simplex equiangular tight frame (ETF) as the optimal representation structure, which has motivated us to observe that the optimal structure of last-layer representations is disrupted and inter-class representations for minor classes tend to become closer to each other beacuse of data imbalance. To address these issues, we propose Inter-class and Intra-class Neural Collapse Tuning (In2NeCT) to optimize the representations that satisfy the simplex ETF, which facilitates the discrimination of inter-class representations and the coherence of intra-class representations. Extensive experiments on three datasets demonstrate that our In2NeCT consistently leads to significant improvements in performance and outperforms the state-of-the-art methods. Copyright © 2025, Association for the Advancement of Artificial Intelligence (***). All rights reserved.

关键词： Semantic Segmentation

Federated Deep Recommendation System Based on Multi-View Feature Embedding

学校读者我要写书评

暂无评论

Federated Deep Recommendation System Based on Multi-View Fea...

International Conference on Data science and Advanced Analytics (DSAA)

作者： Xinna Wang Shunmei Meng Yanran Chen Qiyan Liu Rui Yuan Qianmu Li Department of Computer Science and Engineering Nanjing University of Science and Technology Nanjing China State Key Laboratory for Novel Software Technology Nanjing University Nanjing China Business School University of Auckland Auckland New Zealand Computer Engineering University of Toronto Toronto Canada

ISBN: (纸本)9781665473316

The application of recommendation systems online services is becoming more and more extensive. However, most existing recommendation algorithms centralize multi-party information into a central processor, which may lead to the risk of privacy leakage. And many enterprises or institutions still have the problem that data cannot be shared. Federated learning has been introduced into recommendation algorithms for privacy- aware distributed learning. A typical federated learning is that each client uses local data to train a shared model, the server uses their gradient information to form a global model, and then each client updates. In this paper, we propose a federated deep recommendation algorithm called FedHe-mlp that applies a federated deep learning for data privacy protection, and combines heterogeneous information network (HIN) and matrix factorization technique for better prediction performance. First, each client obtains heterogeneous information through meta- paths, then we combine matrix factorization and heterogeneous information to mine the latent features and heterogeneous features of each client. Finally, We propose a deep neural network that considers features from multiple views. Extensive experiments on three public datasets demonstrate that FedHe- mlp can provide excellent convergence speed, recommendation accuracy, and communication efficiency while preserving data privacy.

关键词： Deep learning Training Data privacy Privacy Analytical models Federated learning Neural networks

Program Repair: Automated vs. Manual

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhang, Quanjun Zhao, Yuan Sun, Weisong Fang, Chunrong Wang, Ziyuan Zhang, Lingming The State Key Laboratory for Novel Software Technology Nanjing University China Nanjing University of Posts and Telecommunications China Department of Computer Science University of Illinois at Urbana-Champaign United States

Various automated program repair (APR) techniques have been proposed to fix bugs automatically in the last decade. Although recent researches have made significant progress on the effectiveness and efficiency, it is still unclear how APR techniques perform with human intervention in a real debugging scenario. To bridge this gap, we conduct an extensive study to compare three state-of-the-art APR tools with manual program repair, and further investigate whether the assistance of APR tools (i.e., repair reports) can improve manual program repair. To that end, we recruit 20 participants for a controlled experiment, resulting in a total of 160 manual repair tasks and a questionnaire survey. The experiment reveals several notable observations that (1) manual program repair may be influenced by the frequency of repair actions sometimes;(2) APR tools are more efficient in terms of debugging time, while manual program repair tends to generate a correct patch with fewer attempts;(3) APR tools can further improve manual program repair regarding the number of correctly-fixed bugs, while there exists a negative impact on the patch correctness;(4) participants are used to consuming more time to identify incorrect patches, while they are still misguided easily;(5) participants are positive about the tools’ repair performance, while they generally lack confidence about the usability in practice. Besides, we provide some guidelines for improving the usability of APR tools (e.g., the misleading information in reports and the observation of feedback). © 2022, CC BY.

关键词： Automation

Lens-to-Lens Bokeh Effect Transformation. NTIRE 2023 Challenge Report

学校读者我要写书评

暂无评论

Lens-to-Lens Bokeh Effect Transformation. NTIRE 2023 Challen...

2023 IEEE/CVF Conference on computer Vision and Pattern Recognition Workshops, CVPRW 2023

作者： Conde, Marcos V. Kolmet, Manuel Seizinger, Tim Bishop, Tom E. Timofte, Radu Kong, Xiangyu Zhang, Dafeng Wu, Jinlong Wang, Fan Peng, Juewen Pan, Zhiyu Liu, Chengxin Luo, Xianrui Sun, Huiqiang Shen, Liao Cao, Zhiguo Xian, Ke Liu, Chaowei Chen, Zigeng Yang, Xingyi Liu, Songhua Jing, Yongcheng Mi, Michael Bi Wang, Xinchao Yang, Zhihao Lian, Wenyi Lai, Siyuan Zhang, Haichuan Hoang, Trung Yazdani, Amirsaeed Monga, Vishal Luo, Ziwei Gustafsson, Fredrik K. Zhao, Zheng Sjölund, Jens Schön, Thomas B. Zhao, Yuxuan Chen, Baoliang Xu, Yiqing Niu, Jixiang Computer Vision Lab CAIDAS IFI University of Würzburg Germany Glass Imaging Inc. China Huazhong University of Science and Technology China Nanyang Technological University Singapore National University of Singapore Singapore University of Sydney Australia Huawei Uppsala University Sweden Department of Electrical Engineering Pennsylvania State University United States Department of Information Technology Uppsala University Sweden Key Laboratory of Intelligent Perception and Image Understanding of the Ministry of Education Xidian University Xi'an China North China University of Technology China

ISBN: (纸本)9798350302493

We present the new Bokeh Effect Transformation Dataset (BETD), and review the proposed solutions for this novel task at the NTIRE 2023 Bokeh Effect Transformation Challenge. Recent advancements of mobile photography aim to reach the visual quality of full-frame cameras. Now, a goal in computational photography is to optimize the Bokeh effect itself, which is the aesthetic quality of the blur in out-of-focus areas of an image. Photographers create this aesthetic effect by benefiting from the lens optical *** aim of this work is to design a neural network capable of converting the the Bokeh effect of one lens to the effect of another lens without harming the sharp foreground regions in the image. For a given input image, knowing the target lens type, we render or transform the Bokeh effect accordingly to the lens properties. We build the BETD using two full-frame Sony cameras, and diverse lens *** the best of our knowledge, we are the first attempt to solve this novel task, and we provide the first BETD dataset and benchmark for it. The challenge had 99 registered participants. The submitted methods gauge the state-of-the-art in Bokeh effect rendering and transformation. © 2023 IEEE.

关键词： Color photography