检索结果-内蒙古大学图书馆

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Wang, Mingze Wang, Jinbo He, Haotian Wang, Zilin Huang, Guanhua Xiong, Feiyu Li, Zhiyu Weinan, E. Wu, Lei School of Mathematical Sciences Peking University China Center for Machine Learning Research Peking University China China AI for Science Institute China School of Data Science University of Science and Technology of China China ByteDance Research China

In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with generic base optimizers without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). Surprisingly, IRE also achieves a 2× speed-up compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in sharpness-aware minimization (SAM). © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Optimal bounds for p sensitivity sampling via 2 augmentation 41

Optimal bounds for p sensitivity sampling via 2 augmentation

引用

41st International Conference on machine learning, ICML 2024

作者： Munteanu, Alexander Omlor, Simon Dortmund Data Science Center Faculties of Statistics and Computer Science TU Dortmund University Dortmund Germany Faculty of Statistics TU Dortmund University Dortmund Germany Lamarr-Institute for Machine Learning and Artificial Intelligence Dortmund Germany

data subsampling is one of the most natural methods to approximate a massively large data set by a small representative proxy. In particular, sensitivity sampling received a lot of attention, which samples points proportional to an individual importance measure called sensitivity. This framework reduces in very general settings the size of data to roughly the VC dimension d times the total sensitivity S while providing strong (1 ± Ε) guarantees on the quality of approximation. The recent work of Woodruff & Yasuda (2023c) improved substantially over the general Õ(Ε−2Sd) bound for the important problem of p subspace embeddings to Õ(Ε−2S2/p) for p ∈ [1, 2]. Their result was subsumed by an earlier Õ(Ε−2Sd1−p/2) bound which was implicitly given in the work of Chen & Derezinski (2021). We show that their result is tight when sampling according to plain p sensitivities. We observe that by augmenting the p sensitivities by 2 sensitivities, we obtain better bounds improving over the aforementioned results to optimal linear Õ(Ε−2(S+d)) = Õ(Ε−2d) sampling complexity for all p ∈ [1, 2]. In particular, this resolves an open question of Woodruff & Yasuda (2023c) in the affirmative for p ∈ [1, 2] and brings sensitivity subsampling into the regime that was previously only known to be possible using Lewis weights (Cohen & Peng, 2015). As an application of our main result, we also obtain an Õ(Ε−2µd) sensitivity sampling bound for logistic regression, where µ is a natural complexity measure for this problem. This improves over the previous Õ(Ε−2µ2d) bound of Mai et al. (2021) which was based on Lewis weights subsampling. Copyright 2024 by the author(s)

关键词： Importance sampling

来源：评论

学校读者我要写书评

暂无评论

Turnstile p leverage score sampling with applications 41

Turnstile p leverage score sampling with applications

引用

41st International Conference on machine learning, ICML 2024

The turnstile data stream model offers the most flexible framework where data can be manipulated dynamically, i.e., rows, columns, and even single entries of an input matrix can be added, deleted, or updated multiple times in a data stream. We develop a novel algorithm for sampling rows ai of a matrix A ∈ Rn×d, proportional to their p norm, when A is presented in a turnstile data stream. Our algorithm not only returns the set of sampled row indexes, it also returns slightly perturbed rows ãi ≈ ai, and approximates their sampling probabilities up to Ε relative error. When combined with preconditioning techniques, our algorithm extends to p leverage score sampling over turnstile data streams. With these properties in place, it allows us to simulate subsampling constructions of coresets for important regression problems to operate over turnstile data streams with very little overhead compared to their respective off-line subsampling algorithms. For logistic regression, our framework yields the first algorithm that achieves a (1 + Ε) approximation and works in a turnstile data stream using polynomial sketch/subsample size, improving over O(1) approximations, or exp(1/Ε) sketch size of previous work. We compare experimentally to plain oblivious sketching and plain leverage score sampling algorithms for p and logistic regression. Copyright 2024 by the author(s)

关键词： data streams

来源：评论

学校读者我要写书评

暂无评论

On the Effectiveness of Heterogeneous Ensemble Methods for Re-Identification 23

On the Effectiveness of Heterogeneous Ensemble Methods for R...

引用

23rd IEEE International Conference on machine learning and Applications, ICMLA 2024

作者： Klüttermann, Simon Rutinowski, Jérôme Polachowski, Frederik Nguyen, Anh Grimme, Britta Roidl, Moritz Müller, Emmanuel Paderborn University Paderborn Germany Tu Dortmund University Dortmund Germany Lamarr Institute for Machine Learning and Artificial Intelligence Dortmund Germany Research Center Trustworthy Data Science and Security Dortmund Germany

ISBN: (纸本)9798350374889

In this contribution, we introduce a novel ensemble method for the re-identification of industrial entities, using images of chipwood pallets and galvanized metal plates as dataset examples. Our algorithms replace commonly used, complex siamese neural networks with an ensemble of simplified, rudimentary models, providing wider applicability, especially in hardware-restricted scenarios. Each ensemble sub-model uses different types of extracted features of the given data as its input, allowing for the creation of effective ensembles in a fraction of the training duration needed for more complex state-of-the-art models. We reach state-of-the-art performance at our task, with a Rank-1 accuracy of over 77% and a Rank-10 accuracy of over 99%, and introduce five distinct feature extraction approaches, and study their combination using different ensemble methods. © 2024 IEEE.

关键词： Contrastive learning

来源：评论

学校读者我要写书评

暂无评论

An Improved Finite-time Analysis of Temporal Difference learning with Deep Neural Networks 41

An Improved Finite-time Analysis of Temporal Difference Lear...

引用

41st International Conference on machine learning, ICML 2024

作者： Ke, Zhifa Wen, Zaiwen Zhang, Junyu Center for Data Science Peking University China Beijing International Center for Mathematical Research Center for Machine Learning Research Changsha Institute for Computing and Digital Economy Beijing China Department of Industrial Systems Engineering and Management National University of Singapore Singapore

Temporal difference (TD) learning algorithms with neural network function parameterization have well-established empirical success in many practical large-scale reinforcement learning tasks. However, theoretical understanding of these algorithms remains challenging due to the nonlinearity of the action-value approximation. In this paper, we develop an improved non-asymptotic analysis of the neural TD method with a general L-layer neural network. New proof techniques are developed and an improved new Õ(ϵ-1) sample complexity is derived. To our best knowledge, this is the first finite-time analysis of neural TD that achieves an Õ(ϵ-1) complexity under the Markovian sampling, as opposed to the best known Õ(ϵ-2) complexity in the existing literature. Copyright 2024 by the author(s)

关键词： Adversarial machine learning

来源：评论

学校读者我要写书评

暂无评论

Can Gaussian Sketching Converge Faster on a Preconditioned Landscape? 41

Can Gaussian Sketching Converge Faster on a Preconditioned L...

引用

41st International Conference on machine learning, ICML 2024

作者： Wang, Yilong Ye, Haishan Dai, Guang Tsang, Ivor W. Center for Intelligent Decision-Making and Machine Learning School of Management Xi'an Jiaotong University China SGIT AI Lab State Grid Corporation of China China Singapore College of Computing and Data Science NTU Singapore

This paper focuses on the large-scale optimization which is very popular in the big data era. The gradient sketching is an important technique in the large-scale optimization. Specifically, the random coordinate descent algorithm is a kind of gradient sketching method with the random sampling matrix as the sketching matrix. In this paper, we propose a novel gradient sketching called GSGD (Gaussian Sketched Gradient Descent). Compared with the classical gradient sketching methods such as the random coordinate descent and SEGA (Hanzely et al., 2018), our GSGD does not require the importance sampling but can achieve a fast convergence rate matching the ones of these methods with importance sampling. Furthermore, if the objective function has a non-smooth regularization term, our GSGD can also exploit the implicit structure information of the regularization term to achieve a fast convergence rate. Finally, our experimental results substantiate the effectiveness and efficiency of our algorithm. Copyright 2024 by the author(s)

关键词： Big data

来源：评论

学校读者我要写书评

暂无评论

Double Stochasticity Gazes Faster: Snap-Shot Decentralized Stochastic Gradient Tracking Methods 41

Double Stochasticity Gazes Faster: Snap-Shot Decentralized S...

引用

41st International Conference on machine learning, ICML 2024

作者： Di, Hao Ye, Haishan Chang, Xiangyu Dai, Guang Tsang, Ivor W. Center for Intelligent Decision-Making and Machine Learning School of Management Xi'an Jiaotong University China SGIT AI Lab State Grid Corporation of China China College of Computing and Data Science NTU Singapore Singapore

In decentralized optimization, m agents form a network and only communicate with their neighbors, which gives advantages in data ownership, privacy, and scalability. At the same time, decentralized stochastic gradient descent (SGD) methods, as popular decentralized algorithms for training large-scale machine learning models, have shown their superiority over centralized counterparts. Distributed stochastic gradient tracking (DSGT) (Pu & Nedić, 2021) has been recognized as the popular and state-of-the-art decentralized SGD method due to its proper theoretical guarantees. However, the theoretical analysis of DSGT (Koloskova et al., 2021) shows that its iteration complexity is (equation presented) where the doubly stochastic matrix W represents the network topology and CW is a parameter that depends on W. Thus, it indicates that the convergence property of DSGT is heavily affected by the topology of the communication network. To overcome the weakness of DSGT, we resort to the snapshot gradient tracking skill and propose two novel algorithms, snap-shot DSGT (SS DSGT) and accelerated snap-shot DSGT (ASS DSGT). We further justify that SS DSGT exhibits a lower iteration complexity compared to DSGT in the general communication network topology. Additionally, ASS DSGT matches DSGT's iteration complexity (equation presented) under the same conditions as DSGT. Numerical experiments validate SS DSGT's superior performance in the general communication network topology and exhibit better practical performance of ASS DSGT on the specified W compared to DSGT. Copyright 2024 by the author(s)

关键词： Network topology

来源：评论

学校读者我要写书评

暂无评论

Noisy Recovery in Unlimited Sampling via Adaptive Modulo Representations

Noisy Recovery in Unlimited Sampling via Adaptive Modulo Rep...

引用

2024 International Workshop on the Theory of Computational Sensing and its Applications to Radar, Multimodal Sensing and Imaging, CoSeRa 2024

作者： Patricio, Felipe Pagginelli Catala, Paul Krahmer, Felix Tu Munich Dept. of Mathematics Garching Germany Ibmi Helmholtz Munich Neuherberg Germany Institute Tu Munich Munich Center for Machine Learning Dept. of Mathematics&Munich Data Science Garching Germany

ISBN: (纸本)9798350365504

Recent works put forth the Unlimited Sensing Framework (USF), a novel approach to analog-to-digital conversion for high dynamic range sensing. It addresses the saturation phenomenon that commonly arises when physical measurements exceed the dynamic range of a sensor, yielding permanent loss of the input data. However, the USF still has some limitations when dealing with random noise. In the present paper, we propose a novel iterative method to tackle unlimited sensing in a noisy setting. In one step, our approach applies local transformations of the range to remove strong artifacts caused by the noise on local subdivisions of the domain. In the following step, the signal is then approximated via a least squares method. These two types of steps are then alternated. We illustrate the performances of our algorithm in high noise regime. © 2024 IEEE.

关键词： Inverse problems

来源：评论

学校读者我要写书评

暂无评论

The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks 38

The Challenges of the Nonlinear Regime for Physics-Informed ...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Bonfanti, Andrea Bruno, Giuseppe Cipriani, Cristina BMW AG Basque Center for Applied Mathematics University of the Basque Country Digital Campus Munich Spain BMW AG Digital Campus Munich Germany Technical University of Munich Munich Center for Machine Learning Munich Data Science Institute Germany

The Neural Tangent Kernel (NTK) viewpoint is widely employed to analyze the training dynamics of overparameterized Physics-Informed Neural Networks (PINNs). However, unlike the case of linear Partial Differential Equations (PDEs), we show how the NTK perspective falls short in the nonlinear scenario. Specifically, we establish that the NTK yields a random matrix at initialization that is not constant during training, contrary to conventional belief. Another significant difference from the linear regime is that, even in the idealistic infinite-width limit, the Hessian does not vanish and hence it cannot be disregarded during training. This motivates the adoption of second-order optimization methods. We explore the convergence guarantees of such methods in both linear and nonlinear cases, addressing challenges such as spectral bias and slow convergence. Every theoretical result is supported by numerical examples with both linear and nonlinear PDEs, and we highlight the benefits of second-order methods in benchmark test cases. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Robust Autonomous Vehicle Pursuit Without Expert Steering Labels

引用

IEEE Robotics and Automation Letters 2023年第10期8卷 6595-6602页

作者： Pan, Jiaxin Zhou, Changyao Gladkova, Mariia Khan, Qadeer Cremers, Daniel Technical University of Munich Computer Vision Group Garching85748 Germany Munich Data Science Institute Garching85748 Germany Munich Center for Machine Learning Munchen80333 Germany University of Oxford OxfordOX1 3AZ United Kingdom

In this work, we present a learning method for both lateral and longitudinal motion control of an ego-vehicle for the task of vehicle pursuit. The car being controlled does not have a pre-defined route, rather it reactively adapts to follow a target vehicle while maintaining a safety distance. To train our model, we do not rely on steering labels recorded from an expert driver, but effectively leverage a classical controller as an offline label generation tool. In addition, we account for the errors in the predicted control values, which can lead to a loss of tracking and catastrophic crashes of the controlled vehicle. To this end, we propose an effective data augmentation approach, which allows to train a network that is capable of handling different views of the target vehicle. During the pursuit, the target vehicle is firstly localized using a Convolutional Neural Network. The network takes a single RGB image along with cars' velocities and estimates target vehicle's pose with respect to the ego-vehicle. This information is then fed to a Multi-Layer Perceptron, which regresses the control commands for the ego-vehicle, namely throttle and steering angle. We extensively validate our approach using the CARLA simulator on a wide range of terrains. Our method demonstrates real-time performance, robustness to different scenarios including unseen trajectories and high route completion. © 2016 IEEE.

关键词： Deep learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：