检索结果-内蒙古大学图书馆

The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Bonfanti, Andrea Bruno, Giuseppe Cipriani, Cristina BMW AG Basque Center for Applied Mathematics University of the Basque Country Digital Campus Munich Spain BMW AG Digital Campus Munich Germany Technical University of Munich Munich Center for Machine Learning Munich Data Science Institute Germany

The Neural Tangent Kernel (NTK) viewpoint is widely employed to analyze the training dynamics of overparameterized Physics-Informed Neural Networks (PINNs). However, unlike the case of linear Partial Differential Equations (PDEs), we show how the NTK perspective falls short in the nonlinear scenario. Specifically, we establish that the NTK yields a random matrix at initialization that is not constant during training, contrary to conventional belief. Another significant difference from the linear regime is that, even in the idealistic infinite-width limit, the Hessian does not vanish and hence it cannot be disregarded during training. This motivates the adoption of second-order optimization methods. We explore the convergence guarantees of such methods in both linear and nonlinear cases, addressing challenges such as spectral bias and slow convergence. Every theoretical result is supported by numerical examples with both linear and nonlinear PDEs, and we highlight the benefits of second-order methods in benchmark test cases. © 2024, CC BY.

关键词： Numerical methods

PDEFORMER: TOWARDS A FOUNDATION MODEL FOR ONE-DIMENSIONAL PARTIAL DIFFERENTIAL EQUATIONS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ye, Zhanhong Huang, Xiang Chen, Leheng Liu, Hongsheng Wang, Zidong Dong, Bin Beijing International Center for Mathematical Research Peking University Beijing China Central Software Institute Huawei Technologies Co. Ltd Hangzhou China Beijing International Center for Mathematical Research The New Cornerstone Science Laboratory Peking University Beijing China Center for Machine Learning Research Peking University Beijing China

This paper introduces PDEformer, a neural solver for partial differential equations (PDEs) capable of simultaneously addressing various types of PDEs. We propose to represent the PDE in the form of a computational graph, facilitating the seamless integration of both symbolic and numerical information inherent in a PDE. A graph Transformer and an implicit neural representation (INR) are employed to generate mesh-free predicted solutions. Following pretraining on data exhibiting a certain level of diversity, our model achieves zero-shot accuracies on benchmark datasets that is comparable to those of specifically trained expert models. Additionally, PDEformer demonstrates promising results in the inverse problem of PDE coefficient recovery. © 2024, CC BY.

关键词： Partial differential equations

On the Origins of Linear Representations in Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Jiang, Yibo Rajendran, Goutham Ravikumar, Pradeep Aragam, Bryon Veitch, Victor Department of Computer Science University of Chicago United States Machine Learning Department Carnegie Mellon University United States Booth School of Business University of Chicago United States Department of Statistics University of Chicago United States Data Science Institute University of Chicago United States

Recent works have argued that high-level semantic concepts are encoded "linearly" in the representation space of large language models. In this work, we study the origins of such linear representations. To that end, we introduce a simple latent variable model to abstract and formalize the concept dynamics of the next token prediction. We use this formalism to show that the next token prediction objective (softmax with cross-entropy) and the implicit bias of gradient descent together promote the linear representation of concepts. Experiments show that linear representations emerge when learning from data matching the latent variable model, confirming that this simple structure already suffices to yield linear representations. We additionally confirm some predictions of the theory using the LLaMA-2 large language model, giving evidence that the simplified model yields generalizable insights. Copyright © 2024, The Authors. All rights reserved.

关键词： Forecasting

Weakly Supervised Panoptic Segmentation for Defect-Based Grading of Fresh Produce

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Knott, Manuel Odion, Divinefavour Sontakke, Sameer Karwa, Anup Defraeye, Thijs Empa Swiss Federal Laboratories for Materials Science and Technology St. Gallen Switzerland Swiss Data Science Center ETH Zurich EPFL Switzerland Institute for Machine Learning Department of Computer Science ETH Zurich Switzerland Constructor University Bremen Germany Innoterra BioScience Private Limited Mumbai India

Visual inspection for defect grading in agricultural supply chains is crucial but traditionally labor-intensive and error-prone. Automated computer vision methods typically require extensively annotated datasets, which are often unavailable in decentralized supply chains. We address this challenge by evaluating the Segment Anything Model (SAM) to generate dense panoptic segmentation masks from sparse annotations. These dense predictions are then used to train a supervised panoptic segmentation model. Focusing on banana surface defects (bruises and scars), we validate our approach using 476 field images annotated with 1440 defects. While SAM-generated masks generally align with human annotations, substantially reducing annotation effort, we explicitly identify failure cases associated with specific defect sizes and shapes. Despite these limitations, our approach offers practical estimates of defect number and relative size from panoptic masks, underscoring the potential and current boundaries of foundation models for defect quantification in low-data agricultural scenarios. GitHub: https://***/manuelknott/banana-defect-segmentation © 2024, CC BY.

关键词： Image segmentation

Optical Character Recognition (OCR) in Handwritten Characters Using Convolutional Neural Networks to Assist in Exam Reader System 2

学校读者我要写书评

暂无评论

Optical Character Recognition (OCR) in Handwritten Character...

2nd International Conference on Advancement in Computation and Computer Technologies, InCACCT 2024

作者： Lekshmy, P.L. Velmurugan, S. Kumari, Indra Kayalvili, S. Teja Sree, B. Karthik Kumar, P. LBS Institute of Technology for Women Department of Computer Science and Engineering Kerala India T.J.S. Engineering College Department of Electronics and Communication Engineering Tamil Nadu Chennai India Department of Machine Learning Data Research Applied AI Daejeon Korea Republic of Department of Artificial Intelligence Tamilnadu Erode India S.R.K.R. Engineering College Department of Information Technology Andhra Pradesh Chinaamiram Bhimavaram India Coimbatore India

ISBN: (纸本)9798350371314

This work aimed to develop a character recognition method to facilitate the correction of answer cards in the Multiprova software through the development of a response card analysis flow that would culminate in the recognition of letters written by students and automatic correction of the tests. To isolate and identify the answers written on the answer cards, image segmentation techniques were used based on fixed marks printed on the cards. To recognize the letters and numbers written on the cards, trained three convolutional neural networks (for digits, letters and true or false). The results achieved (98.84% accuracy for digit CNN, 98.38% accuracy for letter CNN and 99.89% accuracy for true or false CNN) point out a great average success rate. © 2024 IEEE.

关键词： Convolutional neural networks

Adaptive Batch Sizes for Active learning: A Probabilistic Numerics Approach

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Adachi, Masaki Hayakawa, Satoshi Jørgensen, Martin Wan, Xingchen Nguyen, Vu Oberhauser, Harald Osborne, Michael A. Machine Learning Research Group University of Oxford United Kingdom Mathematical Institute University of Oxford United Kingdom Toyota Motor Corporation Japan Department of Computer Science University of Helsinki Finland Amazon United States

Active learning parallelization is widely used, but typically relies on fixing the batch size throughout experimentation. This fixed approach is inefficient because of a dynamic trade-off between cost and speed—larger batches are more costly, smaller batches lead to slower wall-clock run-times—and the tradeoff may change over the run (larger batches are often preferable earlier). To address this trade-off, we propose a novel Probabilistic Numerics framework that adaptively changes batch sizes. By framing batch selection as a quadrature task, our integration-error-aware algorithm facilitates the automatic tuning of batch sizes to meet predefined quadrature precision objectives, akin to how typical optimizers terminate based on convergence thresholds. This approach obviates the necessity for exhaustive searches across all potential batch sizes. We also extend this to scenarios with constrained active learning and constrained optimization, interpreting constraint violations as reductions in the precision requirement, to subsequently adapt batch construction. Through extensive experiments, we demonstrate that our approach significantly enhances learning efficiency and flexibility in diverse Bayesian batch active learning and Bayesian optimization *** Codes 62C10, 62F15 © 2023, CC BY-NC-ND.

关键词： Constrained optimization

DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement learning from Human Feedback 39

学校读者我要写书评

暂无评论

DUO: Diverse, Uncertain, On-Policy Query Generation and Sele...

39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025

作者： Feng, Xuening Jiang, Zhaohui Kaufmann, Timo Xu, Puchen Hüllermeier, Eyke Weng, Paul Zhu, Yifei UM-SJTU Joint Institute Shanghai Jiao Tong University Shanghai China Institute for Informatics LMU Munich Munich Germany Munich Center of Machine Learning Munich Germany German Research Center for Artificial Intelligence Germany Data Science Research Center Duke Kunshan University Kunshan China

ISBN: (纸本)157735897X

Defining a reward function is usually a challenging but critical task for the system designer in reinforcement learning, especially when specifying complex behaviors. Reinforcement learning from human feedback (RLHF) emerges as a promising approach to circumvent this. In RLHF, the agent typically learns a reward function by querying a human teacher using pairwise comparisons of trajectory segments. A key question in this domain is how to reduce the number of queries necessary to learn an informative reward function, since asking a human teacher too many queries is impractical and costly. To tackle this question, most existing methods mainly focus on improving exploration, introducing data augmentation or designing sophisticated training objectives for RLHF, while the potential of query generation and selection schemes have not been fully exploited. In this paper, we propose DUO, a novel method for diverse, uncertain, on-policy query generation and selection in RLHF. Our method produces queries that are (1) more relevant for policy training (via an on-policy criterion), (2) more informative (via a principled measure of epistemic uncertainty), and (3) diverse (via a clustering-based filter). Experimental results on a variety of locomotion and robotic manipulation tasks demonstrate that our method can outperform state-of-the-art RLHF methods given the same total budget of queries while being robust to possibly irrational teachers. Copyright © 2025, Association for the Advancement of Artificial Intelligence (***). All rights reserved.

关键词： Reinforcement learning

A duality framework for analyzing random feature and two-layer neural networks

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Chen, Hongrui Long, Jihao Wu, Lei Department of Mathematics Stanford University United States Institute for Advanced Algorithmic Research Shanghai China School of Mathematical Sciences Peking University China Center for Machine Learning Research Peking University China AI for Science Institute Beijing China

We consider the problem of learning functions within the Fp,π and Barron spaces, which play crucial roles in understanding random feature models (RFMs), two-layer neural networks, as well as kernel methods. Leveraging tools from information-based complexity (IBC), we establish a dual equivalence between approximation and estimation, and then apply it to study the learning of the preceding function spaces. The duality allows us to focus on the more tractable problem between approximation and estimation. To showcase the efficacy of our duality framework, we delve into two important but under-explored problems: • Random feature learning beyond kernel regime: We derive sharp bounds for learning Fp,π using RFMs. Notably, the learning is efficient without the curse of dimensionality for p > 1. This underscores the extended applicability of RFMs beyond the traditional kernel regime, since Fp,π with p ∞ learning of RKHS: We establish sharp, spectrum-dependent characterizations for the convergence of L∞ learning error in both noiseless and noisy settings. Surprisingly, we show that popular kernel ridge regression can achieve near-optimal performance in L∞ learning, despite it primarily minimizing square loss. To establish the aforementioned duality, we introduce a type of IBC, termed I-complexity, to measure the size of a function class. Notably, I-complexity offers a tight characterization of learning in noiseless settings, yields lower bounds comparable to Le Cam’s in noisy settings, and is versatile in deriving upper bounds. We believe that our duality framework holds potential for broad application in learning analysis across more scenarios. Copyright © 2023, The Authors. All rights reserved.

关键词： Multilayer neural networks

Quantization of Bandlimited Graph Signals

学校读者我要写书评

暂无评论

Quantization of Bandlimited Graph Signals

International Conference on Sampling Theory and Applications (SampTA)

作者： Felix Krahmer He Lyu Rayan Saab Anna Veselovska Rongrong Wang Department of Mathematics & Munich Data Science Institute Technical University of Munich and Munich Center for Machine Learning Garching/Munich Germany Department of Mathematics & Halicioglu Data Science Institute University of California San Diego San Diego USA Department of Computational Mathematics Science and Engineering & Department of Mathematics Michigan State University East Lansing USA

Graph models and graph-based signals are becoming increasingly important in machine learning, natural sciences, and modern signal processing. In this paper, we address the problem of quantizing bandlimited graph signals. We introduce two classes of noise-shaping algorithms for graph signals that differ in their sampling methodologies. We demonstrate that these algorithms can be efficiently used to construct quantized representatives of bandlimited graph-based signals with bounded amplitude. Moreover, for one of the algorithms, we provide theoretical guarantees on the relative error between the quantized representative and the true signal.

关键词：