检索结果-内蒙古大学图书馆

applied mathematics and Mechanics(English Edition) 2023年第7期44卷 1199-1224页

作者： J.WU S.F.WANG P.PERDIKARIS Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania PhiladelphiaPA 19104U.S.A. Department of Mechanical Engineering and Applied Mechanics University of Pennsylvania PhiladelphiaPA 19104U.S.A.

We propose a self-supervising learning framework for finding the dominant eigenfunction-eigenvalue pairs of linear and self-adjoint *** represent target eigenfunctions with coordinate-based neural networks and employ the Fourier positional encodings to enable the approximation of high-frequency *** formulate a self-supervised training objective for spectral learning and propose a novel regularization mechanism to ensure that the network finds the exact eigenfunctions instead of a space spanned by the ***,we investigate the effect of weight normalization as a mechanism to alleviate the risk of recovering linear dependent modes,allowing us to accurately recover a large number of *** effectiveness of our methods is demonstrated across a collection of representative benchmarks including both local and non-local diffusion operators,as well as high-dimensional time-series data from a video *** results indicate that the present algorithm can outperform competing approaches in terms of both approximation accuracy and computational cost.

关键词： spectral learning partial differential equation(PDE) neural network slow features analysis

STOCHASTIC CONTROLLED AVERAGING FOR FEDERATED LEARNING WITH COMMUNICATION COMPRESSION 12

学校读者我要写书评

暂无评论

STOCHASTIC CONTROLLED AVERAGING FOR FEDERATED LEARNING WITH ...

12th International Conference on Learning Representations, ICLR 2024

作者： Huang, Xinmeng Li, Ping Li, Xiaoyun LinkedIn BellevueWA98004 United States The Graduate Group of Applied Mathematics and Computational Science The University of Pennsylvania United States

Communication compression has been an important topic in Federated Learning (FL) for alleviating the communication overhead. However, communication compression brings forth new challenges in FL due to the interplay of compression-incurred information distortion and inherent characteristics of FL such as partial participation and data heterogeneity. Despite the recent development, the existing approaches either cannot accommodate arbitrary data heterogeneity or partial participation, or require stringent conditions on compression. In this paper, we revisit the seminal stochastic controlled averaging method by proposing an equivalent but more efficient/simplified formulation with halved uplink communication costs, building upon which we propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression, respectively. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities. Moreover, SCALLION and SCAFCOM attain fast convergence rates under arbitrary data heterogeneity without any additional assumptions on compression errors. Experiments show that SCALLION and SCAFCOM outperform recent compressed FL methods under the same communication budget. © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.

关键词： Stochastic systems

A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks 41

学校读者我要写书评

暂无评论

A Theory of Non-Linear Feature Learning with One Gradient St...

41st International Conference on Machine Learning, ICML 2024

作者： Moniri, Behrad Lee, Donghwan Hassani, Hamed Dobriban, Edgar Department of Electrical and Systems Engineering University of Pennsylvania PA United States Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania PA United States Department of Statistics and Data Science University of Pennsylvania PA United States

Feature learning is thought to be one of the fundamental reasons for the success of deep neural networks. It is rigorously known that in two-layer fully-connected neural networks under certain conditions, one step of gradient descent on the first layer can lead to feature learning;characterized by the appearance of a separated rank-one component-spike-in the spectrum of the feature matrix. However, with a constant gradient descent step size, this spike only carries information from the linear component of the target function and therefore learning non-linear components is impossible. We show that with a learning rate that grows with the sample size, such training in fact introduces multiple rank-one components, each corresponding to a specific polynomial feature. We further prove that the limiting large-dimensional and large sample training and test errors of the updated neural networks are fully characterized by these spikes. By precisely analyzing the improvement in the training and test errors, we demonstrate that these non-linear features can enhance learning. Copyright 2024 by the author(s)

关键词： Adversarial machine learning

Stochastic Controlled Averaging for Federated Learning with Communication Compression

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Huang, Xinmeng Li, Ping Li, Xiaoyun The Graduate Group of Applied Mathematics and Computational Science The University of Pennsylvania United States

Communication compression, a technique aiming to reduce the information volume to be transmitted over the air, has gained great interest in Federated Learning (FL) for the potential of alleviating its communication overhead. However, communication compression brings forth new challenges in FL due to the interplay of compression-incurred information distortion and inherent characteristics of FL such as partial participation and data heterogeneity. Despite the recent development, the performance of compressed FL approaches has not been fully exploited. The existing approaches either cannot accommodate arbitrary data heterogeneity or partial participation, or require stringent conditions on compression. In this paper, we revisit the seminal stochastic controlled averaging method by proposing an equivalent but more efficient/simplified formulation with halved uplink communication costs. Building upon this implementation, we propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression, respectively. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities. Moreover, SCALLION and SCAFCOM accommodate arbitrary data heterogeneity and do not make any additional assumptions on compression errors. Experiments show that SCALLION and SCAFCOM can match the performance of corresponding full-precision FL approaches with substantially reduced uplink communication, and outperform recent compressed FL methods under the same communication budget. Copyright © 2023, The Authors. All rights reserved.

关键词： Budget control

Demystifying Disagreement-on-the-Line in High Dimensions 40

学校读者我要写书评

暂无评论

Demystifying Disagreement-on-the-Line in High Dimensions

40th International Conference on Machine Learning, ICML 2023

作者： Lee, Donghwan Moniri, Behrad Huang, Xinmeng Dobriban, Edgar Hassani, Hamed Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania PA United States Department of Electrical and Systems Engineering University of Pennsylvania PA United States Department of Statistics and Data Science University of Pennsylvania PA United States

Evaluating the performance of machine learning models under distribution shifts is challenging, especially when we only have unlabeled data from the shifted (target) domain, along with labeled data from the original (source) domain. Recent work suggests that the notion of disagreement, the degree to which two models trained with different randomness differ on the same input, is a key to tackling this problem. Experimentally, disagreement and prediction error have been shown to be strongly connected, which has been used to estimate model performance. Experiments have led to the discovery of the disagreement-on-the-line phenomenon, whereby the classification error under the target domain is often a linear function of the classification error under the source domain;and whenever this property holds, disagreement under the source and target domain follow the same linear relation. In this work, we develop a theoretical foundation for analyzing disagreement in high-dimensional random features regression;and study under what conditions the disagreement-on-the-line phenomenon occurs in our setting. Experiments on CIFAR-10-C, Tiny ImageNet-C, and Camelyon17 are consistent with our theory and support the universality of the theoretical findings. © 2023 Proceedings of Machine Learning Research. All rights reserved.

关键词： Errors

SURREAL-GAN:SEMI-SUPERVISED REPRESENTATION LEARNING VIA GAN FOR UNCOVERING HETEROGENEOUS DISEASE-RELATED IMAGING PATTERNS 10

学校读者我要写书评

暂无评论

SURREAL-GAN:SEMI-SUPERVISED REPRESENTATION LEARNING VIA GAN ...

10th International Conference on Learning Representations, ICLR 2022

作者： Yang, Zhijian Wen, Junhao Davatzikos, Christos Center for Biomedical Image Computing and Analytics University of Pennsylvania United States Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania United States

A plethora of machine learning methods have been applied to imaging data, enabling the construction of clinically relevant imaging signatures of neurological and neuropsychiatric diseases. Oftentimes, such methods do not explicitly model the heterogeneity of disease effects, or approach it via nonlinear models that are not interpretable. Moreover, unsupervised methods may parse heterogeneity that is driven by nuisance confounding factors that affect global brain structure or function, rather than heterogeneity relevant to a pathology of interest. On the other hand, semi-supervised clustering methods seek to derive a pathology-related dichotomous subtypes, ignoring the truth that disease heterogeneity spatially and temporally extends along a continuum. To address the aforementioned limitations, herein, we propose a novel method, termed Surreal-GAN (Semi-SUpeRvised ReprEsentAtion Learning via GAN). Using cross-sectional imaging data, Surreal-GAN dissects underlying disease-related heterogeneity under the principle of semi-supervised clustering [cluster mappings from the normal control (CN) to patient (PT) domain], proposes a continuously dimensional representation, and infers the disease severity of patients at individual level along each dimension. The model first learns a transformation function from the CN domain to the PT domain with latent variables controlling transformation directions. An inverse mapping function together with regularization on function continuity, pattern orthogonality and monotonicity was also imposed to make sure that the transformation function captures necessarily meaningful imaging patterns with clinical significance. We first validated the model through semi-synthetic experiments, and then demonstrated its potential in capturing biologically plausible imaging patterns in Alzheimer's disease (AD). © 2022 ICLR 2022 - 10th International Conference on Learning Representationss. All rights reserved.

关键词： Pathology

Simultaneous Estimation of Many Sparse Networks via Hierarchical Poisson Log-Normal Model

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ge, Changhao Li, Hongzhe Graduate Group of Applied Mathematics and Computational Science University of Pennsylvania United States Department of Biostatistics Epidemiology and Informatics University of Pennsylvania United States

The advancement of single-cell RNA-sequencing (scRNA-seq) technologies allow us to study the individual level cell-type-specific gene expression networks by direct inference of genes’ conditional independence structures. scRNA-seq data facilitates the analysis of gene expression data across different conditions or samples, enabling simultaneous estimation of condition- or sample-specific gene networks. Since the scRNA-seq data are count data with many zeros, existing network inference methods based on Gaussian graphs cannot be applied to such single cell data directly. We propose a hierarchical Poisson Log-Normal model to simultaneously estimate many such networks to effectively incorporate the shared network structures. We develop an efficient simultaneous estimation method that uses the variational EM and alternating direction method of multipliers (ADMM) algorithms, optimized for parallel processing. Simulation studies show this method outperforms traditional methods in network structure recovery and parameter estimation across various network models. We apply the method to two single cell RNA-seq datasets, a yeast single-cell gene expression dataset measured under 11 different environmental conditions, and a single-cell gene expression data from 13 inflammatory bowel disease patients. We demonstrate that simultaneous estimation can uncover a wider range of conditional dependence networks among genes, offering deeper insights into gene expression mechanisms. © 2024, CC BY.

关键词： Expectation maximization algorithm

The Loser’s Curse and the Critical Role of the Utility Function

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Brill, Ryan S. Wyner, Abraham J. Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania United States Dept. of Statistics and Data Science The Wharton School University of Pennsylvania United States

A longstanding question in the judgment and decision making literature is whether experts, even in high-stakes environments, exhibit the same cognitive biases observed in controlled experiments with inexperienced participants. In their seminal work, Massey and Thaler (2013) provide a notable example of bias and irrationality in expert decision making: general managers’ behavior in the National Football League draft pick trade market. They argue that general managers systematically overvalue top draft picks, which generate less surplus value on average than later first-round picks, a phenomenon known as the loser’s curse. Their conclusion hinges on the assumption that general managers should use expected surplus value as their utility function for evaluating draft picks. This assumption, however, is neither explicitly justified nor necessarily aligned with the strategic complexities of constructing a National Football League roster. In this paper, we challenge their framework by considering alternative utility functions, particularly those that emphasize the acquisition of transformational players––those capable of dramatically increasing a team’s chances of winning the Super Bowl. Under a decision rule that prioritizes the probability of acquiring elite players, which we construct from a novel Bayesian hierarchical Beta regression model, general managers’ draft trade behavior appears rational rather than systematically flawed. More broadly, our findings highlight the critical role of carefully specifying a utility function when evaluating the quality of decisions. © 2024, CC BY.

关键词： Hierarchical systems

A Variational Spike-and-Slab Approach for group Variable Selection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Lin, Buyu Ge, Changhao Liu, Jun S. Department of Statistics Harvard University United States Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania United States

We introduce a class of generic spike-and-slab priors for high-dimensional linear regression with grouped variables and present a Coordinate-ascent Variational Inference (CAVI) algorithm for obtaining an optimal variational Bayes approximation. Using parameter expansion for a specific, yet comprehensive, family of slab distributions, we obtain a further gain in computational efficiency. The method can be easily extended to fitting additive models. Theoretically, we present general conditions on the generic spike-and-slab priors that enable us to derive the contraction rates for both the true posterior and the VB posterior for linear regression and additive models, of which some previous theoretical results can be viewed as special cases. Our simulation studies and real data application demonstrate that the proposed method is superior to existing methods in both variable selection and parameter estimation. Our algorithm is implemented in the R package GVSSB. Copyright © 2023, The Authors. All rights reserved.

关键词： computational efficiency

A theory of non-linear feature learning with one gradient step in two-layer neural networks 24

学校读者我要写书评

暂无评论

A theory of non-linear feature learning with one gradient st...

Proceedings of the 41st International Conference on Machine Learning

作者： Behrad Moniri Donghwan Lee Hamed Hassani Edgar Dobriban Department of Electrical and Systems Engineering University of Pennsylvania PA Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania PA Department of Statistics and Data Science University of Pennsylvania PA

Feature learning is thought to be one of the fundamental reasons for the success of deep neural networks. It is rigorously known that in two-layer fully-connected neural networks under certain conditions, one step of gradient descent on the first layer can lead to feature learning; characterized by the appearance of a separated rank-one component--spike--in the spectrum of the feature matrix. However, with a constant gradient descent step size, this spike only carries information from the linear component of the target function and therefore learning non-linear components is impossible. We show that with a learning rate that grows with the sample size, such training in fact introduces multiple rank-one components, each corresponding to a specific polynomial feature. We further prove that the limiting largedimensional and large sample training and test errors of the updated neural networks are fully characterized by these spikes. By precisely analyzing the improvement in the training and test errors, we demonstrate that these non-linear features can enhance learning.

关键词：