检索结果-内蒙古大学图书馆

A Canonicalization Perspective on Invariant and Equivariant Learning 38

学校读者我要写书评

暂无评论

A Canonicalization Perspective on Invariant and Equivariant ...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Ma, George Wang, Yifei Lim, Derek Jegelka, Stefanie Wang, Yisen School of EECS Peking University China MIT CSAIL United States TUM CIT MCML MDSI MIT EECS CSAIL United States State Key Lab of General Artificial Intelligence School of Intelligence Science and Technology Peking University China Institute for Artificial Intelligence Peking University China

In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods emerged to be a unified framework for attaining symmetries efficiently by averaging over input-dependent subsets of the group, i.e., frames. What we currently lack is a principled understanding of the design of frames. In this work, we introduce a canonicalization perspective that provides an essential and complete view of the design of frames. Canonicalization is a classic approach for attaining invariance by mapping inputs to their canonical forms. We show that there exists an inherent connection between frames and canonical forms. Leveraging this connection, we can efficiently compare the complexity of frames as well as determine the optimality of certain frames. Guided by this principle, we design novel frames for eigenvectors that are strictly superior to existing methods-some are even optimal-both theoretically and empirically. The reduction to the canonicalization perspective further uncovers equivalences between previous methods. These observations suggest that canonicalization provides a fundamental understanding of existing frame-averaging methods and unifies existing equivariant and invariant learning methods. Code is available at https://***/PKU-ML/canonicalization. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

HIGHER-ORDER GRAPHON NEURAL NETWORKS: APPROXIMATION AND CUT DISTANCE

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Herbst, Daniel Jegelka, Stefanie TUM Germany School of CIT Germany MCML and MDSI Germany MIT Department of EECS and CSAIL United States

Graph limit models, like graphons for limits of dense graphs, have recently been used to study size transferability of graph neural networks (GNNs). While most literature focuses on message passing GNNs (MPNNs), in this work we attend to the more powerful higher-order GNNs. First, we extend the k-WL test for graphons (Böker, 2023) to the graphon-signal space and introduce signal-weighted homomorphism densities as a key tool. As an exemplary focus, we generalize Invariant Graph Networks (IGNs) to graphons, proposing Invariant Graphon Networks (IWNs) defined via a subset of the IGN basis corresponding to bounded linear operators. Even with this restricted basis, we show that IWNs of order k are at least as powerful as the k-WL test, and we establish universal approximation results for graphon-signals in Lp distances. This significantly extends the prior work of Cai & Wang (2022), showing that IWNs—a subset of their IGN-small—retain effectively the same expressivity as the full IGN basis in the limit. In contrast to their approach, our blueprint of IWNs also aligns better with the geometry of graphon space, for example facilitating comparability to MPNNs. We highlight that, while typical higher-order GNNs are discontinuous w.r.t. cut distance—which causes their lack of convergence and is inherently tied to the definition of k-WL—their transferability remains comparable to MPNNs. © 2025, CC BY.

关键词： Graph neural networks

ON THE ROLE OF DEPTH AND LOOPING FOR IN-CONTEXT LEARNING WITH TASK DIVERSITY

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gatmiry, Khashayar Saunshi, Nikunj Reddi, Sashank J. Jegelka, Stefanie Kumar, Sanjiv MIT United States Google Research United States TUM∗and MIT United States CIT MCML MDSI EECS and CSAIL United States

The intriguing in-context learning (ICL) abilities of deep Transformer models have lately garnered significant attention. By studying in-context linear regression on unimodal Gaussian data, recent empirical and theoretical works have argued that ICL emerges from Transformers’ abilities to simulate learning algorithms like gradient descent. However, these works fail to capture the remarkable ability of Transformers to learn multiple tasks in context. To this end, we study in-context learning for linear regression with diverse tasks, characterized by data covariance matrices with condition numbers ranging from [1, κ], and highlight the importance of depth in this setting. More specifically, (a) we show theoretical lower bounds of log(κ) (or √κ) linear attention layers in the unrestricted (or restricted) attention setting and, (b) we show that multilayer Transformers can indeed solve such tasks with a number of layers that matches the lower bounds. However, we show that this expressivity of multilayer Transformer comes at the price of robustness. In particular, multilayer Transformers are not robust to even distributional shifts as small as O(e−L) in Wasserstein distance, where L is the depth of the network. We then demonstrate that Looped Transformers —a special class of multilayer Transformers with weight-sharing— not only exhibit similar expressive power but are also provably robust under mild assumptions. Besides out-of-distribution generalization, we also show that Looped Transformers are the only models that exhibit a monotonic behavior of loss with respect to depth. Copyright © 2024, The Authors. All rights reserved.

关键词： Distribution transformers

BEYOND INTERPRETABILITY: THE GAINS OF FEATURE MONOSEMANTIcitY ON MODEL ROBUSTNESS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Qi Wang, Yifei Cui, Jingyi Pan, Xiang Lei, Qi Jegelka, Stefanie Wang, Yisen Peking University China MIT CSAIL United States New York University United States TUM CIT MCML MDSI Germany MIT EECS CSAIL United States

Deep learning models often suffer from a lack of interpretability due to polysemanticity, where individual neurons are activated by multiple unrelated semantics, resulting in unclear attributions of model behavior. Recent advances in monosemanticity, where neurons correspond to consistent and distinct semantics, have significantly improved interpretability but are commonly believed to compromise accuracy. In this work, we challenge the prevailing belief of the accuracy-interpretability tradeoff, showing that monosemantic features not only enhance interpretability but also bring concrete gains in model performance. Across multiple robust learning scenarios—including input and label noise, few-shot learning, and out-of-domain generalization—our results show that models leveraging monosemantic features significantly outperform those relying on polysemantic features. Furthermore, we provide empirical and theoretical understandings on the robustness gains of feature monosemanticity. Our preliminary analysis suggests that monosemanticity, by promoting better separation of feature representations, leads to more robust decision boundaries. This diverse evidence highlights the generality of monosemanticity in improving model robustness. As a first step in this new direction, we embark on exploring the learning benefits of monosemanticity beyond interpretability, supporting the longstanding hypothesis of linking interpretability and robustness. Code is available at https://***/PKU-ML/Beyond_Interpretability. © 2024, CC BY.

关键词： Semantics

Understanding the Role of Equivariance in Self-supervised Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Yifei Hu, Kaiwen Gupta, Sharut Ye, Ziyu Wang, Yisen Jegelka, Stefanie MIT United States Peking University China The University of Chicago United States TUM CIT MCML MDSI Germany MIT EECS CSAIL United States

Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (e.g., colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at https://***/kaotty/Understanding-ESSL. © 2024, CC BY.

关键词： Self-supervised learning

A canonicalization perspective on invariant and equivariant learning 24

学校读者我要写书评

暂无评论

A canonicalization perspective on invariant and equivariant ...

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： George Ma Yifei Wang Derek Lim Stefanie Jegelka Yisen Wang School of EECS Peking University MIT CSAIL TUM CIT/MCML/MDSI & MIT EECS/CSAIL State Key Lab of General Artificial Intelligence School of Intelligence Science and Technology Peking University and Institute for Artificial Intelligence Peking University

ISBN: (纸本)9798331314385

In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods emerged to be a unified framework for attaining symmetries efficiently by averaging over input-dependent subsets of the group, i.e., frames. What we currently lack is a principled understanding of the design of frames. In this work, we introduce a canonicalization perspective that provides an essential and complete view of the design of frames. Canonicalization is a classic approach for attaining invariance by mapping inputs to their canonical forms. We show that there exists an inherent connection between frames and canonical forms. Leveraging this connection, we can efficiently compare the complexity of frames as well as determine the optimality of certain frames. Guided by this principle, we design novel frames for eigenvectors that are strictly superior to existing methods—some are even optimal—both theoretically and empirically. The reduction to the canonicalization perspective further uncovers equivalences between previous methods. These observations suggest that canonicalization provides a fundamental understanding of existing frame-averaging methods and unifies existing equivariant and invariant learning methods. Code is available at https://***/PKU-ML/canonicalization.

关键词：

A Canonicalization Perspective on Invariant and Equivariant Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, George Wang, Yifei Lim, Derek Jegelka, Stefanie Wang, Yisen School of EECS Peking University China MIT CSAIL United States TUM CIT/MCML/MDSI & MIT EECS CSAIL United States State Key Lab of General Artificial Intelligence School of Intelligence Science and Technology Peking University China Institute for Artificial Intelligence Peking University China

In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods emerged to be a unified framework for attaining symmetries efficiently by averaging over input-dependent subsets of the group, i.e., frames. What we currently lack is a principled understanding of the design of frames. In this work, we introduce a canonicalization perspective that provides an essential and complete view of the design of frames. Canonicalization is a classic approach for attaining invariance by mapping inputs to their canonical forms. We show that there exists an inherent connection between frames and canonical forms. Leveraging this connection, we can efficiently compare the complexity of frames as well as determine the optimality of certain frames. Guided by this principle, we design novel frames for eigenvectors that are strictly superior to existing methods—some are even optimal—both theoretically and empirically. The reduction to the canonicalization perspective further uncovers equivalences between previous methods. These observations suggest that canonicalization provides a fundamental understanding of existing frame-averaging methods and unifies existing equivariant and invariant learning methods. Code is available at https://***/PKU-ML/canonicalization. © 2024, CC BY.

关键词： Contrastive Learning

A Theoretical Understanding of Self-Correction through In-context Alignment 38

学校读者我要写书评

暂无评论

A Theoretical Understanding of Self-Correction through In-co...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Wang, Yifei Wu, Yuyang Wei, Zeming Jegelka, Stefanie Wang, Yisen MIT CSAIL United States School of EECS Peking University China School of Mathematical Sciences Peking University China State Key Lab of General Artificial Intelligence School of Intelligence Science and Technology Peking University China CIT MCML MDSI TU Munich Germany Institute for Artificial Intelligence Peking University China

Going beyond mimicking limited human experiences, recent studies show initial evidence that, like humans, large language models (LLMs) are capable of improving their abilities purely by self-correction, i.e., correcting previous responses through self-examination, as seen in models like OpenAI o1. Nevertheless, little is known about how such capabilities arise. In this work, based on a simplified setup akin to an alignment task, we theoretically analyze self-correction from an in-context learning perspective, showing that when LLMs give relatively accurate self-examinations as rewards, they are capable of refining responses in an in-context way. Notably, going beyond previous theories on over-simplified linear transformers, our theoretical construction underpins the roles of several key designs of realistic transformers for self-correction: softmax attention, multi-head attention, and the MLP block. We validate these findings extensively on synthetic datasets. Inspired by these findings, we propose a simple self-correction strategy, Checking as Context (CaC), which finds novel applications in alleviating social bias and defending against LLM jailbreaks. We believe that these findings will inspire further research on understanding, exploiting, and enhancing self-correction for building better foundation models. Code is at https://***/yifeiwang77/Self-Correction. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

WHAT IS WRONG WITH PERPLEXITY FOR LONG-CONTEXT LANGUAGE MODELING?

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Fang, Lizhe Wang, Yifei Liu, Zhaoyang Zhang, Chenheng Jegelka, Stefanie Gao, Jinyang Ding, Bolin Wang, Yisen State Key Lab of General Artificial Intelligence School of Intelligence Science and Technology Peking University China MIT CSAIL United States Alibaba Group China TUM CIT MCML MDSI MIT EECS CSAIL United States Institute for Artificial Intelligence Peking University China

Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the context windows of LLMs and employed perplexity (PPL) as a standard evaluation metric, PPL has proven unreliable for assessing long-context capabilities. The underlying cause of this limitation has remained unclear. In this work, we provide a comprehensive explanation for this issue. We find that PPL overlooks key tokens, which are essential for long-context understanding, by averaging across all tokens and thereby obscuring the true performance of models in long-context scenarios. To address this, we propose LongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them. Our experiments demonstrate that LongPPL strongly correlates with performance on various long-context benchmarks (e.g., Pearson correlation of -0.96), significantly outperforming traditional PPL in predictive accuracy. Additionally, we introduce LongCE (Long-context Cross-Entropy) loss, a re-weighting strategy for fine-tuning that prioritizes key tokens, leading to consistent improvements across diverse benchmarks. These contributions offer deeper insights into the limitations of PPL and present effective solutions for accurately evaluating and enhancing the long-context capabilities of LLMs. Code is available at https://***/PKU-ML/LongPPL. © 2024, CC BY-NC-SA.

关键词： Modeling languages