检索结果-内蒙古大学图书馆

TRANSFER LEARNING BEYOND BOUNDED DENSITY RATIOS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kalavasis, Alkis Zadik, Ilias Zampetakis, Manolis Department of Statistics and Data Science Yale University United States Department of Computer Science Yale University United States Institute of Foundations of Data Science Yale University United States

We study the fundamental problem of transfer learning where a learning algorithm collects data from some source distribution P but needs to perform well with respect to a different target distribution Q. A standard change of measure argument implies that transfer learning happens when the density ratio dQ{dP is bounded. Yet, prior thought-provoking works by Kpotufe and Martinet (COLT, 2018) and Hanneke and Kpotufe (NeurIPS, 2019) demonstrate cases where the ratio dQ{dP is unbounded, but transfer learning is possible. In this work, we focus on transfer learning over the class of low-degree polynomial estimators. Our main result is a general transfer inequality over the domain n, proving that non-trivial transfer learning for low-degree polynomials is possible under very mild assumptions, going well beyond the classical assumption that dQ{dP is bounded. For instance, it always applies if Q is a log-concave measure and the inverse ratio dQ{dP is bounded. To demonstrate the applicability of our inequality, we obtain new results in the settings of: (1) the classical truncated regression setting, where dQ{dP equals infinity, and (2) the more recent out-of-distribution generalization setting for in-context learning linear functions with transformers. We also provide a discrete analogue of our transfer inequality on the Boolean Hypercube t´1, 1un, and study its connections with the recent problem of Generalization on the Unseen of Abbe, Bengio, Lotfi and Rizk (ICML, 2023). Our main conceptual contribution is that the maximum influence of the error of the estimator ˆˆf´ *under Q, Imaxp ˆˆf´ *q, acts as a sufficient condition for transferability;when Imaxp ˆˆf´ *q is appropriately bounded, transfer is possible over the Boolean domain. The main techniques for proving our results are the Carbery-Wright anti-concentration inequality and the invariance principle of Mossel, O’Donnell and Oleszkiewicz (Annals of Mathematics, 2010). © 2024, CC BY.

关键词： Learning algorithms

Cell2Sentence: Teaching Large Language Models the Language of Biology 41

学校读者我要写书评

暂无评论

Cell2Sentence: Teaching Large Language Models the Language o...

41st International Conference on Machine Learning, ICML 2024

作者： Levine, Daniel Rizvi, Syed Asad Lévy, Sacha Pallikkavaliyaveetil, Nazreen Zhang, David Chen, Xingyu Ghadermarzi, Sina Wu, Ruiming Zheng, Zihe Vrkic, Ivan Zhong, Anna Raskin, Daphne Han, Insu de Oliveira Fonseca, Antonio Henrique Caro, Josue Ortega Karbasi, Amin Dhodapkar, Rahul M. van Dijk, David Department of Computer Science Yale University New HavenCT United States School of Engineering Applied Science University of Pennsylvania PhiladelphiaPA United States School of Computer and Communication Sciences Swiss Federal Institute of Technology Lausanne Lausanne Switzerland Department of Neuroscience Yale School of Medicine New HavenCT United States Wu Tsai Institute Yale University New HavenCT United States Google United States Yale Institute for Foundations of Data Science New HavenCT United States Yale School of Engineering and Applied Science New HavenCT United States Roski Eye Institute University of Southern California Los AngelesCA United States Yale School of Medicine New HavenCT United States Cardiovascular Research Center Yale School of Medicine New HavenCT United States Interdepartmental Program in Computational Biology & Bioinformatics Yale University New HavenCT United States

We introduce Cell2Sentence (C2S), a novel method to directly adapt large language models to a biological context, specifically single-cell transcriptomics. By transforming gene expression data into"cell sentences," C2S bridges the gap between natural language processing and biology. We demonstrate cell sentences enable the finetuning of language models for diverse tasks in biology, including cell generation, complex cell-type annotation, and direct data-driven text generation. Our experiments reveal that GPT-2, when fine-tuned with C2S, can generate biologically valid cells based on cell type inputs, and accurately predict cell types from cell sentences. This illustrates that language models, through C2S finetuning, can acquire a significant understanding of single-cell biology while maintaining robust text generation capabilities. C2S offers a flexible, accessible framework to integrate natural language processing with transcriptomics, utilizing existing models and libraries for a wide range of biological applications. Copyright 2024 by the author(s)

关键词： Gene expression

From task structures to world models: What do LLMs know?

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yildirim, Ilker Paul, L.A. Department of Psychology Yale University United States Department of Philosophy Yale University United States Department of Statistics & Data Science Yale University United States Wu-Tsai Institute Yale University United States Foundations of the Data Science Institute Yale University United States

In what sense does a large language model have knowledge? The answer to this question extends beyond the capabilities of a particular AI system, and challenges our assumptions about the nature of knowledge and intelligence. We answer by granting LLMs "instrumental knowledge";knowledge defined by a certain set of abilities. We then ask how such knowledge is related to the more ordinary, "worldly" knowledge exhibited by human agents, and explore this in terms of the degree to which instrumental knowledge can be said to incorporate the structured world models of cognitive science. We discuss ways LLMs could recover degrees of worldly knowledge, and suggest such recovery will be governed by an implicit, resource-rational tradeoff between world models and task demands. Copyright © 2023, The Authors. All rights reserved.

关键词：

Cell2Sentence: teaching large language models the language of biology 24

学校读者我要写书评

暂无评论

Cell2Sentence: teaching large language models the language o...

Proceedings of the 41st International Conference on Machine Learning

作者： Daniel Levine Syed Asad Rizvi Sacha Lévy Nazreen Pallikkavaliyaveetil David Zhang Xingyu Chen Sina Ghadermarzi Ruiming Wu Zihe Zheng Ivan Vrkic Anna Zhong Daphne Raskin Insu Han Antonio Henrique De Oliveira Fonseca Josue Ortega Caro Amin Karbasi Rahul M. Dhodapkar David Van Dijk Department of Computer Science Yale University New Haven CT School of Engineering Applied Science University of Pennsylvania Philadelphia PA School of Computer and Communication Sciences Swiss Federal Institute of Technology Lausanne Lausanne Switzerland Department of Computer Science Yale University New Haven CT and Department of Neuroscience Yale School of Medicine New Haven CT Department of Computer Science Yale University New Haven CT and Department of Neuroscience Yale School of Medicine New Haven CT and Wu Tsai Institute Yale University New Haven CT Google and Yale Institute for Foundations of Data Science New Haven CT and Department of Computer Science Yale University New Haven CT and Yale School of Engineering and Applied Science New Haven CT Roski Eye Institute University of Southern California Los Angeles CA and Department of Internal Medicine (Cardiology) Yale School of Medicine New Haven CT Department of Computer Science Yale University New Haven CT and Yale Institute for Foundations of Data Science New Haven CT and Wu Tsai Institute Yale University New Haven CT and Cardiovascular Research Center Yale School of Medicine New Haven CT and Interdepartmental Program in Computational Biology & Bioinformatics Yale University New Haven CT and Department of Internal Medicine (Cardiology) Yale School of Medicine New Haven CT

We introduce Cell2Sentence (C2S), a novel method to directly adapt large language models to a biological context, specifically single-cell transcriptomics. By transforming gene expression data into "cell sentences," C2S bridges the gap between natural language processing and biology. We demonstrate cell sentences enable the fine-tuning of language models for diverse tasks in biology, including cell generation, complex cell-type annotation, and direct data-driven text generation. Our experiments reveal that GPT-2, when fine-tuned with C2S, can generate biologically valid cells based on cell type inputs, and accurately predict cell types from cell sentences. This illustrates that language models, through C2S fine-tuning, can acquire a significant understanding of single-cell biology while maintaining robust text generation capabilities. C2S offers a flexible, accessible framework to integrate natural language processing with transcriptomics, utilizing existing models and libraries for a wide range of biological applications.

关键词：

MambaLRP: Explaining Selective State Space Sequence Models 38

学校读者我要写书评

暂无评论

MambaLRP: Explaining Selective State Space Sequence Models

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Jafari, Farnoush Rezaei Montavon, Grégoire Müller, Klaus-Robert Eberle, Oliver Machine Learning Group Technische Universität Berlin Berlin10587 Germany BIFOLD - Berlin Institute for the Foundations of Learning and Data Berlin10587 Germany Department of Mathematics and Computer Science Freie Universität Berlin Arnimallee 14 Berlin14195 Germany Department of Artificial Intelligence Korea University Seoul136-713 Korea Republic of Max Planck Institute for Informatics Stuhlsatzenhausweg 4 Saarbrücken66123 Germany Google DeepMind Berlin Germany

Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

Improved classical shadows from local symmetries in the Schur basis

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Grier, Daniel Liu, Sihan Mahajan, Gaurav Department of Computer Science and Engineering Department of Mathematics UC San Diego United States Department of Computer Science and Engineering UCSD California CA92092 United States Institute for Foundations of Data Science Yale University Connecticut CT06511 United States

We study the sample complexity of the classical shadows task: what is the fewest number of copies of an unknown state you need to measure to predict expected values with respect to some class of observables? Large joint measurements are likely required in order to minimize sample complexity, but previous joint measurement protocols only work when the unknown state is pure. We present the first joint measurement protocol for classical shadows whose sample complexity scales with the rank of the unknown state. In particular we prove O(√rB/ǫ2) samples suffice, where r is the rank of the state, B is a bound on the squared Frobenius norm of the observables, and ϵ is the target accuracy. In the low-rank regime, this is a nearly quadratic advantage over traditional approaches that use single-copy measurements. We present several intermediate results that may be of independent interest: a solution to a new formulation of classical shadows that captures functions of non-identical input states;a generalization of a "nice" Schur basis used for optimal qubit purification and quantum majority vote;and a measurement strategy that allows us to use local symmetries in the Schur basis to avoid intractable Weingarten calculations in the analysis. © 2024, CC BY.

关键词：

MambaLRP: explaining selective state space sequence models 24

学校读者我要写书评

暂无评论

MambaLRP: explaining selective state space sequence models

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Farnoush Rezaei Jafari Grégoire Montavon Klaus-Robert Müller Oliver Eberle Machine Learning Group Technische Universität Berlin Berlin Germany and BIFOLD - Berlin Institute for the Foundations of Learning and Data Berlin Germany Department of Mathematics and Computer Science Freie Universitat Berlin Berlin Germany and BIFOLD - Berlin Institute for the Foundations of Learning and Data Berlin Germany and Machine Learning Group Technische Universität Berlin Berlin Germany BIFOLD - Berlin Institute for the Foundations of Learning and Data Berlin Germany and Machine Learning Group Technische Universität Berlin Berlin Germany and Department of Artificial Intelligence Korea University Seoul South Korea and Max Planck Institute for Informatics Saarbrücken Germany and Google DeepMind Berlin Germany

ISBN: (纸本)9798331314385

关键词：

Analyzing Atomic Interactions in Molecules as Learned by Neural Networks

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Esders, Malte Schnake, Thomas Lederer, Jonas Kabylda, Adil Montavon, Grégoire Tkatchenko, Alexandre Müller, Klaus-Robert BIFOLD Berlin Institute for the Foundations of Learning and Data Germany Machine Learning Group Berlin Institute of Technology Berlin10587 Germany Department of Physics and Materials Science University of Luxembourg Luxembourg CityL-1511 Luxembourg Department of Mathematics and Computer Science Free University of Berlin Germany Google Deepmind Berlin Germany Department of Artificial Intelligence Korea University Seoul136-713 Korea Republic of Max Planck Institute for Informatics Saarbrücken66123 Germany

While machine learning (ML) models have been able to achieve unprecedented accuracies across various prediction tasks in quantum chemistry, it is now apparent that accuracy on a test set alone is not a guarantee for robust chemical modeling such as stable molecular dynamics (MD). To go beyond accuracy, we use explainable artificial intelligence (XAI) techniques to develop a general analysis framework for atomic interactions and apply it to the SchNet and PaiNN neural network models. We compare these interactions with a set of fundamental chemical principles to understand how well the models have learned the underlying physicochemical concepts from the data. We focus on the strength of the interactions for different atomic species, how predictions for intensive and extensive quantum molecular properties are made, and analyze the decay and many-body nature of the interactions with interatomic distance. Models that deviate too far from known physical principles produce unstable MD trajectories, even when they have very high energy and force prediction accuracy. We also suggest further improvements to the ML architectures to better account for the polynomial decay of atomic interactions. Copyright © 2024, The Authors. All rights reserved.

关键词： Molecular dynamics

The Clever Hans Effect in Unsupervised Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Kauffmann, Jacob Dippel, Jonas Ruff, Lukas Samek, Wojciech Müller, Klaus-Robert Montavon, Grégoire Department of Electrical Engineering and Computer Science Technische Universität Berlin Germany BIFOLD – Berlin Institute for the Foundations of Learning and Data Berlin Germany Aignostics Berlin Germany Department of Mathematics and Computer Science Freie Universität Berlin Germany Department of Artificial Intelligence Fraunhofer HHI Berlin Germany Department of Artificial Intelligence Korea University Seoul Korea Republic of Max-Planck Institute for Informatics Saarbrücken Germany Google Deepmind Berlin Germany

Unsupervised learning has become an essential building block of AI systems. The representations it produces, e.g. in foundation models, are critical to a wide variety of downstream applications. It is therefore important to carefully examine unsupervised models to ensure not only that they produce accurate predictions, but also that these predictions are not "right for the wrong reasons", the so-called Clever Hans (CH) effect. Using specially developed Explainable AI techniques, we show for the first time that CH effects are widespread in unsupervised learning. Our empirical findings are enriched by theoretical insights, which interestingly point to inductive biases in the unsupervised learning machine as a primary source of CH effects. Overall, our work sheds light on unexplored risks associated with practical applications of unsupervised learning and suggests ways to make unsupervised learning more robust. Copyright © 2024, The Authors. All rights reserved.

关键词： Adversarial machine learning