检索结果-内蒙古大学图书馆

Parallel-in-Time Kalman Smoothing Using Orthogonal Transformations

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Gargir, Shahaf Toledo, Sivan Blavatnik School of Computer Science AI Tel Aviv University Israel

We present a numerically-stable parallel-in-time linear Kalman smoother. The smoother uses a novel highly-parallel QR factorization for a class of structured sparse matrices for state estimation, and an adaptation of the SelInv selective-inversion algorithm to evaluate the covariance matrices of estimated states. Our implementation of the new algorithm, using the Threading Building Blocks (TBB) library, scales well on both Intel and ARM multi-core servers, achieving speedups of up to 47x on 64 cores. The algorithm performs more arithmetic than sequential smoothers;consequently it is 1.8x to 2.5x slower on a single core. The new algorithm is faster and scales better than the parallel Kalman smoother proposed by Särkkä and García-Fernández in 2021. Copyright © 2025, The Authors. All rights reserved.

关键词： Mathematical transformations

LEARNING FROM END USER DATA WITH SHUFFLED DIFFERENTIAL PRIVACY OVER KERNEL DENSITIES

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wagner, Tal The Blavatnik School of Computer Science and AI Tel-Aviv University Israel

We study a setting of collecting and learning from private data distributed across end users. In the shuffled model of differential privacy, the end users partially protect their data locally before sharing it, and their data is also anonymized during its collection to enhance privacy. This model has recently become a prominent alternative to central DP, which requires full trust in a central data curator, and local DP, where fully local data protection takes a steep toll on downstream accuracy. Our main technical result is a shuffled DP protocol for privately estimating the kernel density function of a distributed dataset, with accuracy essentially matching central DP. We use it to privately learn a classifier from the end user data, by learning a private density function per class. Moreover, we show that the density function itself can recover the semantic content of its class, despite having been learned in the absence of any unprotected data. Our experiments show the favorable downstream performance of our approach, and highlight key downstream considerations and trade-offs in a practical ML deployment of shuffled DP. © 2025, CC BY.

关键词： Differential privacy

SCROLLS: Standardized CompaRison Over Long Language Sequences

学校读者我要写书评

暂无评论

SCROLLS: Standardized CompaRison Over Long Language Sequence...

2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

作者： Shaham, Uri Segal, Elad Ivgi, Maor Efrat, Avia Yoran, Ori Haviv, Adi Gupta, Ankit Xiong, Wenhan Geva, Mor Berant, Jonathan Levy, Omer The Blavatnik School of Computer Science Tel Aviv University Israel Allen Institute for AI IBM Research Meta AI

NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input. SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment. Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods. © 2022 Association for Computational Linguistics.

关键词： Natural language processing systems

Weakly Supervised Text-to-SQL Parsing through Question Decomposition

学校读者我要写书评

暂无评论

Weakly Supervised Text-to-SQL Parsing through Question Decom...

2022 Findings of the Association for Computational Linguistics: NAACL 2022

作者： Wolfson, Tomer Deutch, Daniel Berant, Jonathan Blavatnik School of Computer Science Tel Aviv University Israel Allen Institute for AI United States

ISBN: (纸本)9781955917766

Text-to-SQL parsers are crucial in enabling non-experts to effortlessly query relational data. Training such parsers, by contrast, generally requires expertise in annotating natural language (NL) utterances with corresponding SQL queries. In this work, we propose a weak supervision approach for training text-to-SQL parsers. We take advantage of the recently proposed question meaning representation called QDMR, an intermediate between NL and formal query languages. Given questions, their QDMR structures (annotated by non-experts or automatically predicted), and the answers, we are able to automatically synthesize SQL queries that are used to train text-to-SQL models. We test our approach by experimenting on five benchmark datasets. Our results show that the weakly supervised models perform competitively with those trained on annotated NL-SQL data. Overall, we effectively train text-to-SQL parsers, while using zero SQL annotations. © Findings of the Association for Computational Linguistics: NAACL 2022 - Findings.

关键词： Query languages

Better Rates for Random Task Orderings in Continual Linear Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Evron, Itay Levinstein, Ran Schliserman, Matan Sherman, Uri Koren, Tomer Soudry, Daniel Srebro, Nathan Department of Electrical and Computer Engineering Technion Department of Computer Science Technion Blavatnik School of Computer Science and AI Tel Aviv University Blavatnik School of Computer Science and AI Tel Aviv University Google Research Toyota Technological Institute Chicago United States

We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze the forgetting—loss on previously seen tasks—after k iterations. For linear models, we prove that fitting a task is equivalent to a single stochastic gradient descent (SGD) step on a modified objective. We develop novel last-iterate SGD upper bounds in the realizable least squares setup, and apply them to derive new results for continual learning. Focusing on random orderings over T tasks, we establish universal forgetting rates, whereas existing rates depend on the problem dimensionality or complexity. Specifically, in continual regression with replacement, we improve the best existing rate from O((d − r¯)/k) to O(min(1/√4 k, √d − r¯/k, √Tr¯/k)), where d is the dimensionality and r¯ the average task rank. Furthermore, we establish the first rates for random task orderings without replacement. The obtained rate of O(min(1/√4T, (d − r)/T)) proves for the first time that randomization alone—with no task repetition—can _prevent catastrophic forgetting in sufficiently long task sequences. Finally, we prove a similar O(1/√4 k) universal rate for the forgetting in continual linear classification on separable data. Our universal rates apply for broader projection methods, such as block Kaczmarz and POCS, illuminating their loss convergence under i.i.d. and one-pass orderings. © 2025, CC BY.

关键词： Least squares approximations

Enhancing Automated Interpretability with Output-Centric Feature Descriptions

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Gur-Arieh, Yoav Mayan, Roy Agassy, Chen Geiger, Atticus Geva, Mor Blavatnik School of Computer Science and AI Tel Aviv University 2R Group

Automated interpretability pipelines generate natural language descriptions for the concepts represented by features in large language models (LLMs), such as plants or the first word in a sentence. These descriptions are derived using inputs that activate the feature, which may be a dimension or a direction in the model’s representation space. However, identifying activating inputs is costly, and the mechanistic role of a feature in model behavior is determined both by how inputs cause a feature to activate and by how feature activation affects outputs. Using steering evaluations, we reveal that current pipelines provide descriptions that fail to capture the causal effect of the feature on outputs. To fix this, we propose efficient, output-centric methods for automatically generating feature descriptions. These methods use the tokens weighted higher after feature stimulation or the highest weight tokens after applying the vocabulary "unembedding" head directly to the feature. Our output-centric descriptions better capture the causal effect of a feature on model outputs than input-centric descriptions, but combining the two leads to the best performance on both input and output evaluations. Lastly, we show that output-centric descriptions can be used to find inputs that activate features previously thought to be "dead". © 2025, CC BY.

关键词： Feature extraction

ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Shaham, Uri Ivgi, Maor Efrat, Avia Berant, Jonathan Levy, Omer The Blavatnik School of Computer Science Tel Aviv University Meta AI United States

We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their ideas on the live ZeroSCROLLS leaderboard. © 2023, CC BY.

关键词： Zero-shot learning

An Adversarial Scheme for Integrating Multi-modal Data on Protein Function 29th

学校读者我要写书评

暂无评论

An Adversarial Scheme for Integrating Multi-modal Data on ...

29th International Conference on Research in Computational Molecular Biology, RECOMB 2025

作者： Nasser, Rami Schaffer, Leah V. Ideker, Trey Sharan, Roded Blavatnik School of Computer Science and AI Tel Aviv University Tel Aviv69978 Israel Department of Medicine University of California San Diego La JollaCA United States Department of Computer Science and Engineering University of California San Diego La JollaCA United States Department of Bioengineering University of California San Diego La JollaCA United States

ISBN: (纸本)9783031902512

In order to begin to decipher the structure of the cell, we need to integrate multiple types of data of different scales on subcellular organization. Such integration requires dealing with multiple data modalities and with missing data. To this end, we developed MIRAGE, a multi-modal generative model for integrating protein sequence, protein-protein interaction, and protein localization data. Our approach successfully learns a joint embedding space that captures the complex relationships between these diverse modalities. We evaluate our model’s performance against existing methods, obtaining superior performance in several key tasks, including protein function prediction and module detection. MIRAGE source code is available at https://***/raminass/MIRAGE. A full manuscript describing MIRAGE and its applications is available at https://***/content/10.1101/2025.01.16.633332. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： Modal analysis

Crawling the Internal Knowledge-Base of Language Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cohen, Roi Geva, Mor Berant, Jonathan Globerson, Amir Blavatnik School of Computer Science Tel Aviv University Allen Institute for AI Google Research.

Language models are trained on large volumes of text, and as a result their parameters might contain a significant body of factual knowledge. Any downstream task performed by these models implicitly builds on these facts, and thus it is highly desirable to have means for representing this body of knowledge in an interpretable way. However, there is currently no mechanism for such a representation. Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model. We describe a procedure for "crawling" the internal knowledge-base of a language model. Specifically, given a seed entity, we expand a knowledge-graph around it. The crawling procedure is decomposed into sub-tasks, realized through specially designed prompts that control for both precision (i.e., that no wrong facts are generated) and recall (i.e., the number of facts generated). We evaluate our approach on graphs crawled starting from dozens of seed entities, and show it yields high precision graphs (82-92%), while emitting a reasonable number of facts per entity. © 2023, CC BY.

关键词： Knowledge graph