检索结果-内蒙古大学图书馆

S$^\text{3}$Attention: Improving Long Sequence Attention With Smoothed Skeleton Sketching

IEEE Journal of Selected Topics in Signal Processing 2024年第6期18卷 985-996页

作者： Xue Wang Tian Zhou Jianqing Zhu Jialin Liu Kun Yuan Tao Yao Wotao Yin Rong Jin HanQin Cai Alibaba Group Bellevue WA USA Computer Electrical and Mathematical Science and Engineering Division King Abdullah University of Science and Technology Thuwal Saudi Arabia Department of Statistics and Data Science University of Central Florida Orlando FL USA Center for Machine Learning Research Peking University Beijing China Antai College of Economics and Management Shanghai Jiao Tong University Shanghai China Meta Menlo Park CA USA Department of Statistics and Data Science and the Department of Computer Science University of Central Florida Orlando FL USA

Attention based models have achieved many remarkable breakthroughs in numerous applications. However, the quadratic complexity of Attention makes the vanilla Attention based models hard to apply to long sequence tasks. Various improved Attention structures are proposed to reduce the computation cost by inducing low rankness and approximating the whole sequence by sub-sequences. The most challenging part of those approaches is maintaining the proper balance between information preservation and computation reduction: the longer sub-sequences used, the better information is preserved, but at the price of introducing more noise and computational costs. In this paper, we propose a smoothed skeleton sketching based Attention structure, coined S $^{3}$ Attention, which significantly improves upon the previous attempts to negotiate this trade-off. S $^{3}$ Attention has two mechanisms to effectively minimize the impact of noise while keeping the linear complexity to the sequence length: a smoothing block to mix information over long sequences and a matrix sketching method that simultaneously selects columns and rows from the input matrix. We verify the effectiveness of S $^{3}$ Attention both theoretically and empirically. Extensive studies over Long Range Arena (LRA) datasets and six time-series forecasting show that S $^{3}$ Attention significantly outperforms both vanilla Attention and other state-of-the-art variants of Attention structures.

关键词： Computational modeling Complexity theory Transformers Task analysis Skeleton Kernel Noise

来源：评论

学校读者我要写书评

暂无评论

COMPUTER AUDITION: FROM TASK-SPECIFIC machine learning TO FOUNDATION MODELS

arXiv

引用

arXiv 2024年

作者： Triantafyllopoulos, Andreas Tsangko, Iosif Gebhard, Alexander Mesaros, Annamaria Virtanen, Tuomas Schuller, Björn W. CHI – Chair of Health Informatics Technical University of Munich MRI Munich Germany Audio Research Group Tampere University Tampere Finland EIHW – Chair of Embedded Intelligence for Health Care and Wellbeing University of Augsburg Augsburg Germany GLAM – Group on Language Audio & Music Imperial College London United Kingdom MCML – Munich Center for Machine Learning Munich Germany MDSI – Munich Data Science Institute Munich Germany

Foundation models (FMs) are increasingly spearheading recent advances on a variety of tasks that fall under the purview of computer audition – the use of machines to understand sounds. They feature several advantages over traditional pipelines: among others, the ability to consolidate multiple tasks in a single model, the option to leverage knowledge from other modalities, and the readily-available interaction with human users. Naturally, these promises have created substantial excitement in the audio community, and have led to a wave of early attempts to build new, general-purpose foundation models for audio. In the present contribution, we give an overview of computational audio analysis as it transitions from traditional pipelines towards auditory foundation models. Our work highlights the key operating principles that underpin those models, and showcases how they can accommodate multiple tasks that the audio community previously tackled separately. Copyright © 2024, The Authors. All rights reserved.

关键词： Pipelines

来源：评论

学校读者我要写书评

暂无评论

Predicting High School Students' Academic Performance: A Comparative Study of Supervised machine learning Techniques

Predicting High School Students' Academic Performance: A Com...

引用

machine learning-Driven Digital Technologies for Educational Innovation Workshop

作者： Nadia N. Sánchez-Pozo Juan S. Mejía-Ordóñez Diana C. Chamorro Dagoberto Mayorca-Torres Diego H. Peluffo-Ordóñez Machine learning Research Program SDAS Research Group Ben Guerir Morocco Carrera de Ingeniería Química Faculted de Ciencias Químicas y de la Salud Universidad Técnica de Machala Machala Ecuador Programa de Mecatrónica Facultad de Ingeniería Universidad Mariana Pasto Colombia Modeling Simulation and Data Analysis (MSDA) Research Program Mohamed VI Polytechnique University Ben Guerir Morocco

ISBN: (纸本)9781665427647

The proliferation of mobile devices and the rapid development of information and communication technologies have revolutionized education. Educational data has evolved to be voluminously massive, broadly various, and produced at high velocity. Therefore, computerized techniques for integrating, processing, and transforming data into valuable knowledge have become necessary to improve internal academic processes. Specifically, educational data mining is an emerging discipline concerned with analyzing the massive amounts of academic data generated and stored by educational institutions. In this sense, machine learning algorithms aid decision-makers who are establishing strategies to improve students' learning experience and institutional effectiveness by revealing hidden patterns in academic performance. Thus, this paper describes our comparative study of machine learning techniques to predict academic performance. We selected the features that best fit the discovery of patterns in the academic performance of high school students, resulting in a balance between accuracy and interpretability. We implemented six supervised learning algorithms for pattern recognition: Light Gradient Boosting machine, Gradient Boosting, AdaBoost, Logistic Regression, Random Forest, and K-nearest Neighbors. The experimental results showed that the Gradient Boosting (Gbc) algorithm achieved the highest accuracy (96.77%), superior to other classification techniques considered.

关键词： Technological innovation machine learning algorithms Social networking (online) Predictive models Boosting Prediction algorithms Classification algorithms

来源：评论

学校读者我要写书评

暂无评论

Improving Generative Model-based Unfolding with Schrödinger Bridges

arXiv

引用

arXiv 2023年

作者： Diefenbacher, Sascha Liu, Guan-Horng Mikuni, Vinicius Nachman, Benjamin Nie, Weili Physics Division Lawrence Berkeley National Laboratory BerkeleyCA94720 United States Autonomous Control and Decision Systems Laboratory Georgia Institute of Technology AtlantaGA30332 United States National Energy Research Scientific Computing Center Berkeley Lab BerkeleyCA94720 United States Berkeley Institute for Data Science University of California BerkeleyCA94720 United States Machine Learning Research Group NVIDIA Research United States

machine learning-based unfolding has enabled unbinned and high-dimensional differential cross section measurements. Two main approaches have emerged in this research area: one based on discriminative models and one based on generative models. The main advantage of discriminative models is that they learn a small correction to a starting simulation while generative models scale better to regions of phase space with little data. We propose to use Schrödinger Bridges and diffusion models to create SBUnfold, an unfolding approach that combines the strengths of both discriminative and generative models. The key feature of SBUnfold is that its generative model maps one set of events into another without having to go through a known probability density as is the case for normalizing flows and standard diffusion models. We show that SBUnfold achieves excellent performance compared to state of the art methods on a synthetic Z+jets dataset. © 2023, CC BY.

关键词： Phase space methods

来源：评论

学校读者我要写书评

暂无评论

Accurate machine Learned Quantum-Mechanical Force Fields for Biomolecular Simulations

arXiv

引用

arXiv 2022年

作者： Unke, Oliver T. Stöhr, Martin Ganscha, Stefan Unterthiner, Thomas Maennel, Hartmut Kashubin, Sergii Ahlin, Daniel Gastegger, Michael Sandonas, Leonardo Medrano Tkatchenko, Alexandre Müller, Klaus-Robert Google Research Brain Team Machine Learning Group Technische Universität Berlin Berlin10587 Germany Technische Universität Berlin Berlin10623 Germany Department of Physics and Materials Science University of Luxembourg Luxembourg CityL-1511 Luxembourg BASLEARN TU Berlin Berlin10587 Germany BASF Joint Lab for Machine Learning Technische Universität Berlin Berlin10587 Germany Department of Artificial Intelligence Korea University Anam-dong Seongbuk-gu Seoul02841 Korea Republic of Max Planck Institute for Informatics Stuhlsatzenhausweg Saarbrücken66123 Germany BIFOLD Berlin Institute for the Foundations of Learning and Data Berlin Germany

Molecular dynamics (MD) simulations allow atomistic insights into chemical and biological processes. Accurate MD simulations require computationally demanding quantum-mechanical calculations, being practically limited to short timescales and few atoms. For larger systems, efficient, but much less reliable empirical force fields are used. Recently, machine learned force fields (MLFFs) emerged as an alternative means to execute MD simulations, offering similar accuracy as ab initio methods at orders-of-magnitude speedup. Until now, MLFFs mainly capture short-range interactions in small molecules or periodic materials, due to the increased complexity of constructing models and obtaining reliable reference data for large molecules, where long-ranged many-body effects become important. This work proposes a general approach to constructing accurate MLFFs for large-scale molecular simulations (GEMS) by training on "bottom-up" and "top-down" molecular fragments of varying size, from which the relevant physicochemical interactions can be learned. GEMS is applied to study the dynamics of alanine-based peptides and the 46-residue protein crambin in aqueous solution, allowing nanosecond-scale MD simulations of >25k atoms at essentially ab initio quality. Our findings suggest that structural motifs in peptides and proteins are more flexible than previously thought, indicating that simulations at ab initio accuracy might be necessary to understand dynamic biomolecular processes such as protein (mis)folding, drug–protein binding, or allosteric regulation. Copyright © 2022, The Authors. All rights reserved.

关键词： Molecular dynamics

来源：评论

学校读者我要写书评

暂无评论

Layerwise learning for quantum neural networks

arXiv

引用

arXiv 2020年

作者： Skolik, Andrea McClean, Jarrod R. Mohseni, Masoud van der Smagt, Patrick Leib, Martin Volkswagen Data:Lab Ungererstraße 69 Munich80805 Germany Ludwig Maximilian University Theresienstraße 39 Munich80333 Germany Leiden University Niels Bohrweg 1 Leiden2333 CA Netherlands Google Research 340 Main Street VeniceCA90291 United States Volkswagen Group Machine Learning Research Lab Munich Germany Eötvös Loránd University Budapest Hungary

With the increased focus on quantum circuit learning for near-term applications on quantum devices, in conjunction with unique challenges presented by cost function landscapes of parametrized quantum circuits, strategies for effective training are becoming increasingly important. In order to ameliorate some of these challenges, we investigate a layerwise learning strategy for parametrized quantum circuits. The circuit depth is incrementally grown during optimization, and only subsets of parameters are updated in each training step. We show that when considering sampling noise, this strategy can help avoid the problem of barren plateaus of the error surface due to the low depth of circuits, low number of parameters trained in one step, and larger magnitude of gradients compared to training the full circuit. These properties make our algorithm preferable for execution on noisy intermediate-scale quantum devices. We demonstrate our approach on an image-classification task on handwritten digits, and show that layerwise learning attains an 8% lower generalization error on average in comparison to standard learning schemes for training quantum circuits of the same size. Additionally, the percentage of runs that reach lower test errors is up to 40% larger compared to training the full circuit, which is susceptible to creeping onto a plateau during training. Copyright © 2020, The Authors. All rights reserved.

关键词： Timing circuits

来源：评论

学校读者我要写书评

暂无评论

Heat flux for semilocal machine-learning potentials

引用

Physical Review B 2023年第10期108卷 L100302-L100302页

作者： Marcel F. Langer Florian Knoop Christian Carbogno Matthias Scheffler Matthias Rupp Machine Learning Group Technische Universität Berlin 10587 Berlin Germany Berlin Institute for the Foundations of Learning and Data 10623 Berlin Germany The NOMAD Laboratory at the FHI of the Max-Planck-Gesellschaft and IRIS Adlershof of the Humboldt Universität zu Berlin 14195 Berlin Germany Theoretical Physics Division Department of Physics Chemistry and Biology (IFM) Linköping University 581 83 Linköping Sweden Department of Computer and Information Science University of Konstanz 78464 Konstanz Germany Materials Research and Technology Department Luxembourg Institute of Science and Technology Belvaux Luxembourg

The Green-Kubo (GK) method is a rigorous framework for heat transport simulations in materials. However, it requires an accurate description of the potential-energy surface and carefully converged statistics. machine-learning potentials can achieve the accuracy of first-principles simulations while allowing to reach well beyond their simulation time and length scales at a fraction of the cost. In this Letter, we explain how to apply the GK approach to the recent class of message-passing machine-learning potentials, which iteratively consider semilocal interactions beyond the initial interaction cutoff. We derive an adapted heat flux formulation that can be implemented using automatic differentiation without compromising computational efficiency. The approach is demonstrated and validated by calculating the thermal conductivity of zirconium dioxide across temperatures.

关键词： Lattice thermal conductivity machine learning Molecular dynamics Neural network simulations

来源：评论

学校读者我要写书评

暂无评论

xMIL: Insightful Explanations for Multiple Instance learning in Histopathology

arXiv

引用

arXiv 2024年

作者： Hense, Julius Idaji, Mina Jamshidi Eberle, Oliver Schnake, Thomas Dippel, Jonas Ciernik, Laure Buchstab, Oliver Mock, Andreas Klauschen, Frederick Müller, Klaus-Robert Berlin Institute for the Foundations of Learning and Data Berlin Germany Machine Learning Group Technische Universität Berlin Berlin Germany Aignostics GmbH Berlin Germany Institute of Pathology Ludwig Maximilian University Munich Germany German Cancer Research Center Heidelberg Germany German Cancer Consortium Munich Germany Institute of Pathology Charité Universitätsmedizin Berlin Germany Department of Artificial Intelligence Korea University Seoul Korea Republic of Max-Planck Institute for Informatics Saarbrücken Germany

Multiple instance learning (MIL) is an effective and widely used approach for weakly supervised machine learning. In histopathology, MIL models have achieved remarkable success in tasks like tumor detection, biomarker prediction, and outcome prognostication. However, MIL explanation methods are still lagging behind, as they are limited to small bag sizes or disregard instance interactions. We revisit MIL through the lens of explainable AI (XAI) and introduce xMIL, a refined framework with more general assumptions. We demonstrate how to obtain improved MIL explanations using layer-wise relevance propagation (LRP) and conduct extensive evaluation experiments on three toy settings and four real-world histopathology datasets. Our approach consistently outperforms previous explanation attempts with particularly improved faithfulness scores on challenging biomarker prediction tasks. Finally, we showcase how xMIL explanations enable pathologists to extract insights from MIL models, representing a significant advance for knowledge discovery and model debugging in digital histopathology. Codes are available at: https://***/bifold-pathomics/xMIL. © 2024, CC BY-NC-ND.

关键词： Self-supervised learning

来源：评论

学校读者我要写书评

暂无评论

DORA: Exploring Outlier Representations in Deep Neural Networks

arXiv

引用

arXiv 2022年

作者： Bykov, Kirill Deb, Mayukh Grinwald, Dennis Müller, Klaus-Robert Höhne, Marina M.-C. Potsdam Germany Technical University of Berlin Berlin Germany Machine Learning Group Technical University of Berlin Berlin Germany BIFOLD – Berlin Institute for the Foundations of Learning and Data Berlin Germany Department of Artificial Intelligence Korea University Seoul136-713 Korea Republic of Max Planck Institut für Informatik Saarbrücken66123 Germany Google Research Brain Team Berlin Germany Department of Computer Science University of Potsdam Germany Department of Physics and Technology UiT Arctic University of Norway Norway

Deep Neural Networks (DNNs) excel at learning complex abstractions within their internal representations. However, the concepts they learn remain opaque, a problem that becomes particularly acute when models unintentionally learn spurious correlations. In this work, we present DORA (data-agnOstic Representation Analysis), the first data-agnostic framework for analyzing the representational space of DNNs. Central to our framework is the proposed Extreme-Activation (EA) distance measure, which assesses similarities between representations by analyzing their activation patterns on data points that cause the highest level of activation. As spurious correlations often manifest in features of data that are anomalous to the desired task, such as watermarks or artifacts, we demonstrate that internal representations capable of detecting such artifactual concepts can be found by analyzing relationships within neural representations. We validate the EA metric quantitatively, demonstrating its effectiveness both in controlled scenarios and real-world applications. Finally, we provide practical examples from popular Computer Vision models to illustrate that representations identified as outliers using the EA metric often correspond to undesired and spurious concepts. © 2022, CC BY.

关键词： Chemical activation

来源：评论

学校读者我要写书评

暂无评论

Robust and Fast Measure of Information via Low-rank Representation

arXiv

引用

arXiv 2022年

作者： Dong, Yuxin Gong, Tieliang Yu, Shujian Chen, Hong Li, Chen School of Computer Science and Technology Xi’an Jiaotong University Xi’an710049 China Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering Ministry of Education Xi’an710049 China Machine Learning Group UiT - The Arctic University of Norway Norway College of Science Huazhong Agriculture University Wuhan430070 China Engineering Research Center of Intelligent Technology for Agriculture Ministry of Education Wuhan430070 China

The matrix-based Rényi’s entropy allows us to directly quantify information measures from given data, without explicit estimation of the underlying probability distribution. This intriguing property makes it widely applied in statistical inference and machine learning tasks. However, this information theoretical quantity is not robust against noise in the data, and is computationally prohibitive in large-scale applications. To address these issues, we propose a novel measure of information, termed low-rank matrix-based Rényi’s entropy, based on low-rank representations of infinitely divisible kernel matrices. The proposed entropy functional inherits the specialty of of the original definition to directly quantify information from data, but enjoys additional advantages including robustness and effective calculation. Specifically, our low-rank variant is more sensitive to informative perturbations induced by changes in underlying distributions, while being insensitive to uninformative ones caused by noises. Moreover, low-rank Rényi’s entropy can be efficiently approximated by random projection and Lanczos iteration techniques, reducing the overall complexity from O(n3) to O(n2s) or even O(ns2), where n is the number of data samples and s n. We conduct large-scale experiments to evaluate the effectiveness of this new information measure, demonstrating superior results compared to matrix-based Rényi’s entropy in terms of both performance and computational efficiency. Copyright © 2022, The Authors. All rights reserved.

关键词： Entropy

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：