检索结果-内蒙古大学图书馆

ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNs

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Becking, Daniel Dreyer, Maximilian Samek, Wojciech Müller, Karsten Lapuschkin, Sebastian Department of Artificial Intelligence Fraunhofer Heinrich Hertz Institute Berlin Germany BIFOLD - Berlin Institute for the Foundations of Learning and Data Berlin Germany

The remarkable success of deep neural networks (DNNs) in various applications is accompanied by a significant increase in network parameters and arithmetic operations. Such increases in memory and computational demands make deep learning prohibitive for resource-constrained hardware platforms such as mobile devices. Recent efforts aim to reduce these overheads, while preserving model performance as much as possible, and include parameter reduction techniques, parameter quantization, and lossless compression techniques. In this chapter, we develop and describe a novel quantization paradigm for DNNs: Our method leverages concepts of explainable AI (XAI) and concepts of information theory: Instead of assigning weight values based on their distances to the quantization clusters, the assignment function additionally considers weight relevances obtained from Layer-wise Relevance Propagation (LRP) and the information content of the clusters (entropy optimization). The ultimate goal is to preserve the most relevant weights in quantization clusters of highest information content. Experimental results show that this novel Entropy-Constrained and XAI-adjusted Quantization (ECQx) method generates ultra low-precision (2-5 bit) and simultaneously sparse neural networks while maintaining or even improving model performance. Due to reduced parameter precision and high number of zero-elements, the rendered networks are highly compressible in terms of file size, up to 103× compared to the full-precision unquantized DNN model. Our approach was evaluated on different types of models and datasets (including Google Speech Commands, CIFAR-10 and Pascal VOC) and compared with previous work. Copyright © 2021, The Authors. All rights reserved.

关键词： Deep neural networks

So3krates: equivariant attention for interactions on arbitrary length-scales in molecular systems 22

学校读者我要写书评

暂无评论

So3krates: equivariant attention for interactions on arbitra...

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： J. Thorben Frank Oliver T. Unke Klaus-Robert Müller Machine Learning Group TU Berlin Berlin Germany and BIFOLD Berlin Institute for the Foundations of Learning and Data Germany Machine Learning Group TU Berlin Berlin Germany and BIFOLD Berlin Institute for the Foundations of Learning and Data Germany and Google Research Brain team Berlin Machine Learning Group TU Berlin Berlin Germany and BIFOLD Berlin Institute for the Foundations of Learning and Data Germany and Google Research Brain team Berlin and Department of Artificial Intelligence Korea University Seoul Korea and Max Planck Institut für Informatik Saarbrücken Germany

ISBN: (纸本)9781713871088

The application of machine learning methods in quantum chemistry has enabled the study of numerous chemical phenomena, which are computationally intractable with traditional ab-initio methods. However, some quantum mechanical properties of molecules and materials depend on non-local electronic effects, which are often neglected due to the difficulty of modeling them efficiently. This work proposes a modified attention mechanism adapted to the underlying physics, which allows to recover the relevant non-local effects. Namely, we introduce spherical harmonic coordinates (SPHCs) to reflect higher-order geometric information for each atom in a molecule, enabling a non-local formulation of attention in the SPHC space. Our proposed model So3KRATES- a self-attention based message passing neural network - uncouples geometric information from atomic features, making them independently amenable to attention mechanisms. Thereby we construct spherical filters, which extend the concept of continuous filters in Euclidean space to SPHC space and serve as foundation for a spherical self-attention mechanism. We show that in contrast to other published methods, So3krates is able to describe nonlocal quantum mechanical effects over arbitrary length scales. Further, we find evidence that the inclusion of higher-order geometric correlations increases data efficiency and improves generalization. So3krates matches or exceeds state- of-the-art performance on popular benchmarks, notably, requiring a significantly lower number of parameters (0.25-0.4x) while at the same time giving a substantial speedup (6-14x for training and 2-11x for inference) compared to other models.

关键词：

MambaLRP: Explaining Selective State Space Sequence Models 38

学校读者我要写书评

暂无评论

MambaLRP: Explaining Selective State Space Sequence Models

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Jafari, Farnoush Rezaei Montavon, Grégoire Müller, Klaus-Robert Eberle, Oliver Machine Learning Group Technische Universität Berlin Berlin10587 Germany BIFOLD - Berlin Institute for the Foundations of Learning and Data Berlin10587 Germany Department of Mathematics and Computer Science Freie Universität Berlin Arnimallee 14 Berlin14195 Germany Department of Artificial Intelligence Korea University Seoul136-713 Korea Republic of Max Planck Institute for Informatics Stuhlsatzenhausweg 4 Saarbrücken66123 Germany Google DeepMind Berlin Germany

Recent sequence modeling approaches using selective state space sequence models, referred to as Mamba models, have seen a surge of interest. These models allow efficient processing of long sequences in linear time and are rapidly being adopted in a wide range of applications such as language modeling, demonstrating promising performance. To foster their reliable use in real-world scenarios, it is crucial to augment their transparency. Our work bridges this critical gap by bringing explainability, particularly Layer-wise Relevance Propagation (LRP), to the Mamba architecture. Guided by the axiom of relevance conservation, we identify specific components in the Mamba architecture, which cause unfaithful explanations. To remedy this issue, we propose MambaLRP, a novel algorithm within the LRP framework, which ensures a more stable and reliable relevance propagation through these components. Our proposed method is theoretically sound and excels in achieving state-of-the-art explanation performance across a diverse range of models and datasets. Moreover, MambaLRP facilitates a deeper inspection of Mamba architectures, uncovering various biases and evaluating their significance. It also enables the analysis of previous speculations regarding the long-range capabilities of Mamba models. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

xMIL: Insightful Explanations for Multiple Instance learning in Histopathology 38

学校读者我要写书评

暂无评论

xMIL: Insightful Explanations for Multiple Instance Learning...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Hense, Julius Idaji, Mina Jamshidi Eberle, Oliver Schnake, Thomas Dippel, Jonas Ciernik, Laure Buchstab, Oliver Mock, Andreas Klauschen, Frederick Müller, Klaus-Robert Berlin Institute for the Foundations of Learning and Data Berlin Germany Machine Learning Group Technische Universität Berlin Berlin Germany Aignostics GmbH Berlin Germany Institute of Pathology Ludwig Maximilian University Munich Germany German Cancer Research Center Heidelberg Germany German Cancer Consortium Munich Germany Institute of Pathology Charité Universitätsmedizin Berlin Germany Department of Artificial Intelligence Korea University Seoul Korea Republic of Max-Planck Institute for Informatics Saarbrücken Germany

Multiple instance learning (MIL) is an effective and widely used approach for weakly supervised machine learning. In histopathology, MIL models have achieved remarkable success in tasks like tumor detection, biomarker prediction, and outcome prognostication. However, MIL explanation methods are still lagging behind, as they are limited to small bag sizes or disregard instance interactions. We revisit MIL through the lens of explainable AI (XAI) and introduce xMIL, a refined framework with more general assumptions. We demonstrate how to obtain improved MIL explanations using layer-wise relevance propagation (LRP) and conduct extensive evaluation experiments on three toy settings and four real-world histopathology datasets. Our approach consistently outperforms previous explanation attempts with particularly improved faithfulness scores on challenging biomarker prediction tasks. Finally, we showcase how xMIL explanations enable pathologists to extract insights from MIL models, representing a significant advance for knowledge discovery and model debugging in digital histopathology. Codes are available at: https://***/bifold-pathomics/xMIL. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

Understanding the (Extra-)Ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Dreyer, Maximilian Achtibat, Reduan Samek, Wojciech Lapuschkin, Sebastian Fraunhofer Heinrich Hertz Institute Germany Technical University of Berlin Germany BIFOLD – Berlin Institute for the Foundations of Learning and Data Germany

Ensuring both transparency and safety is critical when deploying Deep Neural Networks (DNNs) in high-risk applications, such as medicine. The field of explainable AI (XAI) has proposed various methods to comprehend the decision-making processes of opaque DNNs. However, only few XAI methods are suitable of ensuring safety in practice as they heavily rely on repeated labor-intensive and possibly biased human assessment. In this work, we present a novel post-hoc concept-based XAI framework that conveys besides instance-wise (local) also class-wise (global) decision-making strategies via prototypes. What sets our approach apart is the combination of local and global strategies, enabling a clearer understanding of the (dis-)similarities in model decisions compared to the expected (prototypical) concept use, ultimately reducing the dependence on human long-term assessment. Quantifying the deviation from prototypical behavior not only allows to associate predictions with specific model sub-strategies but also to detect outlier behavior. As such, our approach constitutes an intuitive and explainable tool for model validation. We demonstrate the effectiveness of our approach in identifying out-of-distribution samples, spurious model behavior and data quality issues across three datasets (ImageNet, CUB-200, and CIFAR-10) utilizing VGG, ResNet, and EfficientNet architectures. Code is available at https://***/maxdreyer/pcx. Copyright © 2023, The Authors. All rights reserved.

关键词： Deep neural networks

Reactive Model Correction: Mitigating Harm to Task-Relevant Features via Conditional Bias Suppression

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Bareeva, Dilyara Dreyer, Maximilian Pahde, Frederik Samek, Wojciech Lapuschkin, Sebastian Fraunhofer Heinrich Hertz Institute Germany Technical University of Berlin Germany BIFOLD - Berlin Institute for the Foundations of Learning and Data Germany

Deep Neural Networks are prone to learning and relying on spurious correlations in the training data, which, for high-risk applications, can have fatal consequences. Various approaches to suppress model reliance on harmful features have been proposed that can be applied post-hoc without additional training. Whereas those methods can be applied with efficiency, they also tend to harm model performance by globally shifting the distribution of latent features. To mitigate unintended overcorrection of model behavior, we propose a reactive approach conditioned on model-derived knowledge and eXplainable Artificial Intelligence (XAI) insights. While the reactive approach can be applied to many post-hoc methods, we demonstrate the incorporation of reactivity in particular for P-ClArC (Projective Class Artifact Compensation), introducing a new method called R-ClArC (Reactive Class Artifact Compensation). Through rigorous experiments in controlled settings (FunnyBirds) and with a real-world dataset (ISIC2019), we show that introducing reactivity can minimize the detrimental effect of the applied correction while simultaneously ensuring low reliance on spurious features. © 2024, CC BY-SA.

关键词： Deep neural networks

LIT-4-RSVQA: LIGHTWEIGHT TRANSFORMER-BASED VISUAL QUESTION ANSWERING IN REMOTE SENSING

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hackel, Leonard Clasen, Kai Norman Ravanbakhsh, Mahdyar Demir, Begüm Technische Universität Berlin Germany Zalando SE Berlin Germany BIFOLD Berlin Institute for the Foundations of Learning and Data Germany

Visual question answering (VQA) methods in remote sensing (RS) aim to answer natural language questions with respect to an RS image. Most of the existing methods require a large amount of computational resources, which limits their application in operational scenarios in RS. To address this issue, in this paper we present an effective lightweight transformer-based VQA in RS (LiT-4-RSVQA) architecture for efficient and accurate VQA in RS. Our architecture consists of: i) a lightweight text encoder module;ii) a lightweight image encoder module;iii) a fusion module;and iv) a classification module. The experimental results obtained on a VQA benchmark dataset demonstrate that our proposed LiT-4-RSVQA architecture provides accurate VQA results while significantly reducing the computational requirements on the executing hardware. Our code is publicly available at https://***/rsim/lit4rsvqa. Copyright © 2023, The Authors. All rights reserved.

关键词： Remote sensing

LIT-4-RSVQA: Lightweight Transformer-Based Visual Question Answering in Remote Sensing

学校读者我要写书评

暂无评论

LIT-4-RSVQA: Lightweight Transformer-Based Visual Question A...

IEEE International Symposium on Geoscience and Remote Sensing (IGARSS)

作者： Leonard Hackel Kai Norman Clasen Mahdyar Ravanbakhsh Begüm Demir Technische Universität Berlin Germany BIFOLD – Berlin Institute for the Foundations of Learning and Data Germany Zalando SE Berlin Germany

Visual question answering (VQA) methods in remote sensing (RS) aim to answer natural language questions with respect to an RS image. Most of the existing methods require a large amount of computational resources, which limits their application in operational scenarios in RS. To address this issue, in this paper we present an effective lightweight transformer-based VQA in RS (LiT-4-RSVQA) architecture for efficient and accurate VQA in RS. Our architecture consists of: i) a lightweight text encoder module; ii) a lightweight image encoder module; iii) a fusion module; and iv) a classification module. The experimental results obtained on a VQA benchmark dataset demonstrate that our proposed LiT-4-RSVQA architecture provides accurate VQA results while significantly reducing the computational requirements on the executing hardware.

关键词：

PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Dreyer, Maximilian Purelku, Erblina Vielhaben, Johanna Samek, Wojciech Lapuschkin, Sebastian Fraunhofer Heinrich Hertz Institute Germany Technical University of Berlin Germany BIFOLD – Berlin Institute for the Foundations of Learning and Data Germany

The field of mechanistic interpretability aims to study the role of individual neurons in Deep Neural Networks. Single neurons, however, have the capability to act poly-semantically and encode for multiple (unrelated) features, which renders their interpretation difficult. We present a method for disentangling polysemanticity of any Deep Neural Network by decomposing a polysemantic neuron into multiple monosemantic "virtual" neurons. This is achieved by identifying the relevant sub-graph ("circuit") for each "pure" feature. We demonstrate how our approach allows us to find and disentangle various polysemantic units of ResNet models trained on ImageNet. While evaluating feature visualizations using CLIP, our method effectively disentangles representations, improving upon methods based on neuron activations. Our code is available at https://***/maxdreyer/PURE. Copyright © 2024, The Authors. All rights reserved.

关键词： Neurons