检索结果-内蒙古大学图书馆

Learning Physics-Informed Neural Networks without Stacked Back-propagation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： He, Di Li, Shanda Shi, Wenlei Gao, Xiaotian Zhang, Jia Bian, Jiang Wang, Liwei Liu, Tie-Yan National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University China Machine Learning Department School of Computer Science Carnegie Mellon University United States Microsoft Research Center for Data Science Peking University China

Physics-Informed Neural Network (PINN) has become a commonly used machine learning approach to solve partial differential equations (PDE). But, facing high-dimensional second-order PDE problems, PINN will suffer from severe scalability issues since its loss includes second-order derivatives, the computational cost of which will grow along with the dimension during stacked back-propagation. In this work, we develop a novel approach that can significantly accelerate the training of Physics-Informed Neural Networks. In particular, we parameterize the PDE solution by the Gaussian smoothed model and show that, derived from Stein's Identity, the second-order derivatives can be efficiently calculated without back-propagation. We further discuss the model capacity and provide variance reduction methods to address key limitations in the derivative estimation. Experimental results show that our proposed method can achieve competitive error compared to standard PINN training but is significantly faster. Our code is released at https://***/LithiumDA/PINN-without-Stacked-BP. Copyright © 2022, The Authors. All rights reserved.

关键词： Backpropagation

Online training through time for spiking neural networks 22

学校读者我要写书评

暂无评论

Online training through time for spiking neural networks

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Mingqing Xiao Qingyan Meng Zongpeng Zhang Di He Zhouchen Lin Key Lab. of Machine Perception (MoE) School of Intelligence Science and Technology Peking University The Chinese University of Hong Kong Shenzhen and Shenzhen Research Institute of Big Data Center for Data Science Academy for Advanced Interdisciplinary Studies Peking University Key Lab. of Machine Perception (MoE) School of Intelligence Science and Technology Peking University and Institute for Artificial Intelligence Peking University and Peng Cheng Laboratory China

ISBN: (纸本)9781713871088

Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to enable models to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsistency with the online property of biological learning rules and rules on neuromorphic hardware. Other works connect the spike representations of SNNs with equivalent artificial neural network formulation and train SNNs by gradients from equivalent mappings to ensure descent directions. But they fail to achieve low latency and are also not online. In this work, we propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove that the gradients of OTTT can provide a similar descent direction for optimization as gradients from equivalent mapping between spike representations under both feedforward and recurrent conditions. OTTT only requires constant training memory costs agnostic to time steps, avoiding the significant memory costs of BPTT for GPU training. Furthermore, the update rule of OTTT is in the form of three-factor Hebbian learning, which could pave a path for online on-chip learning. With OTTT, it is the first time that the two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile it is in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on large-scale static and neuromorphic datasets in a small number of time steps.

关键词：

Topology-Preserving Automatic Labeling of Coronary Arteries via Anatomy-aware Connection Classifier

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Zhixing Zhao, Ziwei Wang, Dong Zhao, Shishuang Liu, Yuhang Liu, Jia Wang, Liwei Center for Data Science Peking University Beijing China National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University Beijing China Yizhun Medical AI Co. Ltd Beijing China Peking University First Hospital Beijing China Center for Machine Learning Research Peking University Beijing China Pazhou Lab Guangzhou China

Automatic labeling of coronary arteries is an essential task in the practical diagnosis process of cardiovascular diseases. For experienced radiologists, the anatomically predetermined connections are important for labeling the artery segments accurately, while this prior knowledge is barely explored in previous studies. In this paper, we present a new framework called TopoLab which incorporates the anatomical connections into the network design explicitly. Specifically, the strategies of intra-segment feature aggregation and inter-segment feature interaction are introduced for hierarchical segment feature extraction. Moreover, we propose the anatomy-aware connection classifier to enable classification for each connected segment pair, which effectively exploits the prior topology among the arteries with different categories. To validate the effectiveness of our method, we contribute high-quality annotations of artery labeling to the public orCaScore dataset. The experimental results on both the orCaScore dataset and an in-house dataset show that our TopoLab has achieved state-of-the-art performance. © 2023, CC BY.

关键词： Topology

NC-ALG: Graph-Based Active Learning Under Noisy Crowd

学校读者我要写书评

暂无评论

NC-ALG: Graph-Based Active Learning Under Noisy Crowd

International Conference on data Engineering

作者： Wentao Zhang Yexin Wang Zhenbang You Yang Li Gang Cao Zhi Yang Bin Cui Center for Machine Learning Research Peking University Institute of Advanced Algorithms Research Shanghai National Engineering Labratory for Big Data Analytics and Applications Key Lab of High Confidence Software Technologies Peking University Department of Data Platform TEG Tencent Inc. Beijing Academy of Artificial Intelligence Institute of Computational Social Science Peking University Qingdao

ISBN: (数字)9798350317152

ISBN: (纸本)9798350317169

Graph Neural Networks (GNNs) have achieved great success in various data mining tasks but they heavily rely on a large number of annotated nodes, requiring considerable human efforts. Despite the effectiveness of existing GNN-based Active Learning (AL) methods, they assume that the annotated labels are always correct, which is contradictory to the error-prone labeling process in a practical crowdsourcing environment. Besides, due to this impractical assumption, existing works only focus on optimizing the node selection in AL but neglect optimizing the labeling process. Therefore, we present NC-ALG, the first GNN-based AL framework that optimizes both the node selection and node labeling process under a noisy crowd. For node selection, NC-ALG introduces a new measurement to model influence reliability and an effective influence maximization objective to select nodes. For node labeling, NC-ALG significantly reduces the labeling cost by considering the model-predicted labels and the labels of mirror nodes. To the best of our knowledge, this is the first attempt to consider GNN-based AL under the practical noisy crowd. Empirical studies on public datasets demonstrate that NC-ALG significantly outperforms existing methods in terms labeling efficiency. Notably, it only takes NC-ALG one-third of the labeling budget that the competitive baseline GRAIN needs to achieve an accuracy of 70.7 % on PubMed.

关键词： Crowdsourcing Costs Noise Predictive models Graph neural networks Labeling Noise measurement

Enhancing SNN-based Spatio-Temporal Learning: A Benchmark dataset and Cross-Modality Attention Model

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Shibo Yang, Bo Yuan, Mengwen Jiang, Runhao Yan, Rui Pan, Gang Tang, Huajin Research Center for Data Hub and Security Zhejiang Lab Hangzhou China College of Computer Science and Technology Zhejiang University Hangzhou China Research Center for High Efficiency Computing System Zhejiang Lab Hangzhou China College of Computer Science and Technology Zhejiang University of Technology Hangzhou China The State Key Lab of Brain-Machine Intelligence Zhejiang University Hangzhou China

Spiking Neural Networks (SNNs), renowned for their low power consumption, brain-inspired architecture, and spatio-temporal representation capabilities, have garnered considerable attention in recent years. Similar to Artificial Neural Networks (ANNs), high-quality benchmark datasets are of great importance to the advances of SNNs. However, our analysis indicates that many prevalent neuromorphic datasets lack strong temporal correlation, preventing SNNs from fully exploiting their spatio-temporal representation capabilities. Meanwhile, the integration of event and frame modalities offers more comprehensive visual spatio-temporal information. Yet, the SNN-based cross-modality fusion remains underexplored. In this work, we present a neuromorphic dataset called DVS-SLR that can better exploit the inherent spatio-temporal properties of SNNs. Compared to existing datasets, it offers advantages in terms of higher temporal correlation, larger scale, and more varied scenarios. In addition, our neuromorphic dataset contains corresponding frame data, which can be used for developing SNN-based fusion methods. By virtue of the dual-modal feature of the dataset, we propose a Cross-Modality Attention (CMA) based fusion method. The CMA model efficiently utilizes the unique advantages of each modality, allowing for SNNs to learn both temporal and spatial attention scores from the spatio-temporal features of event and frame modalities, subsequently allocating these scores across modalities to enhance their synergy. Experimental results demonstrate that our method not only improves recognition accuracy but also ensures robustness across diverse scenarios. © 2024, CC0.

关键词： Spatio-temporal data

A Privacy-Preserving Framework for Collaborative machine Learning with Kernel Methods

学校读者我要写书评

暂无评论

A Privacy-Preserving Framework for Collaborative Machine Lea...

IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA)

作者： Anika Hannemann Ali Burak Ünal Arjhun Swaminathan Erik Buchmann Mete Akgün Dept. of Computer Science Leipzig University Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig Germany Medical Data Privacy and Privacy-preserving Machine Learning (MDPPML) University of Tübingen Institute for Bioinformatics and Medical Informatics (IBMI) University of Tübingen Germany

It is challenging to implement Kernel methods, if the data sources are distributed and cannot be joined at a trusted third party for privacy reasons. It is even more challenging, if the use case rules out privacy-preserving approaches that introduce noise or entail significant computational overhead. An example for such a use case is machine learning on clinical data. To realize exact and efficient privacy preserving computation of kernel methods, we propose FLAKE, a Framework for Learning with Anonymized KErnels on horizontally distributed data. With our method, the data sources mask their data so that a Gram matrix can be computed without compromising privacy or utility. The Gram matrix allows to calculate many kernel matrices, which can be used to train kernel-based machine learning algorithms such as Support Vector machines. We prove that our framework prevents an adversary from learning the input data or the number of input features under a semi-honest threat model. The conducted experiments on clinical, genomic, and image data provide confirmation that our approach is applicable across a wide range of settings. Additionally, our method outperforms comparable approaches in both computational efficiency and accuracy. Thus, FLAKE is a lightweight, applicable approach suitable for various use cases.

关键词：

Is L2 physics-informed loss always suitable for training physics-informed neural network? 22

学校读者我要写书评

暂无评论

Is L2 physics-informed loss always suitable for training phy...

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Chuwei Wang Shanda Li Di He Liwei Wang School of Mathematical Sciences Peking University Machine Learning Department School of Computer Science Carnegie Mellon University and Zhejiang Lab National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University and Center for Data Science Peking University

ISBN: (纸本)9781713871088

The Physics-Informed Neural Network (PINN) approach is a new and promising way to solve partial differential equations using deep learning. The L2 Physics- Informed Loss is the de-facto standard in training Physics-Informed Neural Networks. In this paper, we challenge this common practice by investigating the relationship between the loss function and the approximation quality of the learned solution. In particular, we leverage the concept of stability in the literature of partial differential equation to study the asymptotic behavior of the learned solution as the loss approaches zero. With this concept, we study an important class of high-dimensional non-linear PDEs in optimal control, the Hamilton-Jacobi- Bellman (HJB) Equation, and prove that for general Lp Physics-Informed Loss, a wide class of HJB equation is stable only if p is sufficiently large. Therefore, the commonly used L2 loss is not suitable for training PINN on those equations, while L∞ loss is a better choice. Based on the theoretical insight, we develop a novel PINN training algorithm to minimize the L∞ loss for HJB equations which is in a similar spirit to adversarial training. The effectiveness of the proposed algorithm is empirically demonstrated through experiments.

关键词：

Your transformer may not be as powerful as you expect 22

学校读者我要写书评

暂无评论

Your transformer may not be as powerful as you expect

Proceedings of the 36th International Conference on Neural Information Processing Systems

作者： Shengjie Luo Shanda Li Shuxin Zheng Tie-Yan Liu Liwei Wang Di He National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University and Zhejiang Lab Machine Learning Department School of Computer Science Carnegie Mellon University Microsoft Research National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University and Center for Data Science Peking University National Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University

ISBN: (纸本)9781713871088

Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far as we know, theoretical understanding of the RPE-based Transformers is largely unexplored. In this work, we mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. One may naturally assume the answer is in the affirmative—RPE-based Transformers are universal function approximators. However, we present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. One key reason lies in that most RPEs are placed in the softmax attention that always generates a right stochastic matrix. This restricts the network from capturing positional information in the RPEs and limits its capacity. To overcome the problem and make the model more powerful, we first present sufficient conditions for RPE-based Transformers to achieve universal function approximation. With the theoretical guidance, we develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions. Therefore, the corresponding URPE-based Transformers become universal function approximators. Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications.

关键词：

An automatic analysis of ultrasound vocalisations for the prediction of interaction context in captive Egyptian fruit bats

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Triantafyllopoulos, Andreas Gebhard, Alexander Milling, Manuel Rampp, Simon Schuller, Björn Technical University of Munich MRI Munich Germany EIHW - Embedded Intelligence for Health Care and Wellbeing Augsburg Germany MCML - Munich Center for Machine Learning Munich Germany MDSI - Munich Data Science Institute Munich Germany GLAM - Group on Language Audio & Music Imperial College London United Kingdom

Prior work in computational bioacoustics has mostly focused on the detection of animal presence in a particular habitat. However, animal sounds contain much richer information than mere presence;among others, they encapsulate the interactions of those animals with other members of their species. Studying these interactions is almost impossible in a naturalistic setting, as the ground truth is often lacking. The use of animals in captivity instead offers a viable alternative pathway. However, most prior works follow a traditional, statistics-based approach to analysing interactions. In the present work, we go beyond this standard framework by attempting to predict the underlying context in interactions between captive Rousettus Aegyptiacus using deep neural networks. We reach an unweighted average recall of over 30% - more than thrice the chance level - and show error patterns that differ from our statistical analysis. This work thus represents an important step towards the automatic analysis of states in animals from sound. © 2024, CC BY-NC-ND.

关键词： Animals