检索结果-内蒙古大学图书馆

ENN: A Neural Network With DCT adaptive activation functions

IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING 2024年第2期18卷 232-241页

作者： Martinez-Gost, Marc Perez-Neira, Ana Lagunas, Miguel Angel Ctr Tecnol Telecomun Catalunya Castelldefels 08860 Spain Univ Politecn Cataluna Dept Signal Theory Commun Barcelona 08034 Spain ICREA Acad Barcelona 08010 Spain

The expressiveness of neural networks highly depends on the nature of the activation function, although these are usually assumed predefined and fixed during the training stage. Under a signal processing perspective, in this article we present Expressive Neural Network (ENN), a novel model in which the non-linear activation functions are modeled using the Discrete Cosine Transform (DCT) and adapted using backpropagation during training. This parametrization keeps the number of trainable parameters low, is appropriate for gradient-based schemes, and adapts to different learning tasks. This is the first non-linear model for activation functions that relies on a signal processing perspective, providing high flexibility and expressiveness to the network. We contribute with insights in the explainability of the network at convergence by recovering the concept of bump, this is, the response of each activation function in the output space. Finally, through exhaustive experiments we show that the model can adapt to classification and regression tasks. The performance of ENN outperforms state of the art benchmarks, providing above a 40% gap in accuracy in some scenarios.

关键词： Discrete cosine transforms Biological neural networks Task analysis Adaptation models Neurons Backpropagation Signal processing Neural networks adaptive activation functions discrete cosine transform explainable machine learning

来源：评论

学校读者我要写书评

暂无评论

Simple yet effective adaptive activation functions for physics-informed neural networks

引用

COMPUTER PHYSICS COMMUNICATIONS 2025年 307卷

作者： Zhang, Jun Ding, Chensen Peking Univ Coll Engn Dept Mech & Engn Sci Beijing 100871 Peoples R China Chongqing Univ Coll Aerosp Engn Dept Engn Mech Chongqing 400030 Peoples R China

Physics-informed neural networks (PINNs) gained widespread advancements in solving differential equations, where the performance tightly hinges on the choice of activation functions that are inefficient when selected manually. To tackle this issue, we propose two straightforward yet powerful adaptive activation functions: a weighted average function that adjusts activation functions by directly manipulating their weights, and a L2normalization function that compresses learnable parameters. These methods ensure a consistent sum of weights for each activation function, thereby enhancing optimization efficiency. We assess the performance of these approaches across a range of differential equation problems, encompassing Poisson equation, Wave equation, Burgers equation, Navier-Stokes equation, and linear/nonlinear solid mechanics problems. Through comparisons with exact solutions, we demonstrate significant improvements in convergence rate and solution accuracy. Our results underscore the efficacy of these techniques, providing a simple yet promising pathway for augmenting PINNs performance across diverse applications. The source codes and software implementation are available at https://***/jzhange/AAF-for-PINNs.

关键词： Physics-informed neural networks (PINNs) adaptive activation functions Weighted average function Partial differential equations L2-normalization function

来源：评论

学校读者我要写书评

暂无评论

Using deep convolutional neural networks with adaptive activation functions for medical CT brain image Classification 25

Using deep convolutional neural networks with adaptive activ...

引用

25th Iranian Conference on Biomedical Engineering / 3rd International Iranian Conference on Biomedical Engineering (ICBME)

作者： Zahedinasab, Roxana Mohseni, Hadis Shahid Bahonar Univ Kerman Comp Engn Dept Kerman Iran

ISBN: (纸本)9781538679524

recently, imaging has become an essential component in many fields of medical research. Analysis of the diverse medical image types requires sophisticated visualization and processing tools. Deep neural networks have introduced themselves as one of the most important branches of machine learning and have been successfully used in many fields of pattern recognition and medical imaging applications. Among the different networks, convolutional neural networks which are biologically inspired variants of multilayer perceptions are widely used in the medical imaging field. In these networks, activation function plays a significant role especially when the data come in different scales. There is a hope to improve the performance of these networks by using adaptive activation functions which adapts their parameters to the input data. In this paper, we have used a modified version of a successful convolutional neural network tuned for medical image classification and investigated the effect of applying three types of adaptive activation functions on that. These activation functions combine basic activation functions in linear (mixed) and nonlinear (gated and hierarchical) ways. The effectiveness of using these adaptive functions is shown on a CT brain images dataset (as a complex medical dataset). The experiments show that the classification accuracy of the proposed network with adaptive activation functions is higher compared to the ones using basic activation functions.

关键词： deep learning Convolutional neural networks adaptive activation functions CT brain images Image classification

来源：评论

学校读者我要写书评

暂无评论

Enhancement of CT brain images Classification based on deep learning network with adaptive activation functions 8

Enhancement of CT brain images Classification based on deep ...

引用

8th International Conference on Computer and Knowledge Engineering (ICCKE)

作者： Zahedinasab, Roxana Mohseni, Hadis Shahid Bahonar Univ Kerman Comp Engn Dept Kerman Iran

ISBN: (纸本)9781538695692

Deep neural networks are one of the most important branches of machine learning that have been recently used in many fields of pattern recognition and machine vision applications successfully. One of the most famous networks in this area is convolutional neural networks which are biologically inspired variants of multi-layer perceptions. In these networks, activation function plays a significant role especially when the data come in different scales. Recently, there is an interest to adaptive activation functions which adapts their parameters to the input data during network training process. Therefore, in this paper, inspired from a successful convolutional neural network tuned for medical image classification, we have investigated the effect of applying adaptive activation functions in a modified convolutional network by combining basic activation functions in linear (mixed) and nonlinear (gated) ways. The effectiveness of using these adaptive functions is shown on a CT brain images dataset (as a complex medical dataset) and the well-known MN 1ST hand-written digits dataset. The done experiments show that the classification accuracy of the proposed network with adaptive activation functions is higher compared to the ones using basic activation functions.

关键词： deep Learning Convolutional neural networks adaptive activation functions CT brain images

来源：评论

学校读者我要写书评

暂无评论

Learning Specialized activation functions for Physics-Informed Neural Networks

引用

Communications in Computational Physics 2023年第9期34卷 869-906页

作者： Honghui Wang Lu Lu Shiji Song Gao Huang Department of Automation Tsinghua UniversityBeijing 100084P.R.China Department of Statistics and Data Science Yale UniversityNew HavenCT 06511USA

Physics-informed neural networks(PINNs)are known to suffer from optimization *** this work,we reveal the connection between the optimization difficulty of PINNs and activation ***,we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct *** works usually choose activation functions by inefficient *** avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs,we introduce adaptive activation functions to search for the optimal function when solving different *** compare different adaptive activation functions and discuss their limitations in the context of ***,we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization,which has a higher requirement for the smoothness and diversity on learned *** is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at *** further enhance the search space with adaptive *** proposed adaptive activation function can be used to solve different PDE systems in an interpretable *** effectiveness is demonstrated on a series of *** is available at https://***/LeapLabTHU/AdaAFforPINNs.

关键词： Partial differential equations deep learning adaptive activation functions physicsinformed neural networks

来源：评论

学校读者我要写书评

暂无评论

Optimal Reusable Rocket Landing Guidance: A Cutting-Edge Approach Integrating Scientific Machine Learning and Enhanced Neural Networks

引用

IEEE ACCESS 2024年 12卷 16805-16829页

作者： Celik, Ugurcan Demirezen, Mustafa Umut Cranfield Univ Ctr Cyberphys & Autonomous Syst Bedford MK43 0AL England UDemy Inc Dept Data Prod San Francisco CA 94107 USA

This study presents an innovative approach that utilizes scientific machine learning and two types of enhanced neural networks for modeling a parametric guidance algorithm within the framework of ordinary differential equations to optimize the landing phase of reusable rockets. Our approach addresses various challenges, such as reducing prediction uncertainty, minimizing the need for extensive training data, improving convergence speed, decreasing computational complexity, and enhancing prediction accuracy for unseen data. We developed two distinct enhanced neural network architectures to achieve these objectives: adaptive (AQResNet) and Rowdy adaptive (RAQResNet) Quadratic Residual Neural Networks. These architectures exhibited outstanding performance in our simulations. Notably, the RAQResNet model achieved a validation loss approximately 300 times lower than the standard architecture with an equal number of trainable parameters and 50 times lower than the standard architecture with twice the number of trainable parameters. Furthermore, these models require significantly less computational power, enabling real-time computation on modern flight hardware. The inference times of our proposed models were measured in approximately microseconds on a single-board computer. Additionally, we conducted an extensive Monte Carlo analysis that considers a wide range of factors, extending beyond aerodynamic uncertainty, to assess the robustness of our models. The results demonstrate the impressive adaptability of our proposed guidance policy to new conditions and distributions outside the training domain. Overall, this study makes a substantial contribution to the field of reusable rocket landing guidance and establishes a foundation for future advancements.

关键词： adaptive activation functions guidance navigation optimal control scientific machine learning quadratic residual neural networks

来源：评论

学校读者我要写书评

暂无评论

Review of adaptive activation Function in Deep Neural Network

Review of Adaptive Activation Function in Deep Neural Networ...

引用

IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES)

作者： Lau, Mian Mian Lim, King Hann Curtin Univ Malaysia Curtin Sarawak Res Inst CDT 250 Miri Sarawak Malaysia

ISBN: (纸本)9781538624715

A biological inspired algorithm from human brain known as deep neural network (DNN) containing of multiple hidden layers often occurs vanishing gradient problem due to the saturation characteristic of activation function. Thus, the choice of activation function in DNN is crucial to boost up the DNN recognition performance. Unsaturated activation functions i.e. rectified linear unit is recently proposed to prevent vanishing gradient problem happened during the training process of DNN. In this paper, DNN performance is investigated with three categories of activation functions i.e. saturated, unsaturated and adaptive activation functions. The experimental results showed that the saturation problem of hyperbolic tangent activation function can be solved by adding two trainable parameters in its function. The trainable version of rectified linear unit i.e. parametric rectified linear unit (PReLU) obtained lowest misclassification rate among all types of activation function i.e. 1.6% misclassification rate on MNIST handwritten digit dataset. This is due to the adaptive activation functions allows the network to estimate a better solution by training the activation function parameters during the training process. Therefore, adaptive activation functions improves the generalization of the network to deal with the real-world application.

关键词： adaptive activation functions Saturated activation fucntions Unsaturated actuvation fucntions Deep nerual network

来源：评论

学校读者我要写书评

暂无评论

Structured Wavelet-based Neural Network for Control of Nonlinear Systems

Structured Wavelet-based Neural Network for Control of Nonli...

引用

50th IEEE Conference of Decision and Control (CDC)/European Control Conference (ECC)

作者： Karami, A. Yazdanpanah, M. J. Univ Tehran Sch Elect & Comp Engn Control & Intelligent Proc Ctr Excellence Tehran Iran

ISBN: (纸本)9781612848013

In this paper, a wavelet-based neural network is proposed for the control of nonlinear systems. activation functions of neural network nodes are determined based on the wavelet transform. The controller can efficiently compensate for the undesired effects of hard nonlinearities such as saturation and/or dead zone of control input. Compared with standard neuro-controllers, the structure of the controller is definite and simple. The proposed controller is localizable and has a systematically chosen structure, which improves the close-loop performance. An off-line algorithm determines the number of nodes. In addition, an on-line algorithm adjusts the parameters of wavelet bases and network weights. Back propagation algorithm with a momentum term is used for updating the weights and parameters of activation functions. This controller reduces the quantity of network parameters, calculation cost and convergence time of online algorithms with respect to the conventional neural network. Also, the controller is able to control unstable and MIMO systems. To illustrate the capability and performance superiority of the proposed controller, two nonlinear systems are controlled and the corresponding results are compared.

关键词： Wavelet transform neural network nonlinear system control adaptive activation functions

来源：评论

学校读者我要写书评

暂无评论

Structured Wavelet-based Neural Network for Control of Nonlinear Systems

Structured Wavelet-based Neural Network for Control of Nonli...

引用

IEEE Conference on Decision and Control

作者： A. Karami M.J. Yazdanpanah Control & Intelligent Processing Center of Excellence School of Electrical and Computer Engineering University of Tehran Tehran Iran

ISBN: (纸本)9781612848006

关键词： Wavelet transform neural network nonlinear system control adaptive activation functions

来源：评论

学校读者我要写书评

暂无评论

Regularising neural networks using flexible multivariate activation function

引用

NEURAL NETWORKS 2004年第2期17卷 247-260页

作者： Solazzi, M Uncini, A Univ Roma La Sapienza Dipartimento INFOCOM I-00184 Rome Italy Univ Ancona Dipartimento Elettron & Automat I-60131 Ancona Italy

This paper presents a new general neural structure based on nonlinear flexible multivariate function that can be viewed in the framework of the generalised regularisation net-works theory. The proposed architecture is based on multi-dimensional adaptive cubic spline basis activation function that collects information from the previous network layer in aggregate form. In other words, each activation function represents a spline function of a subset of previous layer outputs so the number of network connections (structural complexity) can be very low with respect to the problem complexity. A specific learning algorithm, based on the adaptation of local parameters of the activation function, is derived. This fact improve the network generalisation capabilities and speed up the convergence of the learning process. At last, some experimental results demonstrating the effectiveness of the proposed architecture, are presented. (C) 2003 Elsevier Ltd. All rights reserved.

关键词： neural networks spline neural networks multilayer perceptron generalised sigmoidal functions adaptive activation functions spline regularisation generalisation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：