检索结果-内蒙古大学图书馆

Generating a large family of nonlinear activation functions (LFNAFs) in neural networks

JOURNAL OF SUPERCOMPUTING 2025年第5期81卷 1-31页

作者： Taheri, Morteza Klidbary, Sajad Haghzad Univ Zanjan Fac Engn Dept Comp Engn Zanjan Iran Univ Zanjan Fac Engn Dept Comp & Elect Engn Zanjan Iran

Choosing the optimal activation functions (AFs) for the training of deep neural networks (DNNs) has always posed a significant challenge due to its substantial impact on the network performance and training speed. Despite the impressive performance demonstrated by various nonlinear activation functions, such as rectified linear units, hyperbolic tangent, Sigmoid, Swish, Mish, Smish, and Logish, only a restricted subset of these functions is commonly embraced in most applications, primarily because of inherent inconsistencies or other limitations. This research introduces a novel framework for generating a large family of nonlinear activation functions (LFNAFs) using propositions. These propositions define sufficient conditions for creating continuous and differentiable activation functions with desirable properties, including nonlinearity, zero-centeredness, resilience against vanishing gradients, prevention of gradient saturation, non-monotonicity, smoothness and infinite range. In addition, many well-known activation functions are particular cases of this family, such as ReLU, Logish, Swish and so on. These findings indicate that the activation functions we developed are better suited for simple and intricate deep learning models. The results show that the proposed activation functions achieve significantly better accuracy compared to other well-known activation functions across various datasets, including MNIST, CIFAR-10, and CIFAR-100. Many AFs can be generated, but a few instances of them have been discussed in this paper. For instance, more simulations have been done on H(2 )and H-2+Sigmoid (AFs). Performance of H-2 is obviously better than other AFs, but H-2+Sigmoid is better and in some cases is equal than other AFs. Notably, while the proposed activation functions achieved 100% training accuracy on the MNIST dataset, it is important to acknowledge that achieving such high accuracy may vary based on the complexity of the dataset and specific conditions of th

关键词： activation functions Large family of nonlinear activation functions Deep neural networks MNIST CIFAR-10 CIFAR-100 datasets

来源：评论

学校读者我要写书评

暂无评论

Oscillating activation functions can improve the performance of convolutional neural networks

引用

APPLIED SOFT COMPUTING 2025年 175卷

作者： Noel, Mathew Mithra Arunkumar, L. Trivedi, Advait Dutta, Praneet Vellore Inst Technol Vellore India Red Hat Inc Raleigh NC USA Vellore Inst Technol Sch Elect Engn Vellore VIT Vellore India

Convolutional neural networks have been successful in solving many socially important and economically significant problems. Their ability to learn complex high-dimensional functions hierarchically can be attributed to the use of nonlinear activation functions. A key discovery that made training deep networks feasible was the adoption of the Rectified Linear Unit (ReLU) activation function to alleviate the vanishing gradient problem caused by using saturating activation functions. Since then, many improved variants of the ReLU activation have been proposed. However, a majority of activation functions used today are non-oscillatory and monotonically increasing due to their biological plausibility. This paper demonstrates that oscillatory activation functions can improve gradient flow and reduce network size. Two theorems on limits of non-oscillatory activation functions are presented. A new oscillatory activation function called Growing Cosine Unit(GCU) defined as C(z) = z & sdot;cos z that outperforms Sigmoids, Swish, Mish and ReLU on a variety of architectures and benchmarks is presented. The GCU activation has multiple zeros enabling single GCU neurons to have multiple hyperplanes in the decision boundary. This allows single GCU neurons to learn the XOR function without feature engineering. Extensive experimental comparison with 16 popular activation functions indicate that the GCU activation function significantly improves performance on CIFAR-10, CIFAR-100, Imagenette and the 1000 class ImageNet benchmarks.

关键词： activation functions XOR problem Convolutional neural network Deep learning

来源：评论

学校读者我要写书评

暂无评论

Classification of acute myeloid leukemia by pre-trained deep neural networks: A comparison with different activation functions

引用

MEDICAL ENGINEERING & PHYSICS 2025年 135卷 104277-104277页

作者： Aby, Aswathy Elma Salaji, S. Anilkumar, K. K. Rajan, Tintu Cochin Univ Sci & Technol Cochin Univ Coll Engn Kuttanad Dept Elect & Commun Kochi 688504 Kerala India Cochin Univ Sci &Technol Sch Engn Dept Mech Engn Kochi 682022 Kerala India Vijaya Diagnost Ctr Kottarakara 691531 Kerala India

Acute Myeloid Leukemia(AML) is a rapidly progressing cancer affecting blood and bone marrow, marked by the swift proliferation of abnormal myeloid cells. Effective treatment requires precise classification of AML subtypes. Conventional classification methods rely on manual microscopic analysis, which is time-consuming and variable, while traditional machine learning approaches often struggle with feature extraction and generalization. This underscores the need for automated detection methods which enhance this process by minimizing manual effort. activation functions are critical in neural networks, introducing non-linearity that influences training convergence, computational efficiency, and model performance. This study evaluates the effectiveness of CNN architectures, Sequential (VGG16), Directed Acyclic Graph (InceptionV3), and Residual (ResNet50v2) in distinguishing between AML subtypes: AML without maturation, Acute Monoblastic Leukemia, and Pure Erythroid Leukemia, using peripheral blood smear images, while also investigating the impact of different activation functions on model accuracy and training time. The results show that ResNet50v2 achieves the shortest training time, while InceptionV3 takes the longest due to its complex architecture. GELU delivers the highest accuracy, reaching 94.02 % in InceptionV3 and 92.53 % in ResNet50v2, while SELU achieves the highest accuracy for VGG16 at 92.83 %. Mish provides competitive accuracy with lower training time than GELU, while Softplus and Softsign consistently perform poorly. This research demonstrates the potential of CNNs for automating AML subtype classification and identifies GELU as the most effective activation function. Future work could explore data augmentation, optimized activation functions, and attention mechanisms to improve classification performance.

关键词： Acute myeloid leukemia Peripheral blood smear images WHO classification Deep learning Convolutional neural network activation functions

来源：评论

学校读者我要写书评

暂无评论

Determination of the performance of training algorithms and activation functions in meteorological drought index prediction with nonlinear autoregressive neural network

引用

EARTH SCIENCE INFORMATICS 2025年第2期18卷 1-20页

作者： Gumus, Munevver Gizem Ciftci, Hasan cagatay Gumus, Kutalmis Erciyes Univ Fac Engn Dept Geomat Engn Kayseri Turkiye Nigde Omer Halisdemir Univ Fac Engn Dept Geomat Engn Nigde Turkiye

Analysis of long-term meteorological data is critical for monitoring climate trends and understanding the drought situation in a given region. In this study, monthly average precipitation data from the Ni & gbreve;de meteorological station in Turkey covering the period 1950-2020 were used. Within the scope of the study, seven different drought index methods were used for drought analysis, and the number and percentages of drought conditions were calculated according to these indices. For example, according to the Standardized Precipitation Index (SPI) method, the proportion of dry periods was determined as 16.2% and the proportion of humid periods as 83.8%. The Mann-Kendall trend analysis performed to determine the drought trends of the region revealed an increasing trend towards humidity in all indices (e.g., z = 1.299, p = 0.194 for SPI). In the study, 60-month drought forecasts covering the years 2020-2025 were realized using the Nonlinear Autoregressive Neural Network (NARNN) model, and the results were compared with the Autoregressive (AR) model. In the prediction performance analysis, the NARNN model showed superior prediction performance for all indices with lower RMSE values (e.g., NARNN RMSE = 0.977 for SPI;AR RMSE = 1.704). The prediction performances of different training algorithms and activation functions used in the NARNN model were analyzed. The best performance was obtained with the trainbr training algorithm and sigmoid activation function (e.g., RMSE = 0.997 for SPI). Based on these best parameters, more than 70% of the drought conditions during the 2020-2025 period were found to be normal or humid according to NARNN predictions. This study demonstrates the superiority of the NARNN model in nonlinear time series analyses and that it is a reliable tool, especially for future drought forecasts. In addition, comprehensive analyses with different index methods have significantly contributed to understanding the long-term drought trends in the Ni & gbre

关键词： NARNN Train algorithms activation functions Meteorological drought index

来源：评论

学校读者我要写书评

暂无评论

activation functions enabling the addition of neurons and layers without altering outcomes

引用

Journal of Computational and Applied Mathematics 2025年 470卷

作者： López-Ureña, Sergio Dept. de Matemàtiques Universitat de València Doctor Moliner Street 50 València Burjassot46100 Spain

In this work, we propose activation functions for neuronal networks that are refinable and sum the identity. This new class of activation functions allows the insertion of new layers between existing ones and/or the increase of neurons in a layer, both without altering the network outputs. Our approach is grounded in subdivision theory. The proposed activation functions are constructed from basic limit functions of convergent subdivision schemes. As a showcase of our results, we introduce a family of spline activation functions and provide comprehensive details for their practical implementation. © 2025 The Author(s)

关键词： Neural networks activation functions Refinable functions Subdivision schemes

来源：评论

学校读者我要写书评

暂无评论

Performance analysis of activation functions in molecular property prediction using Message Passing Graph Neural Networks

引用

CHEMICAL PHYSICS 2025年 591卷

作者： Chanana, Garima Vivekananda Inst Profess Studies Sch Engn & Technol Tech Campus Delhi 110034 India

Deep learning has significantly advanced molecular property prediction, with Message-Passing Graph Neural Networks (MPGNN) standing out as an effective method. This study systematically evaluates the performance often activation functions - Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, SELU, Softmax, Swish, Mish, and GeLU - using the MPGNN model on the QM9 dataset. It aims to identify the most suitable activation functions for training neural networks for specific molecular properties. The study examines electronic properties such as HOMO, LUMO, HOMO-LUMO energy gap, dipole moment, and polarizability, as well as thermal properties like zero-point vibrational energy and specific heat capacity. The findings reveal that different activation functions excel for different properties: SELU for HOMO, ELU for LUMO, Sigmoid for the HOMO-LUMO gap, Mish for polarizability, GELU for ZPVE, and Leaky ReLU for dipole moment and specific heat capacity. These insights are vital for optimizing MPGNN design for targeted molecular property prediction.

关键词： Message-Passing Graph Neural Network (MPGNN) Molecular property prediction activation functions QM9 dataset

来源：评论

学校读者我要写书评

暂无评论

activation functions for Convolutional Neural Networks: Proposals and Experimental Study

引用

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023年第3期34卷 1478-1488页

作者： Vargas, Victor Manuel Gutierrez, Pedro Antonio Barbero-Gomez, Javier Hervas-Martinez, Cesar Univ Cordoba Dept Comp Sci & Numer Anal Cordoba 14014 Spain

activation functions lie at the core of every neural network model from shallow to deep convolutional neural networks. Their properties and characteristics shape the output range of each layer and, thus, their capabilities. Modern approaches rely mostly on a single function choice for the whole network, usually ReLU or other similar alternatives. In this work, we propose two new activation functions and analyze their properties and compare them with 17 different function proposals from recent literature on six distinct problems with different characteristics. The objective is to shed some light on their comparative performance. The results show that the proposed functions achieved better performance than the most commonly used ones.

关键词： Feature extraction Training Neurons Proposals Neural networks Dispersion Convolution activation functions convolutional networks ELUs+2 exponential linear unit (ELU) s+2

来源：评论

学校读者我要写书评

暂无评论

activation functions in deep learning: A comprehensive survey and benchmark

引用

NEUROCOMPUTING 2022年 503卷 92-108页

作者： Dubey, Shiv Ram Singh, Satish Kumar Chaudhuri, Bidyut Baran Indian Inst Informat Technol Comp Vis & Biometr Lab Allahabad India Techno India Univ Kolkata India Indian Stat Inst Kolkata India Indian Inst Informat Technol Allahabad Allahabad India

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and non-linear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and sur-vey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance compar-ison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: https://github.-com/shivram1987/activationfunctions.(c) 2022 Elsevier B.V. All rights reserved.

关键词： activation functions Neural networks Convolutional neural networks Deep learning Overview Recurrent Neural Networks

来源：评论

学校读者我要写书评

暂无评论

Physics-informed neural networks with trainable sinusoidal activation functions for approximating the solutions of the Navier-Stokes equations

引用

Computer Physics Communications 2025年 314卷

作者： Amirhossein Khademi Steven Dufour MAGI Department of Mathematics and Industrial Engineering Polytechnique Montreal 2500 Chemin de Polytechnique H3T 1J4 Montreal Quebec Canada

We present TSA-PINN, a novel Physics-Informed Neural Network (PINN) that leverages a Trainable Sinusoidal activation (TSA) mechanism to approximate solutions to the Navier-Stokes equations. By incorporating neuron-wise sinusoidal activation functions with trainable frequencies and a dynamic slope recovery mechanism, TSA-PINN achieves superior accuracy and convergence. Its ability to dynamically adjust activation frequencies enables efficient modeling of complex fluid behaviors, reducing training time and computational cost. Our testing goes beyond canonical problems, to study less-explored and more challenging scenarios, which have typically posed difficulties for prior models. Various numerical tests underscore the efficacy of the TSA-PINN model across five different scenarios. These include steady-state two-dimensional flows in a lid-driven cavity at two different Reynolds numbers; a cylinder wake problem characterized by oscillatory fluid behavior; and two time-dependent three-dimensional turbulent flow cases. In the turbulent cases, the focus is on detailed near-wall phenomena—including the viscous sub-layer, buffer layer, and log-law region—as well as the complex interactions among eddies of various scales. Both numerical and quantitative analyses demonstrate that TSA-PINN offers substantial improvements over conventional PINN models. This research advances physics-informed machine learning, setting a new benchmark for modeling dynamic systems in scientific computing and engineering.

关键词： Deep learning activation functions Physics-informed neural networks Partial differential equations Navier-Stokes equations Machine learning

来源：评论

学校读者我要写书评

暂无评论

Revisiting activation functions: empirical evaluation for image understanding and classification

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2024年第6期83卷 18497-18536页

作者： Verma, Shradha Chug, Anuradha Singh, Amit Prakash Guru Gobind Singh Indraprastha Univ GGSIPU Univ Sch Informat Commun & Technol USIC&T New Delhi India

In this paper, the authors have devised four novel activation functions by coupling and combining a few existing functions implemented with four standard CNN architectures namely VGG19, ResNet50, InceptionV3, and DenseNet121 on existing benchmark datasets CIFAR10, CIFAR100, and MNIST. The best-performing function is also implemented with EfficientNet (B0-B7) models, on the abovementioned datasets, with improved performance as compared to the Swish function. Apart from classification accuracy, measures such as top5 accuracy, Cohen's kappa, precision, recall, f1-score, RMSE, MSE, and loss have been recorded. The authors have also implemented Monte-Carlo Dropout to evaluate the uncertainty of the implementations in terms of Brier Score, mean, variance, standard deviation, and peak accuracy values. Lastly, a few instances of classification with the Tiny-ImageNet dataset, semantic segmentation, object detection, and hyperspectral image classification have been implemented for evaluation and comparison purposes. The research work presented in this paper aims to experiment with combinations of different activation functions, analyze the performance and find a better version that improves the performance of DNNs for the task of image understanding and classification.

关键词： activation functions Deep Learning Convolutional Neural Networks (CNN) Image Classification Uncertainty Semantic Segmentation Object Detection Hyperspectral Image Classification

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：