This paper presents a constructive approach to estimating the size of a neural network necessary to solve a given classification problem. The results are derived using an information entropy approach in the context of...
详细信息
This paper presents a constructive approach to estimating the size of a neural network necessary to solve a given classification problem. The results are derived using an information entropy approach in the context of limited precision integer weights. Such weights are particularly suited for hardware implementations since the area they occupy is limited, and the computations performed with them can be efficiently implemented in hardware. The considerations presented use an information entropy perspective and calculate lower bounds on the number of bits needed in order to solve a given classification problem. These bounds are obtained by approximating the classification hypervolumes with the volumes of several regular (i.e., highly symmetric) n-dimensional bodies. The bounds given here allow the user to choose the appropriate size of a neural network such that: (i) the given classification problem can be solved, and (ii) the network architecture is not oversized. All considerations presented take into account the restrictive case of limited precision integer weights, and therefore can be directly applied when designing VLSI implementations of neural networks.
constructive algorithms have proved to be powerful methods for training feedforward neural networks. An important property of these algorithms is generalization. A series of empirical studies were performed to examine...
详细信息
constructive algorithms have proved to be powerful methods for training feedforward neural networks. An important property of these algorithms is generalization. A series of empirical studies were performed to examine the effect of regularization on generalization in constructive cascade algorithms. It was found that the combination of early stopping and regularization resulted in better generalization than the use of early stopping alone. A cubic penalty term that greatly penalizes large weights was shown to be beneficial for generalization in cascade networks. An adaptive method of setting the regularization magnitude in constructive algorithms was introduced and shown to produce generalization results similar to those obtained with a fixed, user-optimized regularization setting. This adaptive method also resulted in the construction of smaller networks for more complex problems. The acasper algorithm, which incorporates the insights obtained from the empirical studies, was shown to have good generalization and network construction properties. This algorithm was compared to the cascade correlation algorithm on the Proben 1 and additional regression data sets.
A study of the function approximation capabilities of single hidden layer neural networks strongly motivates the investigation of constructive learning techniques as a means of realizing established error bounds. Lear...
详细信息
A study of the function approximation capabilities of single hidden layer neural networks strongly motivates the investigation of constructive learning techniques as a means of realizing established error bounds. Learning characteristics employed by constructive algorithms provide ideas for development of new algorithms applicable to the function approximation problem. In addition, constructive techniques offer efficient methods for network construction and weight determination. The development of a novel neural network algorithm, the constructive Locally Fit Sigmoids (CLFS) function approximation algorithm, is presented in detail. Basis functions of global extent (piecewise linear sigmoidal functions) are locally fit to the target function, resulting in a pool of candidate hidden layer nodes from which a function approximation is obtained. This algorithm provides a methodology of selecting nodes in a meaningful way from the infinite set of possibilities and synthesizes an n node single hidden layer network with empirical and analytical results that strongly indicate an O(1/n) mean squared training error bound under certain assumptions. The algorithm operates in polynomial time in the number of network nodes and the input dimension. Empirical results demonstrate its effectiveness on several multidimensional function approximate problems relative to contemporary constructive and nonconstructive algorithms.
In this paper, we study a number of objective functions for training new hidden units in constructive algorithms for multilayer feedforward networks. The aim is to derive a class of objective functions the computation...
详细信息
In this paper, we study a number of objective functions for training new hidden units in constructive algorithms for multilayer feedforward networks. The aim is to derive a class of objective functions the computation of which and the corresponding weight updates can be done in O(N) time, where N is the number of training patterns. Moreover, even though. input weight freezing is applied during the process for computational efficiency, the convergence property of the constructive algorithms using these objective functions is still preserved. We also propose a few computational tricks that can be used to improve the optimization of the objective functions under practical situations. Their relative performance in a set of two-dimensional regression problems is also discussed.
Among binary unit-based constructive algorithms, the Sequential Learning is particularly interesting for many reasons, the most significant one being its ability to treat real valued inputs without preprocessing. Howe...
详细信息
Among binary unit-based constructive algorithms, the Sequential Learning is particularly interesting for many reasons, the most significant one being its ability to treat real valued inputs without preprocessing. However, due to the construction process, the classical algorithms derived from the Perceptron cannot be used for learning each unit of the hidden layer. The BCP Max, recently proposed, appears as the first efficient heuristic algorithm to perform the particular neuron training in the Sequential Learning. But the BCP Max principles can be easily extended to the classical Perceptron derivatives usually used in constructive algorithms. In this paper, we show how to extend the Thermal Perceptron, the Pocket algorithm, the Ratchet and the simple Perceptron to train a neuron in the Sequential Learning. Finally all solutions are compared.
This paper investigates the possibility of improving the classification capability of single-layer and multilayer perceptrons by incorporating additional output layers. This Multi-Output-Layer Perceptron (MOLP) is a n...
详细信息
This paper investigates the possibility of improving the classification capability of single-layer and multilayer perceptrons by incorporating additional output layers. This Multi-Output-Layer Perceptron (MOLP) is a new type of constructive network, though the emphasis is on improving pattern separability rather than network efficiency. The MOLP is trained using the standard back-propagation (BP) algorithm. The studies are concentrated on realizations of arbitrary functions which map from an x-dimensional input vector into a y-dimensional output vector. With the MOLP, all problems existing in an original n-dimensional space in the hidden layer are transformed to a higher (n + 1)-dimensional space, so that the possibility of linear separability is increased. Experimental investigations show that the classification ability of the MOLP is superior to that of an equivalent MLP. In general, this performance increase can be achieved with shorter training times and simpler network architectures.
Neural networks are being tested to monitor aircraft engine condition data, in addition to current techniques and newer methods such as knowledge-based systems or case-based reasoning, in order to increase safety and ...
详细信息
暂无评论