Abstract
According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g : R --> R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g : R --> R and integral of R g(x)dx not equal to 0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.
Keywords
Activation functionFeedforward neural networkPiecewiseConstructiveArtificial neural networkFeed forwardRadial basis functionComputer scienceFunction (biology)MathematicsAlgorithmArtificial intelligenceMathematical optimizationProcess (computing)
Affiliated Institutions
Related Publications
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators
It is widely known that neural networks (NNs) are universal approximators of continuous functions. However, a less known but powerful result is that a NN with a single hidden la...
Regularization Theory and Neural Networks Architectures
We had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units, called regularization networ...
Constructive algorithms for structure learning in feedforward neural networks for regression problems
In this survey paper, we review the constructive algorithms for structure learning in feedforward neural networks for regression problems. The basic idea is to start with a smal...
Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level hu...
Training Recurrent Networks by Evolino
In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for...
Publication Info
- Year
- 2006
- Type
- article
- Volume
- 17
- Issue
- 4
- Pages
- 879-892
- Citations
- 2586
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
2586
OpenAlex
Cite This
Guang-Bin Huang,
Lei Chen,
Chee‐Kheong Siew
(2006).
Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes.
IEEE Transactions on Neural Networks
, 17
(4)
, 879-892.
https://doi.org/10.1109/tnn.2006.875977
Identifiers
- DOI
- 10.1109/tnn.2006.875977