Abstract
We derive new bounds for the generalization error of kernel machines, such as support vector machines and related regularization networks by obtaining new bounds on their covering numbers. The proofs make use of a viewpoint that is apparently novel in the field of statistical learning theory. The hypothesis class is described in terms of a linear operator mapping from a possibly infinite-dimensional unit ball in feature space into a finite-dimensional space. The covering numbers of the class are then determined via the entropy numbers of the operator. These numbers, which characterize the degree of compactness of the operator, can be bounded in terms of the eigenvalues of an integral operator induced by the kernel function used by the machine. As a consequence, we are able to theoretically explain the effect of the choice of kernel function on the generalization performance of support vector machines.
Keywords
MathematicsEntropy (arrow of time)Eigenvalues and eigenvectorsStatistical learning theoryBounded functionSupport vector machinePositive-definite kernelOperator (biology)Discrete mathematicsComputer scienceArtificial intelligenceOperator theoryMathematical analysisFourier integral operator
Affiliated Institutions
Related Publications
Advances in kernel methods: support vector learning
Introduction to support vector learning roadmap. Part 1 Theory: three remarks on the support vector method of function estimation, Vladimir Vapnik generalization performance of ...
On the generalization of soft margin algorithms
Generalization bounds depending on the margin of a classifier are a relatively new development. They provide an explanation of the performance of state-of-the-art learning syste...
Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel
Support vector machines (SVMs) with the gaussian (RBF) kernel have been popular for practical use. Model selection in this class of SVMs involves two hyper parameters: the penal...
Nonlinear Component Analysis as a Kernel Eigenvalue Problem
A new method for performing a nonlinear form of principal component analysis is proposed. By the use of integral operator kernel functions, one can efficiently compute principal...
Domain Adaptation: Learning Bounds and Algorithms
This paper addresses the general problem of domain adaptation which arises in a variety of applications where the distribution of the labeled sample available somewhat differs f...
Publication Info
- Year
- 2001
- Type
- article
- Volume
- 47
- Issue
- 6
- Pages
- 2516-2532
- Citations
- 169
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
169
OpenAlex
Cite This
Robert C. Williamson,
Alex Smola,
Bernhard Schölkopf
(2001).
Generalization performance of regularization networks and support vector machines via entropy numbers of compact operators.
IEEE Transactions on Information Theory
, 47
(6)
, 2516-2532.
https://doi.org/10.1109/18.945262
Identifiers
- DOI
- 10.1109/18.945262