Abstract
Abstract A regression model is fitted to an observed set of data. How accurate is the model for predicting future observations? The apparent error rate tends to underestimate the true error rate because the data have been used twice, both to fit the model and to check its accuracy. We provide simple estimates for the downward bias of the apparent error rate. The theory applies to general exponential family linear models and general measures of prediction error. Special attention is given to the case of logistic regression on binary data, with error rates measured by the proportion of misclassified cases. Several connected ideas are compared: Mallows's Cp , cross-validation, generalized cross-validation, the bootstrap, and Akaike's information criterion.
Keywords
Akaike information criterionStatisticsMathematicsLogistic regressionInformation CriteriaMean squared prediction errorRegressionWord error rateData setSet (abstract data type)Observational errorEconometricsModel selectionComputer scienceArtificial intelligence
Affiliated Institutions
Related Publications
Adaptive Model Selection
AbstractMost model selection procedures use a fixed penalty penalizing an increase in the size of a model. These nonadaptive selection procedures perform well only in one type o...
An optimal selection of regression variables
An asymptotically optimal selection of regression variables is proposed. The key assumption is that the number of control variables is infinite or increases with the sample size...
Smoothing Parameter Selection in Nonparametric Regression Using an Improved Akaike Information Criterion
Summary Many different methods have been proposed to construct nonparametric estimates of a smooth regression function, including local polynomial, (convolution) kernel and smoo...
Improvements on Cross-Validation: The 632+ Bootstrap Method
Abstract A training set of data has been used to construct a rule for predicting future responses. What is the error rate of this rule? This is an important question both for co...
Further analysis of the data by Akaike's information criterion and the finite corrections
Using Akaike's information criterion, three examples of statistical data are reanalyzed and show reasonably definite conclusions. One is concerned with the multiple comparison p...
Publication Info
- Year
- 1986
- Type
- article
- Volume
- 81
- Issue
- 394
- Pages
- 461-461
- Citations
- 419
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
419
OpenAlex
Cite This
Bradley Efron
(1986).
How Biased is the Apparent Error Rate of a Prediction Rule?.
Journal of the American Statistical Association
, 81
(394)
, 461-461.
https://doi.org/10.2307/2289236
Identifiers
- DOI
- 10.2307/2289236