Table of Contents
Preface to first edition ix
Preface to second edition xiii
1 Objectives
1.1 Prediction, explanation, elimination or what? 1
1.2 How many variables in the prediction formula? 3
1.3 Alternatives to using subsets 6
1.4 'Black box' use of best-subsets techniques 8
2 Least-squares computations
2.1 Using sums of squares and products matrices 11
2.2 Orthogonal reduction methods 16
2.3 Gauss-Jordan v. orthogonal reduction methods 20
2.4 Interpretation of projections 27
Appendix A Operation counts for all-subsets regression 29
A.1 Garside's Gauss-Jordan algorithm 30
A.2 Planar rotations and a Hamiltonian cycle 31
A.3 Planar rotations and a binary sequence 32
A.4 Fast planar rotations 34
3 Finding subsets which fit well
3.1 Objectives and limitations of this chapter 37
3.2 Forward selection 39
3.3 Efroymson's algorithm 42
3.4 Backward elimination 44
3.5 Sequential replacement algorithms 46
3.6 Replacing two variables at a time 48
3.7 Generating all subsets 48
3.8 Using branch-and-bound techniques 52
3.9 Grouping variables 54
3.10 Ridge regression and other alternatives 57
3.11 The nonnegative garrote and the lasso 60
3.12 Some examples 67
3.13 Conclusions and recommendations 84
Appendix A An algorithm for the lasso 86
4 Hypothesis testing
4.1 Is there any information in the remaining variables? 89
4.2 Is one subset better than another? 97
4.2.1 Applications of Spjøtvoll's method 101
4.2.2 Using other confidence ellipsoids 104
Appendix A Spjøtvoll's method - detailed description 106
5 When to stop?
5.1 What criterion should we use? 111
5.2 Prediction criteria 112
5.2.1 Mean squared errors of prediction (MSEP) 113
5.2.2 MSEP for the fixed model 114
5.2.3 MSEP for the random model 129
5.2.4 A simulation with random predictors 133
5.3 Cross-validation and the PRESS statistic 143
5.4 Bootstrapping 151
5.5 Likelihood and information-based stopping rules 154
5.5.1 Minimum description length (MDL) 158
Appendix A Approximate equivalence of stopping rules 160
A.1 F-to-enter 160
A.2 Adjusted R2 or Fisher's A-statistic 161
A.3 Akaike's information criterion (AIC) 162
6 Estimation of regression coefficients
6.1 Selection bias 165
6.2 Choice between two variables 166
6.3 Selection bias in the general case and its reduction 175
6.3.1 Monte Carlo estimation of bias in forward selection 178
6.3.2 Shrinkage methods 182
6.3.3 Using the jack-knife 185
6.3.4 Independent data sets 187
6.4 Conditional likelihood estimation 188
6.5 Estimation of population means 191
6.6 Estimating least-squares projections 195
Appendix A Changing projections to equate sums of squares 197
7 Bayesian methods
7.1 Bayesian introduction 201
7.2 'Spike and slab' prior 203
7.3 Normal prior for regression coefficients 206
7.4 Model averaging 211
7.5 Picking the best model 215
8 Conclusions and some recommendations 217
References 223
Index 235
