Larry Wasserman
All of Nonparametric Statistics
With 52 Illustrations
Larry Wasserman
Department of Statistics
Carnegie Mellon University
Pittsburgh, PA 15213-3890
USA
Editorial Board
George Casella Stephen Fienberg Ingram Olkin
Department of Statistics Department of Statistics Department of Statistics
University of Florida Carnegie Mellon University Stanford University
Gainesville, FL 32611-8545 Pittsburgh, PA 15213-3890 Stanford, CA 94305
USA USA USA
Library of Congress Control Number: 2005925603
ISBN-10: 0-387-25145-6
ISBN-13: 978-0387-25145-5
Printed on acid-free paper.
© 2006 Springer Science+Business Media, Inc.
All rights reserved. This work may not be translated or copied in whole or in part without the written
permission of the publisher (Springer Science+Business Media, Inc., 233 Spring Street, New
York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis.
Use in connection with any form of information storage and retrieval, electronic adaptation, computer
software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they
are not identified as such, is not to be taken as an expression of opinion as to whether or not they
are subject to proprietary rights.
Printed in the United States of America. (MVY)
9 8 7 6 5 4 3 2 1
springeronline.com
1 Introduction 1
1.1 What Is Nonparametric Inference? . . . . . . . . . . . . . . . . 1
1.2 Notation and Background . . . . . . . . . . . . . . . . . . . . . 2
1.3 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Useful Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Estimating the cdf and
Statistical Functionals 13
2.1 The cdf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Estimating Statistical Functionals . . . . . . . . . . . . . . . . 15
2.3 Influence Functions . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Empirical Probability Distributions . . . . . . . . . . . . . . . . 21
2.5 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 23
2.6 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3 The Bootstrap and the Jackknife 27
3.1 The Jackknife . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 The Bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 ParametricBootstrap . . . . . . . . . . . . . . . . . . . . . . . 31
3.4 Bootstrap Confidence Intervals . . . . . . . . . . . . . . . . . . 32
3.5 Some Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
x Contents
3.6 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 37
3.7 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4 Smoothing: General Concepts 43
4.1 The Bias–VarianceTradeoff . . . . . . . . . . . . . . . . . . . . 50
4.2 Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3 Which Loss Function? . . . . . . . . . . . . . . . . . . . . . . . 57
4.4 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 The Curse of Dimensionality . . . . . . . . . . . . . . . . . . . 58
4.6 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 59
4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5 Nonparametric Regression 61
5.1 Review of Linear and Logistic Regression . . . . . . . . . . . . 63
5.2 Linear Smoothers . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3 Choosing the Smoothing Parameter . . . . . . . . . . . . . . . 68
5.4 Local Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.5 Penalized Regression, Regularization and Splines . . . . . . . . 81
5.6 Variance Estimation . . . . . . . . . . . . . . . . . . . . . . . . 85
5.7 Confidence Bands . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.8 Average Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.9 Summary of Linear Smoothing . . . . . . . . . . . . . . . . . . 95
5.10 Local Likelihood and Exponential Families . . . . . . . . . . . . 96
5.11 Scale-Space Smoothing . . . . . . . . . . . . . . . . . . . . . . . 99
5.12 Multiple Regression . . . . . . . . . . . . . . . . . . . . . . . . 100
5.13 Other Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.14 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 119
5.15 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6 Density Estimation 125
6.1 Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.2 Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.3 Kernel Density Estimation . . . . . . . . . . . . . . . . . . . . . 131
6.4 Local Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.5 Multivariate Problems . . . . . . . . . . . . . . . . . . . . . . . 138
6.6 Converting Density Estimation Into Regression . . . . . . . . . 139
6.7 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 140
6.8 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7 Normal Means and Minimax Theory 145
7.1 The NormalMeansModel . . . . . . . . . . . . . . . . . . . . . 145
7.2 Function Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Contents xi
7.3 Connection to Regression and Density Estimation . . . . . . . 149
7.4 Stein’s Unbiased Risk Estimator (sure) . . . . . . . . . . . . . 150
7.5 Minimax Risk and Pinsker’s Theorem . . . . . . . . . . . . . . 153
7.6 Linear Shrinkage and the James–Stein Estimator . . . . . . . . 155
7.7 Adaptive Estimation Over Sobolev Spaces . . . . . . . . . . . . 158
7.8 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.9 Optimality of Confidence Sets . . . . . . . . . . . . . . . . . . . 166
7.10 RandomRadius Bands? . . . . . . . . . . . . . . . . . . . . . . 170
7.11 Penalization,Oracles and Sparsity . . . . . . . . . . . . . . . . 171
7.12 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 172
7.13 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.14 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
8 Nonparametric Inference Using Orthogonal Functions 183
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
8.2 Nonparametric Regression . . . . . . . . . . . . . . . . . . . . . 183
8.3 IrregularDesigns . . . . . . . . . . . . . . . . . . . . . . . . . . 190
8.4 Density Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 192
8.5 Comparison ofMethods . . . . . . . . . . . . . . . . . . . . . . 193
8.6 Tensor ProductModels . . . . . . . . . . . . . . . . . . . . . . 193
8.7 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 194
8.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
9 Wavelets and Other Adaptive Methods 197
9.1 HaarWavelets . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.2 ConstructingWavelets . . . . . . . . . . . . . . . . . . . . . . . 203
9.3 Wavelet Regression . . . . . . . . . . . . . . . . . . . . . . . . . 206
9.4 Wavelet Thresholding . . . . . . . . . . . . . . . . . . . . . . . 208
9.5 Besov Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.6 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
9.7 Boundary Corrections and Unequally Spaced Data . . . . . . . 215
9.8 OvercompleteDictionaries . . . . . . . . . . . . . . . . . . . . . 215
9.9 Other AdaptiveMethods . . . . . . . . . . . . . . . . . . . . . 216
9.10 Do AdaptiveMethodsWork? . . . . . . . . . . . . . . . . . . . 220
9.11 Bibliographic Remarks . . . . . . . . . . . . . . . . . . . . . . . 221
9.12 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
10 Other Topics 227
10.1 Measurement Error . . . . . . . . . . . . . . . . . . . . . . . . . 227
10.2 Inverse Problems . . . . . . . . . . . . . . . . . . . . . . . . . . 233
10.3 Nonparametric Bayes . . . . . . . . . . . . . . . . . . . . . . . . 235
10.4 Semiparametric Inference . . . . . . . . . . . . . . . . . . . . . 235
10.5 Correlated Errors . . . . . . . . . . . . . . . . . . . . . . . . . . 236
10.6 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
xii Contents
10.7 Sieves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
10.8 Shape-Restricted Inference . . . . . . . . . . . . . . . . . . . . . 237
10.9 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
10.10Computational Issues . . . . . . . . . . . . . . . . . . . . . . . 240
10.11Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Bibliography 243
List of Symbols 259
Table of Distributions 261
Index 263