Suboptimality of penalties proportional to the dimension for model selection in heteroscedastic regression

Sylvain Arlot (1, 2)

1 LIENS - Laboratoire d'informatique de l'école normale supérieure
2 WILLOW - Models of visual object recognition and scene understanding

We consider the problem of choosing between several models in least-squares regression with heteroscedastic data. We prove that any penalization procedure is suboptimal when the penalty is proportional to the dimension of the model, at least for some typical heteroscedastic model selection problems. In particular, Mallows' $C_p$ is suboptimal in this framework, as well as any ``linear'' penalty depending on both the data and their true distribution. On the contrary, optimal model selection is possible in this framework with data-driven penalties such as $V$-fold or resampling penalties (Arlot, 2008a,b). Therefore, estimating the ``shape'' of the penalty from the data is useful, even at the price of a higher computational cost.

Mots clés: non-parametric regression; model selection; penalization; heteroscedastic data; Mallows Cp; resampling penalties

Domaines: Statistiques [math.ST]; Théorie [stat.TH]

Sylvain Arlot. Suboptimality of penalties proportional to the dimension for model selection in heteroscedastic regression. 2008. ⟨hal-00347811v1⟩