Bias/Variance is not the same as Approximation/Estimation

Gavin Brown, Riccardo Ali

Research output: Contribution to journalArticlepeer-review

21 Downloads (Pure)

Abstract

We study the relation between two classical results: the bias-variance decomposition, and the approximation-estimation decomposition. Both are important conceptual tools in Machine Learning, helping us describe the nature of model fitting. It is commonly stated that they are “closely related”, or “similar in spirit”. However, sometimes it is said they are equivalent. In fact they are different, but have subtle connections cutting across learning theory, classical statistics, and information geometry, that (very surprisingly) have not been previously observed. We present several results for losses expressible as a Bregman divergence: a broad family with a known bias-variance decomposition. Discussion and future directions are presented for more general losses, including the 0/1 classification loss.
Original languageEnglish
JournalTransactions on Machine Learning Research
Publication statusPublished - 5 Mar 2024

Fingerprint

Dive into the research topics of 'Bias/Variance is not the same as Approximation/Estimation'. Together they form a unique fingerprint.

Cite this