A Probabilistic Perspective on Ensemble Diversity

Abstract

We study diversity in classifier ensembles from a broader perspectivethan the 0/1 loss function, the main reason being that thebias-variance decomposition of the 0/1 loss function is not unique,and therefore the relationship between ensemble accuracy and diversityis still unclear. In the parallel field of regression ensembles,where the loss function of interest is the mean squared error, thisdecomposition not only exists, but it has been shown that diversitycan be managed via the Negative Correlation (NC) framework. In thefield of probabilistic modelling the expected value of the negativelog-likelihood loss function is given by its conditional entropy; thisresult suggests that interaction information might provide someinsight into the trade off between accuracy and diversity. Ourobjective is to improve our understanding of classifier diversity byfocusing on two different loss functions -- the mean squared error andthe negative log-likelihood.In a study of mean squared error functions, we reformulate the Tumer &Ghosh model for the classification error as a regression problem, andwe show how the NC learning framework can be deployed to managediversity in classification problems. In an empirical study ofclassifiers that minimise the negative log-likelihood loss function,we discuss model diversity as opposed to error diversity in ensemblesof Naive Bayes classifiers. We observe that diversity in low-varianceclassifiers has to be structurally inferred. We apply interactioninformation to the problem of monitoring diversity in classifierensembles. We present empirical evidence that interaction informationcan capture the trade-off between accuracy and diversity, and thatdiversity occurs at different levels of interactions between baseclassifiers. We use interaction information properties to buildensembles of structurally diverse averaged Augmented Naive Bayesclassifiers. Our empirical study shows that this novel ensembleapproach is computationally more efficient than an accuracy basedapproach and at the same time it does not negatively affect theensemble classification performance.

Date of Award	1 Aug 2011
Original language	English
Awarding Institution	The University of Manchester
Supervisor	Gavin Brown (Supervisor)