Dynamic selection of ensemble of classifiers using meta-learning

Abstract

Dynamic ensemble selection systems work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify each specific test sample. The classifiers’ competences are usually estimated over the neighborhood of the test sample, according to a given criterion, such as the local accuracy estimates or the confidence of the base classifier, computed over the neighborhood of the test sample. However, using only one selection criterion can lead to poor estimation of the classifier’s competence. Consequently, the system end up not selecting the most appropriate classifier for the classification of the given test sample. In this thesis, dynamic ensemble selection is formalized as a meta-problem. From a metalearning perspective, the dynamic ensemble selection problem is considered as another classification problem, called the meta-problem. The meta-features of the meta-problem are the different criteria used to measure the level of competence of the base classifier. Each set captures a different property of the behavior of the base classifier, and can be seen as a different criterion for estimating the competence level of a base classifier; such criteria include, the classification performance in a local region of the feature space and the classifier confidence for the classification of the input sample. The meta-classifier is trained, based on the defined set of meta-features, to predict the competence level of a given base classifier for the classification of a new test sample. Thus, several criteria can be used in conjunction for a better estimation of the classifiers’ competences. In Chapter 2, a novel dynamic ensemble selection framework using meta-learning is proposed, called META-DES. Five distinct sets of meta-features, each corresponding to a different criterion for measuring the level of competence of a classifier for the classification of input samples are introduced for this specific meta-problem. The meta-features are extracted from the training data and used to train a meta-classifier to predict whether or not a base classifier is competent enough to classify an input instance. During the generalization phase, the meta-features are extracted from the query instance and passed down as input to the meta-classifier. The metaclassifier estimates whether a base classifier is competent enough to be added to the ensemble. Experiments are conducted over several small sample size classification problems, i.e., problems with a high degree of uncertainty due to a lack of training data. Experimental results show the proposed meta-learning framework greatly improves classification accuracy when compared against current state-of-the-art dynamic selection techniques. In Chapter 3, a step-by-step analysis of each phase of the META-DES framework is conducted. We show how each set of meta-features is extracted as well as their impact on the estimation of the competence level of the base classifier. Moreover, an analysis of the impact of several factors on the system performance is carried out; these factors include, the number of classifiers in the pool, the use of different linear base classifiers, as well as the size of the validation data. Experimental results demonstrate that using the dynamic selection of linear classifiers through the META-DES framework, it is possible to solve complex non-linear classification problems using only a few linear classifiers. In Chapter 4, a novel version of the META-DES framework based on the formal definition of the Oracle, called META-DES.Oracle is proposed. The Oracle is an abstract method that represents an ideal classifier selection scheme. A meta-feature selection scheme using an overfitting cautious BPSO is proposed for improving the performance of the meta-classifier. The difference between the outputs obtained by the meta-classifier and those presented by the Oracle is minimized. Thus, the meta-classifier is expected to provide results that are similar to those of the Oracle. Experiments carried out using 30 classification problems demonstrate that the optimization procedure based on the Oracle definition leads to a significant improvement in classification accuracy when compared to previous versions of the META-DES framework. Finally, in Chapter 5, two techniques are investigated in order to improve the generalization performance of the META-DES framework as well as any other dynamic selection technique. First, a prototype selection technique is applied over the validation data to reduce the amount of overlap between the classes, producing smoother decision boundaries. During generalization, a local adaptive K-Nearest Neighbor algorithm is employed for a better definition of the neighborhood of the test sample. Thus, DES techniques can better estimate the classifiers’ competences. Experiments were conducted using 10 state-of-the-art DES techniques over 30 classification problems. The results demonstrate that the use of prototype selection in editing the validation data and the local adaptive distance significantly improve the classification accuracy of dynamic selection techniques.

Date	9 Jun 2016
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Robert Sabourin (Supervisor) & George D.C. Cavalcanti (Co-supervisor)