Meta-learning recommendation system for automating the process of selecting the best pool and a dynamic classifier selection algorithm

Abstract

Multiple Classifier Systems(MCS) have been widely studied as an alternative to improve pattern recognition applications in recent years. An ensemble of classifiers or an MCS produces a better recognition performance than a single classifier, according to several empirical studies. Ensembles consist of classifiers that are individually trained and whose predictions are combined to predict novel instances. According to previous research, an ensemble of classifiers is often more accurate than a single classifier. Dynamic Selection (DS) is considered one of the most effective approaches in the field of Multiple Classifier Systems (MCS). In DS, the choice of base classifiers is made dynamically for each new sample that needs to be classified. Instead of using a fixed set of classifiers for all instances, DS adapts and selects the appropriate classifiers based on the characteristics of each individual sample being classified. For DS methods, a pool of classifiers is employed for classification. The existing pool of classifiers used in dynamic selection methods lacks stability, leading to a significant difference between minimum and maximum classification results when different subsets of the same problem are used for training. Also, pool generation methods with a global perspective tend to generate redundant classifiers, making the system computationally expensive. Such methods may not effectively cover certain regions of the feature space, limiting the performance of dynamic selection methods in selecting the most competent models for specific instances. In this study, we present an analysis conducted evaluating multiple pool generation schemes when they are used as input of DS techniques. The significance of selecting an optimal pool of classifiers for dynamic selection (DS) algorithms is explored, and global and local perspective pool generation schemes are compared. Several DS techniques were selected based on their superior performance based on dynamic selection literature and are evaluated in this study to address research questions. The following two research questions are of primary importance: Are local pool generation schemes better than global pools? Does the best pool generation scheme depend on the DS algorithm and dataset in question? The study reveals that local pool generation schemes do not consistently achieve better results for dynamic selection. Furthermore, the selection of an appropriate pool of classifiers depends on both the DS method used and dataset characteristics. Since the above-mentioned analysis reveals that there is no pool generation scheme that can be the best selection for the DS algorithm on all datasets. Therefore, we present a meta-learning automated system to recommend the best pool generation scheme for DS methods for a given dataset. In this algorithm, a meta-model is created by extracting meta-features from datasets and utilizing a meta-target that represents the best DS algorithm’s performance. This meta-model is used to provide three recommendations: predicting a pool generation scheme, a DS method, and a pair recommendation of a pool and DS method simultaneous for a given dataset. Another experiment study is conducted to evaluate the performance of the meta-learning recommendation system, and the results show that the meta-learning recommendation system performs better than selecting a fixed pool of classifiers, a fixed DS method, or a fixed pair of a pool and DS method. In other words, according to the experimental results, meta-learning recommended the best solution with the highest predictive accuracy over the baselines (majority). The code and data used in this work are available on this project’s GitHub repository: https: //github.com/hesamjalalian/Pool_generation_scheme

Date	14 Dec 2023
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Rafael Menelau Oliveira Cruz (Supervisor)