Deep weakly supervised learning networks for object localization

Abstract

Visual recognition, encompassing tasks such as object classification, localization, and segmentation, is crucial for different computer vision applications ranging from autonomous driving to medical image analysis. Recent advances in deep neural networks have yielded significant improvements in these tasks. However, the success of these models is heavily attributed to the large amount of annotated data, which are costly and time-consuming to acquire. Additionally, this paradigm is not scalable across different domains, as it is impractical to collect millions of labeled images for every domain in which we want to deploy our deep learning model. To avoid reliance on extensive annotations, the research community has explored different learning paradigms, including semi-supervised, self-supervised, and weakly supervised learning methods. To contribute to this effort, this thesis focuses on weakly supervised learning for object localization. First, we identified the challenges associated with existing methods, followed by the proposal of new directions aimed at improving object localization performance. DiPS is the first contribution to this thesis; we propose to leverage class-agnostic maps from self-supervised transformers for weakly supervised object localization, where only image-class labels are available. Using different attention maps from self-supervised transformers, we first select a map that contains discriminative regions corresponding to the object of interest for generating object proposals. These proposals are then used to build pseudo-labels by selecting a few background and foreground pixels that fit the model, which is capable of performing both classification and localization tasks. This sampling enables our model to mitigate over-activations and background noise. Extensive experiments on the challenging TelDrone, CUB, OpenImages and ILSVRC datasets show that the proposed method can achieve better localization performance compared to the state-of-the-art methods. As a second contribution, we introduce text distillation for localization (TeD-Loc), a learning strategy for distilling localization information from CLIP text embeddings into the visual backbone of our model to improve localization performance. Conventional WSOL methods typically extract localization maps from pre-trained classifiers, which tend to focus on the most discriminative object parts and fail to highlight different objects. Recent WSOL methods that leverage vision-language models like CLIP also depend on ground truth class labels or class predictions from external classifiers to generate localization maps, thereby limiting their deployment in different applications. To address these limitations, TeD-Loc distills information from the CLIP text embeddings into the backbone of our vision model. Furthermore, by formulating the problem within a multi-instance learning framework, TeD-Loc enable the model to converge for both object localization and classification tasks. Experimental results show that TeD-Loc achieves state-of-the-art performance. The third contribution of this thesis is a realistic evaluation protocol for WSOL. As WSOL models are trained using class-level labels, their evaluation, hyperparameter tuning, and selection still rely on a validation set that includes bounding-box annotations. Employing bounding-box annotation for evaluation is not aligned with the WSOL paradigm, leading to overestimating localization performance. For realistic evaluation, we propose to employ pseudo-boxes for a validation set that can obtained from the off-the-shelf region proposal method. Extensive experiments on both natural and medical image datasets show that the performance of the model evaluated using pseudo-boxes is comparable to the performance obtained using manually annotated GT boxes.

Date	10 Jun 2025
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Éric Granger (Supervisor) & Marco Pedersoli (Co-supervisor)