Computer vision methods adapting to new domains with few samples

Abstract

Deep learning models have achieved remarkable success in a wide array of computer vision tasks, from classification to segmentation and beyond. However, these models are often trained under the assumption that the data encountered at test time will be drawn from the same distribution as the training set. In real-world scenarios, this assumption rarely holds. Even subtle variations in lighting, camera properties, background textures, or object appearances can lead to significant shifts in data distribution—resulting in a sharp drop in model performance. This vulnerability to distribution shifts raises critical concerns about the robustness and reliability of vision systems deployed in the wild. In contrast to deep neural networks, human perception exhibits a natural resilience to such changes. We recognize people and objects under varying conditions without retraining or explicit supervision, guided by contextual cues and generalizable priors. Inspired by this, the focus of this thesis shifts from the traditional training-centered paradigm to an inference-centered approach, where models are not retrained or fine-tuned offline, but instead adapt dynamically and unsupervised at test time. This setting—commonly referred to as Test-Time Adaptation (TTA)—presents unique challenges, such as the absence of labeled data, the inaccessibility of source training data due to privacy or storage constraints, and the need for rapid, on-the-fly adaptation. To address these challenges, this thesis presents a series of modular, architecture-agnostic methods for adapting vision models during inference, with a focus on robustness, computational efficiency, and broad applicability. As a first contribution, we introduce NC-TTT: a noise contrastive test-time training framework tailored for convolutional neural networks. Instead of minimizing entropy or updating batch statistics, our method relies on an auxiliary contrastive task that learns to distinguish between noisy augmentations of feature representations. This allows the model to reinforce its understanding of in-distribution features while suppressing out-of-distribution noise—without any labels or access to source data. NC-TTT demonstrates strong performance under various types of distribution shift, including corrupted inputs and synthetic-to-real domain gaps. In the second contribution, we extend test-time adaptation to Vision-Language Models (VLMs)—particularly CLIP, which has gained prominence for its zero-shot capabilities. We introduce CLIPArTT, a method that leverages the inherent compositionality of language to adapt textual prompts at test time. Rather than using fixed, handcrafted prompts, we propose a strategy for dynamically constructing prompts using the model’s own top-K predictions and multi-modal similarity scores. This pseudo-label-guided adaptation enables CLIP to realign its predictions with target distributions—improving classification accuracy across various corrupted and domain-shifted datasets. Our third core contribution explores the potential of multi-template adaptation in VLMs through a method called WATT (Weight Averaged Test-Time Adaptation). Here, we adapt CLIP to the target distribution using several diverse textual templates, each representing a different linguistic framing of the classes. Rather than selecting a single best template, we aggregate their learned model weights using a principled weight averaging strategy. The result is a more stable and generalizable model that leverages the diversity of prompts while avoiding overfitting to any single linguistic perspective. Together, these contributions form a cohesive and forward-looking approach to building robust, adaptive, and deployment-ready vision models. By minimizing reliance on labeled data and source domain access, and by prioritizing modularity and architectural flexibility, this thesis paves the way for a new generation of intelligent systems that learn not only during training—but also evolve continually during inference.

Date	15 Sept 2025
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Christian Desrosiers (Supervisor) & Ismail Ben Ayed (Co-supervisor)