Backpropagation-free adaptation of vision transformers for robust and efficient 3D and 2D perception

Abstract

Deep learning models have achieved excellent results in computer vision, especially with the advent of Transformers and large segmentation models. However, these models are often trained under controlled conditions and assume that the training and test data follow the same distribution. In practice, this assumption is rarely satisfied, especially in 3D industrial applications, where sensors, capture conditions, and objects can vary greatly. These domain shifts lead to significant drops in performance and limit the reliability of systems in production. This thesis focuses on Test Time Adaptation (TTA) in concrete 3D vision and industrial object scenarios. We consider two main settings. The first concerns robust classification of point clouds with 3D Transformers subject to corruptions and domain shifts. The second addresses instance segmentation of “model-based” objects that are never seen during training but are described by CAD models or a few reference views (the MISUO scenario). In both contexts, we face strong constraints: the absence of labels at test time, the impossibility of accessing source data, and the need for methods that are lightweight in terms of computation time and memory. Our first contribution introduces Purge-Gate, a backpropagation free test-time adaptation method for Transformers on point clouds. The core idea is to operate at the token level. For each token, we estimate how “shifted” it appears with respect to the source distribution, and then remove the most affected tokens before the attention layers. This allows the model to focus its attention on more reliable regions of the point cloud. We propose two variants: PG-SP, which uses source statistics when they are available, and PG-SF, which is completely source-free and uses a prototype derived from the CLS token. Both variants operate without gradients at test time and significantly improve robustness, speed, and memory efficiency on several corrupted 3D benchmarks. In our second contribution, we revisit the role of Layer Normalization (LN) for test time adaptation of 3D Transformers. We show theoretically that, as the feature dimension and batch size increase, the pre-affine output of LN concentrates around zero mean and unit variance. In this regime, re estimating batch statistics in the style of BatchNorm is unnecessary and can even degrade performance. Based on this analysis, we propose LN-TTA, a simple method that keeps the pre-affine part of LN unchanged and adjusts only the affine parameters (bias and gain) in a single forward pass. LN-TTA is source-free, backpropagation-free, very lightweight computationally, and yields significant gains on several corrupted point cloud suites. Finally, our third contribution extends these ideas to a more complex setting: instance segmentation of model-based objects that are never seen during training. We present PRISMA, a training free framework that refines prompts for large segmentation models using visual prototypes built from CAD models or multi-view images. PRISMA relies on three complementary modules: (i) a Mask Proposal Generation (MPG) module that uses DINO-style descriptors to propose object-specific regions of interest, (ii) a Self-Grounding module that injects object prototypes into the backbone and separates foreground and background tokens through block-diagonal attention, without retraining, and (iii) a Confidence Weighted (CW) scheme that leverages the segmenter’s internal scores to better rank predictions. On the BOP benchmark, PRISMA improves both the quality and the speed of MISUO segmentation compared to existing methods, while remaining completely training-free. Overall, this thesis shows that it is possible to make modern 2D/3D vision models more robust and better suited to industrial constraints by relying on simple, analytical, backpropagation-free test-time adaptation mechanisms. By acting directly on normalization, token selection, and the use of visual prototypes, we propose a new toolbox for deploying powerful models in real 3D environments without requiring costly retraining procedures.

Date	31 Mar 2026
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Christian Desrosiers (Supervisor) & Ismail Ben Ayed (Co-supervisor)