Despite the undeniable progress in visual recognition tasks fueled by deep neural networks, there exists recent evidence showing that these models are poorly calibrated, resulting in over-confident predictions. The standard practices of minimizing the cross-entropy loss during training promote the predicted softmax probabilities to match the one-hot label assignments. Nevertheless, this yields a pre-softmax activation of the correct class that is significantly larger than the remaining activations, which exacerbates the miscalibration problem. Recent observations from the classification literature suggest that loss functions that embed implicit or explicit maximization of the entropy of predictions yield state-of-the-art calibration performances. Despite these findings, the impact of these losses in the relevant task of calibrating medical image segmentation networks, novel losses specific to task of segmentation, and vision-language remains unexplored.
In the first objective, we refer to one of the earlier works from our group which provides a unifying constrained-optimization perspective of current state-of-the-art calibration losses. Specifically, these losses are viewed as approximations of a linear penalty (or a Lagrangian term) imposing equality constraints on logit distances. This points to an important limitation of such underlying hard equality constraints, whose ensuing gradients constantly push towards a non-informative solution, which might prevent from reaching the best compromise between the discriminative performance and calibration of the model during gradient-based optimization. Following these insights, we extend the proposed simple and flexible generalization penalty which imposes a controllable margin on logit distances to medical image segmentation. We provide comprehensive experiments and ablation studies on seven different public segmentation benchmarks that focus on diverse targets and modalities, highlighting the generalization capabilities of the proposed approach. Our empirical results demonstrate the superiority of the margin based label smoothing compared to state-of-the-art calibration losses in both calibration and discriminative performance.
In the second objective, we provide a constrained-optimization perspective of Spatially Varying Label Smoothing (SVLS), demonstrating that it could be viewed as a standard cross-entropy loss coupled with an implicit constraint that enforces the softmax predictions to match a soft class proportion of surrounding pixels. Our formulation shows that SVLS lacks a mechanism to control explicitly the importance of the constraint, which may hinder the optimization process as it becomes challenging to balance the constraint with the primary objective effectively. Following these observations, we propose a simple and flexible solution based on equality constraints on the logit distributions. The proposed constraint is enforced with a simple linear penalty, which incorporates an explicit mechanism to control the weight of the penalty. Our approach not only offers a more efficient strategy to model the logit distributions but implicitly decreases the logit values, which results in less overconfident predictions. We conduct comprehensive experiments and ablation studies over multiple medical image segmentation benchmarks, including diverse targets and modalities, and show the superiority of our method compared to state-of-the-art calibration losses. Furthermore, several ablation studies empirically validate the design choices of our approach, as well as demonstrate its model agnostic nature.
In the third objective, we propose a class and region-wise constraint approach to tackle the miscalibration issue in semantic segmentation models. In particular, we formulate a solution that considers the specificities of each category and different regions by introducing independent class and region-wise penalty weights. This contrasts with the prior work, where a uniform scalar penalty weight is employed, regardless of categories or regions. Furthermore, we transfer the constrained problem to its dual unconstrained optimization counterpart by using an Augmented Lagrangian method (ALM). This alleviates the need for manually adjusting each penalty weight and allows, through a series of iterative inner and outer steps, to find the optimal value of each penalty weight, which can be learned in an adaptive manner. Comprehensive experiments on two popular segmentation benchmarks, and with two well-known segmentation backbones, demonstrate the superiority of our approach over a set of relevant recent calibration approaches.
In the fourth objective, we empirically demonstrate that popular CLIP adaptation strategies, such as Adapters, Prompt Learning, and Test-Time Prompt Tuning, substantially degrade the calibration capabilities of the zero-shot baseline in the presence of distributional drift. For these adaptation strategies, we expose that the underlying cause of miscalibration is, in fact, the increase of the logit ranges. This contrasts with recent work in calibrating fully-supervised models, which suggests that the inherent cause of miscalibration is the increase of its norm instead, due to the standard cross-entropy loss used for training. Based on these observations, we present a simple, and model-agnostic solution, which consists in scaling the logit range of each sample based on the zero-shot logits. We further present several alternatives to accommodate our solution, which can be implemented either at training or inference time. Comprehensive experiments on popular OOD classification benchmarks empirically demonstrate the effectiveness of our approaches to reduce the miscalibration error, while keeping the discriminative performance.
| Date | 7 Nov 2025 |
|---|
| Original language | American English |
|---|
| Awarding Institution | - École de technologie supérieure
|
|---|
| Supervisor | José Dolz (Supervisor) & Ismail Ben Ayed (Co-supervisor) |
|---|