Passer à la navigation principale Passer à la recherche Passer au contenu principal

On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do we Really need Prompt Learning?

  • Université catholique de Louvain
  • UMons

Résultats de recherche: Chapitre dans un livre, rapport, actes de conférence › Participation à un ouvrage collectif lié à un colloque ou une conférence › Revue par des pairs

41 Citations (Scopus)

Résumé

The development of large vision-language models, notably CLIP, has catalyzed research into effective adaptation techniques, with a particular focus on soft prompt tuning. Conjointly, test-time augmentation, which utilizes multiple augmented views of a single image to enhance zero-shot generalization, is emerging as a significant area of interest. This has predominantly directed research efforts toward test-time prompt tuning. In contrast, we introduce a robust MeanShift for Test-time Augmentation (MTA), which surpasses prompt-based methods without requiring this intensive training procedure. This positions MTA as an ideal solution for both standalone and API-based applications. Additionally, our method does not rely on ad hoc rules (e.g., confidence threshold) used in some previous test-time augmentation techniques to filter the augmented views. Instead, MTA incorporates a quality assessment variable for each view directly into its optimization process, termed as the inlierness score. This score is jointly optimized with a density mode seeking process, leading to an efficient trainingand hyperparameter-free approach. We extensively benchmark our method on 15 datasets and demonstrate MTA's superiority and computational efficiency. Deployed easily as plug-and-play module on top of zero-shot models and state-of-the-art few-shot methods, MTA shows systematic and consistent improvements.

langue originaleAnglais
titreProceedings - 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
EditeurIEEE Computer Society
Pages23783-23793
Nombre de pages11
ISBN (Electronique)9798350353006
Les DOIs
étatPublié - 2024
Evénement2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Seattle, Etats-Unis
Durée: 16 juin 2024 → 22 juin 2024

Série de publications

NomProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (imprimé)1063-6919

Conférence

Conférence2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
Pays/TerritoireEtats-Unis
La villeSeattle
Période16/06/24 → 22/06/24

Classification disciplinaire ASJC Scopus

  • Logiciel
  • Vision par ordinateur et reconnaissance de formes

Empreinte digitale

Voici les principaux termes ou expressions associés à « On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do we Really need Prompt Learning? ». Ces libellés thématiques sont générés à partir du titre et du résumé de la publication. Ensemble, ils forment une empreinte digitale unique.

Citer cette ressource