Amélioration des modèles basés sur Transformers pour le traitement des nombres dans les documents médicaux

Abstract

Medical records created by healthcare professionals upon patient admission are rich in details critical for diagnosis. Yet, their potential is not fully realized due to obstacles such as complex medical language, inadequate comprehension of medical numerical data by state-of-the-art Language Models (LMs), and the limitations imposed by small annotated training datasets. This research focuses on classifying numerical values extracted from medical documents into seven distinct physiological categories using CamemBERT-bio, a Transformer-based model. While earlier studies suggested that Transformer models might underperform compared to traditional Natural Language Processing (NLP) approaches for such tasks, more recent findings indicate that a significant increase in the number of parameters enables language models to develop new capabilities, thereby improving their performance. However, these large-scale models, known as LLMs, require substantial computational and memory resources. In this study, we focus on medium-sized language models based on the Transformer architecture, such as CamemBERT-bio. To enhance its performance, our approach consists of two phases. First, we introduce two main innovations : integrating keyword embeddings into the model and adopting a number-agnostic strategy by excluding all numerical data from the text. The implementation of label embedding techniques refines the attention mechanisms, while using a ’numerical-blind’ dataset aims to bolster context-centric learning. Another key component of our research is determining the criticality of extracted numerical data. To achieve this, we utilized a straightforward approach that involves verifying if the value falls within established standard ranges. Results : Our findings are encouraging, showing substantial improvements in the effectiveness of CamemBERT-bio, surpassing conventional methods with an F1 score of 0.89. This represents an increase of over 20% compared to the F1 score of 0.73 achieved by traditional approaches, with only a 6% gap from the F1 score of GPT-4—a state-of-the-art model that is several hundred times larger than ours. In the second phase, building on prior findings that reveal potential limitations of transformerbased models, we examine two strategies : fine-tuning CamemBERT-bio on a small medical dataset with the integration of Label Embedding for Self-Attention (LESA), and combining LESA with additional enhancement techniques such as Xval. Given that CamemBERT-bio is already pre-trained on a large medical dataset, the first approach aims to update its encoder with newly added label embeddings, while the second approach seeks to develop multiple representations of numbers (contextual and magnitude-based) for more robust number embeddings. Results : As anticipated, fine-tuning the standard CamemBERT-bio on our small medical dataset did not improve F1 scores. However, similar improvements were observed when combining LESA with Xval, surpassing conventional methods and further narrowing the performance gap with GPT-4, the state-of-the-art model. In summary, our work introduced various methods for handling numerical data, which are also applicable to other modalities. We illustrate how these novel approaches can support transformer-based models in delivering robust performance on classification tasks, even when dealing with small datasets.

Date	12 May 2025
Original language	French
Awarding Institution	École de technologie supérieure
Supervisor	Rita Noumeir (Supervisor) & Philippe Jouvet (Co-supervisor)