Gesture-based exploration of 3D medical images in virtual reality

Abstract

Virtual reality (VR) is a technology that enables more immersive visualization and newinteraction possibilities compared to a standard desktop computer setup. In this thesis, we explore how VR can be used in the field of medical imaging to improve the visualization of human anatomy. This work focuses on two specific problems in anatomical visualization. For each problem, we examine how VR may help address current limitations or enhance the visualization process. The first problem is depth perception when visualizing cerebral angiographic data, which represents blood vessels inside the brain, or unsegmented data. The second problem is self-occlusion, which occurs when visualizing regions of the human body that are densely packed with organs and tissues, where outer layers of anatomy hide inner regions. We propose gesture-based interactive methods implemented in VR to tackle these two problems. For angiographic visualization, multiple depth visualization techniques have been developed to improve the readability of such data. Our first goal is to determine how techniques originally designed for 2D screens translate to VR, and how the depth cues provided by VR headsets affect them. We found that the evaluated depth perception techniques showed very few differences when measured with commonly used metrics, which suggests that VR already provides powerful depth cues. However, using study-specific metrics, we observed that variants with greater interactivity, where hand movements modified the visualization, performed better than less interactive variants. The visualization of datasets with substantial self-occlusion has been the focus of two publications. In both, we study interactive occlusion management in VR to improve data exploration. In the first publication, we propose Contextual Ambient Occlusion, a new implementation of an existing high-quality ambient occlusion technique. This method makes it possible to achieve real-time clipping with dynamically calculated volumetric ambient occlusion at every frame without sacrificing visual quality. The proposed algorithm improves depth perception in volume visualization by leveraging shading cues, thereby facilitating a clearer understanding of the underlying structure. In the second publication, we introduce AnatomyCarve, an interactive segment-aware clipping technique that facilitates exploration of specific segments, understood as pre-labeled anatomical structures obtained by image segmentation, in VR. AnatomyCarve employs the idea of stepwise dissection, making it possible to remove outer tissues layer by layer and reveal inner anatomy. It supports the creation of visualizations similar to anatomical book illustrations and shows high usability and positive ratings for preoperative planning from neurosurgeons and neurosurgical residents. Our work demonstrates that VR is a promising technology for medical visualization. Our findings indicate that in VR, existing depth cues provide strong perceptual support, but that enhancing interactivity may further improve depth perception. Our results also show that interactive clipping in VR is an effective way to visualize anatomical datasets with strong self-occlusion, enabling clear inspection of both segmented and unsegmented structures through precise control of clipping planes. Overall, this thesis highlights how VR can bridge the gap between visual fidelity and intuitive interaction in medical imaging, offering new pathways toward more effective and immersive exploration of complex anatomical data.

Date	27 Feb 2026
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Simon Drouin (Supervisor) & Marta Kersten-Oertel (Co-supervisor)