RVS-CUDA: A Real-Time Asynchronous Pipeline for Immersive View Synthesis

Intermediate view synthesis is a fundamental operation in applications that aim to create compelling, interactive visual experiences from captured scenes, such as virtually visiting museums or touristic sites. To create such experiences, the synthesis must be performed in real-time at a high frame rate. Reference View Synthesizer (RVS) is an open-source solution enabling the synthesis of any view located between cameras positioned in an environment through interpolation techniques. However, the speed of RVS tools in the current version 4.0 does not meet the real-time requirement necessary for practical applications. Even the OpenGL-optimized version of RVS, exploiting the processing power of GPUs, is too slow. Additionally, it only supports raw YUV input files, whereas in real-world scenarios, input views are typically provided as compressed video streams. To address these limitations, we introduce a CUDA-based implementation of RVS, referred to as RVS-CUDA, which is optimized for real-world applications by offering significantly improved speed through GPU acceleration and support for compressed input views. Experimental results show that, using four 4K input views (cameras), the proposed RVS-CUDA achieves real-time performance exceeding 30 frames per second (fps) for both raw YUV and compressed input formats (about 1000 × and 14 × faster than the original RVS and OpenGL RVS versions, respectively). Furthermore, the output quality of RVS-CUDA is comparable to that of the initial RVS implementation. The source code of the proposed method is publicly available at: https://github.com/EnzoN7/RVS-CUDA.