Enhancing data accessibility in built asset digital twins with neural language models and immersive technologies

Abstract

The rapid digital transformation within the architecture, engineering, construction, and facilities management industries has led to a growing interest in digital twinning for built assets—creating virtual, data-driven representations that mirror the states and and behaviors of real-world built asset components and systems. To actualize the full potential of digital twins—as dynamic, data-driven representations of physical assets—it is essential to integrate real-time data streams with static information. In this light, Building Information Modeling (BIM) and Internet of Things (IoT) systems can serve as foundational components of built asset digital twins— combining BIM’s static geometrical and rich semantic data with live IoT sensor data. However, despite extensive research and advancements in interoperability and data exchange frameworks, particularly those centered around the open BIM paradigm, the complexity of accessing and interpreting built asset data remains a significant barrier to intuitive user interactions with digital twin interfaces. This thesis sets out to address the critical issue of data accessibility in digital twin interfaces by investigating the practical potentials of two synergistic approaches: virtual reality (VR) for enhancing data representation and neural language models for improving information retrieval. In particular, the thesis examines the potential of immersive VR environments to intuitively represent the complex built asset data, offering users a more interactive and spatially coherent means of navigating and interpreting multimodal information that comes from various sources. Through a case study focused on real-time thermal comfort monitoring, the research demonstrates how VR-based interfaces can provide an effective medium for seamless representation of BIM and IoT data, which can significantly improve the data-driven decision-making process for practitioners in complex real-world scenarios. In parallel, the thesis investigates the application of neural language models, including customtrained deep neural networks and pre-trained large language models (LLMs), for enhancing information retrieval in BIM-centered digital twin systems. Motivated by the promising potential of advanced natural language processing techniques for improving search systems by incorporating semantics, this work evaluates the effectiveness of state-of-the-art neural language modeling techniques for the specific task of entity extraction from user queries. The experimental results, comparing traditional and emerging deep learning architectures, provide novel insights for both research and practice. They underscore the critical importance of domain adaptation strategies for effective model performance in this specialized context. Moreover, to address the shortcomings of existing general-purpose benchmarks in examining the transferability of the pre-trained LLMs’ capabilities for downstream tasks, such as those related to information retrieval, the thesis presents a comprehensive benchmark of state-of-the-art models. The thesis is accompanied by the public release of the proposed benchmark resources, including large-scale, high-quality datasets curated from industry-renowned sources. Given the surge of interest in applying LLMs to various applications within the built environment research, this is a particularly timely and essential contribution in this rapidly evolving field. The thesis concludes by highlighting key limitations and challenges encountered, accompanied by recommendations for future research with the central theme of fostering more intuitive user interactions with built asset digital twins.

Date	24 Dec 2024
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Ali Motamedi (Supervisor)