Designing End-user Creation Tools for Immersive Experiences

Abstract

Although immersive experiences powered by Augmented Reality (AR) and Virtual Reality (VR) devices are becoming increasingly engaging, democratizing their creation to empower end-users, particularly those with limited technical skills, to create, share, and communicate, remains a significant challenge. This dissertation explores the design and development of end-user creation tools for immersive experiences, focusing on enhancing expressiveness, supporting design exploration and collaboration, and fostering meaningful social interactions. My work investigates the challenges faced by novice programmers and designers in creating immersive content and proposes innovative solutions to democratize the creation process.

To raise the ceiling of expressiveness, the dissertation introduces FlowMatic, an immersive authoring tool that enables novice programmers to define reactive behaviors through visual representations of Functional Reactive Programming (FRP) in VR. This tool allows users to create interactive virtual experiences with ease. To support design exploration and collaboration, the dissertation introduce a new collaborative version control system named VRGit, which facilitates design exploration and rapid iterations by providing rich history-keeping and workspace awareness in VR. Furthermore, the dissertation explores the integration of generative AI models in immersive authoring through VRCopilot, enabling creative design explorations via human-AI co-creation. Lastly, the dissertation also examines the impact of content created with these tools on social interactions, presenting Auggie, an AR-based communication tool, and Jigsaw, an AR and IoT-based storytelling tool. These tools aim to encourage effort in remote communication and engagement in immersive stories, with positive impact on people's social interactions via immersive content creation.

Throughout the dissertation, I demonstrate that by incorporating novel visualization and interaction techniques in immersive environments, we can design end-user creation tools for immersive experiences that can express reactive behaviors, facilitate design exploration and collaboration, and foster meaningful social interactions.