TBD

Extended reality (XR)- also referred to as immersive computing- which encompasses virtual, augmented, and mixed reality (AR, VR, MR), will affect the way we teach, conduct science, practice medicine, entertain ourselves, train professionals, interact socially, and more. It is thought to be the next interface for most of computing. While current XR systems exist today, they are far from providing a tetherless experience approaching perceptual abilities of humans. There is a gap of several orders of magnitude between what is needed and what is achievable in performance, power, and usability, requiring deep innovations from system researchers. At the same time, with the end of Dennard scaling and Moore's Law, application-driven specialization or domain-specific computing has emerged as a key architectural technique to meet the requirements of emerging applications. Computer architects have responded with an explosion of research on highly efficient accelerators, targeting machine learning and other domains.

To truly achieve the promise of efficient, domain-specific computing in general and for the XR domain in particular, systems researchers will need to broaden their portfolio beyond specialization for individual accelerators. Instead, researchers must develop the science for specializing domain-specific systems which may consist of multiple sub-domains requiring multiple parallel heterogeneous accelerators that interact with each other to collectively meet end-user demands. A key obstacle to domain-specific systems research for XR is that there have been no open-source benchmarks or testbeds covering the entire XR workflow to drive such research. Our infrastructure aims to fill this gap.

This project develops an open-source end-to-end infrastructure for XR devices and builds on an initial research prototype, ILLIXR (Illinois Extended Reality Testbed). The system is being designed to contain state-of-the-art components for a complete XR workflow, an extensible runtime that orchestrates the scheduling of these components, and extensive telemetry support to measure performance, power, and end-to-end quality of experience metrics. The system is extensible and supports a variety of operating systems (e.g., Linux, Android) and heterogeneous platforms (e.g., NVIDIA Jetson, Qualcomm Snapdragon, etc.), sensors (e.g., cameras, IMUs, etc.), and various XR applications. It enables new research opportunities in all parts of the computing stack to tackle end-to-end XR system innovations that were previously not possible.

Systems researchers benefit from using the infrastructure to drive new research in post-Moore domain-specific systems in the areas of computer architecture, programming languages, compilers, runtime systems, and security and privacy. The end-to-end infrastructure drives new techniques in co-designed systems that are optimized for end-to-end user experiences. For applications, XR encompasses multiple sub-domains such as computer vision, robotics, graphics, signal processing, and machine learning. Algorithms researchers in these areas can prototype and test new algorithms that are optimized for end-to-end system efficiencies without worrying about implementing the rest of the stack, and XR researchers will be able to design systems optimized for the end-to-end user experience.

This work addresses two of the most important problems in computing: dealing with the end of Moore's Law and designing systems that achieve the potential of immersive computing. Both have the potential for tremendous impact on society at large.