NSF Funding Propels XR Systems Research Vision Toward its 'Next Big Leap'

Believing that now is the time to expand upon the capabilities of Extended Reality (XR) systems research – which combines virtual, augmented, and mixed reality – Illinois Computer Science professor Sarita Adve found a new and collaborative path forward.

Collaborative Expertise Displayed in NSF PPoSS Project

“XR research – or immersive computing, more generally – raises really interesting and non-trivial challenges in many parts of the system, including our group’s work on quality-aware performance optimizations and on programmability of distributed edge devices. In our work to date, we have developed novel compiler techniques to program devices with diverse heterogeneous computing elements, as well as a novel optimization framework that maximizes performance and energy efficiency. Both these classes of techniques – and new ones that build on them – will be essential for delivering on the goals this project.”

“XR is a cross-disciplinary moonshot problem, and this project is a means to take it on with a world-class set of collaborators. The philosophy I will be pushing during the project is how to build systems/hardware that provide a ‘Goldilocks’ degree of security – meaning, not too much and not too little for each given situation. Typically, we design systems that provide too much assurance; given the tight resource constraints in XR, that approach won't be practical.”

“The interdisciplinary nature of the project, also reflected in the well-experienced team, gives us the opportunity to rethink various parts of the computing stack of XR as the foundation of important applications in the future. Some of the key cross-cutting concerns include accuracy, robustness, performance, and energy efficiency. I will work on compilation techniques that incorporate noise and approximation in constrained computation/communication environments and systems for reasoning about accuracy through testing and formal methods.”

“Immersive environments are dealing with resource-constrained devices, systems, and networks such as mobile head-mounted devices (HMD) interconnected with other mobile HMDs. To give users high immersive perception and high quality of experience is the priority but distributing all modalities might not be possible due to the limited end-to-end processing and network resource availability. This is where my expertise comes in. To achieve the network delivery of the right data at the right time to the right place, I will collaborate and optimize not only the application-network algorithms and protocols, but also convey the right information to computer architecture, security, privacy, compilers, and other system components.”

At this point, Adve believes the next step includes a need for multiple collaborators – the computer architects, system designers, compiler writers, as well as algorithm and application developers – that could push this technology forward in a cohesive way.

To do exactly that, she worked with seven colleagues for a five-year, approximately $5 million grant funded by the NSF Principles and Practice of Scalable Systems (PPoSS) program. This multidisciplinary and multi-institutional vision is now a reality.

“The real promise of XR and immersive computing is in multi-user distributed applications,” Adve said. “Since this is a naturally distributed problem, all the work we’ve been doing on XR fit really nicely in the context of this call from PPoSS.

“So, we wrote this proposal as the next big leap within XR research by expanding our efforts wider through a distributed system.”

Beginning work in October, the team surrounding Adve includes five collaborators from Illinois CS and two more from Georgia Tech University:

• Vikram Adve and Sasa Misailovic will lead the work on compilers and a cross-cutting accuracy thrust
• Christopher Fletcher will work on hardware efforts with Sarita
• Klara Nahrstedt, Radhika Mittal (Illinois CS affiliate faculty with Electrical & Computer Engineering), and Georgia Tech’s Ada Gavrilovska will bring systems expertise
• And Fletcher, as well as Vladamir Kolesnikov from Georgia Tech, will lead the work on security

“The aspect of this call that we really appreciated is it’s about creating a multi-disciplinary, co-designed system,” Sarita said. “The program explicitly asked for four different areas of computer science or computer systems. We were interested in bringing all four of these areas together, plus the cross-cutting accuracy aspect of the work.

“We all strongly believe that the future of systems is in a codesigned approach where researchers across the system stack work together. We have created such a team for this project. Many of us have worked together in the past and are now excited about amplifying our impact through this diverse but cohesive team to work towards a complete distributed XR system design.”

Sarita said that this project, entitled “Scalable Specialization in Distributed Edge-Cloud Systems – The Extended Reality Case,” will push beyond her previous work in XR.

As the project abstract states, this team “will develop design methodologies for a scalable, domain-specific, heterogeneous, distributed edge/cloud system with stringent constraints on latency, energy, thermal power, computational requirements, and size. The work will use distributed multiparty augmented / virtual / mixed reality (collectively, extended reality or XR) experiences as target parallel and distributed applications with challenging quality-of-experience goals, scalability requirements, design constraints, and diverse and fast-evolving algorithmic components.”

The end goal for the entirety of this work, according to Sarita, is to produce real-world impact on people’s lives in several different ways.

Many people, she said, still have an instinctual reaction to XR as a source for gaming. However, there are many other applications for the systems this XR project will create. For example, there are transformative benefits for areas like healthcare, education, crisis response, manufacturing, training, and more.

“There are so many potential fields this technology can apply to, but just to focus on one for the moment, let’s think about healthcare,” Sarita said. “We are now talking to physicians at the Carle Illinois College of Medicine right here on campus. They are already using XR for training their students and surgeons in area hospitals are using 3D reconstructions for surgery planning. They all provide glowing reviews of the transformational potential of this technology but also say there is much to be done.

“We want to make this technology is effective and comfortable not only within the local healthcare environment but also to enable remote and mobile hospitals to bring together physicians and patients distributed across geographic areas as a way to provide better healthcare.”

The possibilities may seem endless, but in a short amount of time, the team members, working in subsets, have laid the groundwork that enables them to come together now – ready to meet this moment.

In spring of 2021, Sarita and her students worked to debut the Illinois Extended Reality (ILLIXR) testbed. As a fully open-source, end-to-end Extended Reality (XR) system, ILLIXR allows research, development, and benchmarking of ideas in the context of a complete XR system, and the ability for users to see their impact on end-user experience.

“Many in the team have already been working with ILLIXR on a plethora of research ideas involving hardware acceleration, compilers, and runtimes, and I am excited that this grant will allow us to move all of our efforts forward in a cohesive way to solve some of the most difficult problems in computer systems today for societal impact,” Sarita said.