Illinois Leads $7.1M NSF Project Advancing Low-Latency Networking for XR-Based Medical Collaboration

Consider an emergency scenario with paramedics transporting a patient in an ambulance. The patient has pleural effusion, fluid accumulation around the lungs causing difficulty breathing, and needs immediate intervention to drain the fluid. The paramedic can use a portable ultrasound machine for diagnostic imaging and the necessary thoracentesis procedure, but it takes time to shift focus between the ultrasound monitor and the patient. This makes it hard to quickly guide a needle to exactly the right location with consideration of the surrounding anatomy without making a mistake, especially if the procedure is difficult or performed infrequently.

Now imagine the paramedic puts on an extended reality (XR) headset. The paramedic sees a real-time 3D visualization of the ultrasound overlayed onto the physical view of the patient, providing a precise understanding of the spatial relationships between the needle, surrounding organs, and the fluid pocket. And in a matter of moments, a specialist located at a major hospital is able to join the 3D experience remotely, seeing from the paramedic’s perspective and providing verbal and visual guidance to the paramedic about the necessary procedure.

This scenario is futuristic. One of the key hurdles comes down to network latency – that is, the time delay to transmit a message from sender to receiver. Networks today, especially deployed wireless networks, struggle to provide the consistent low latency that demanding interactive applications like XR need.

A new project led by the University of Illinois Urbana-Champaign seeks to change that. Project Agora, a $7.1 million U.S. National Science Foundation (NSF)-backed effort, is developing novel low-latency networking technology, along with new XR systems that leverage it for remote collaboration in medical procedures and education. The project, which is part of Illinois’s IMMERSE Center for Immersive Computing, brings the university and the Carle Illinois College of Medicine together with industry project members Peraton Labs, MediView, T-Mobile, Qualcomm, and Cisco Systems.

The name Agora represents the two big challenges we’re tackling. In ancient Greece, an agora was an open public space, signifying our goal of enabling collaboration in XR spaces. And in Portuguese, agora means ‘now’ – alluding to how quickly we’d like low-latency networks to communicate.

— Brighten Godfrey lead Principal Investigator (PI) and professor in the Siebel School of  Computing and Data Science at The Grainger College of Engineering at Illinois

Godfrey jointly leads the project with PI Christine Zhang, Director and Chief Research Scientist at Peraton Labs, and co-PI Sarita Adve, Richard T. Cheng Professor of Computer Science at Illinois.

A University-Industry partnership

NSF recognized the latency challenge through a grant program titled “Breaking the Low Latency Barrier for Verticals in Next-G Wireless Networks (Breaking Low)”. Backed by the NSF Directorate for Technology, Innovation and Partnerships (NSF TIP), which seeks to accelerate translation of research technology to industry and society, Agora is one of three projects funded to position the U.S. to take the lead in low latency networking for emerging application areas.

“NSF is investing in the development of innovative low-latency wireless communications technologies that will not only improve connectivity for everyday use but also power a whole range of applications requiring extremely fast data transmission,” said Erwin Gianchandani, NSF assistant director for TIP. “Agora is working to identify and solve critical issues that address key barriers in current wireless technologies that will make important advances like real-time and remote collaboration in medicine and education a reality.”

Assembling a team with the diverse expertise to tackle the Agora project’s goals was no small task. The project’s team spanning academia and industry came together through a combination of an NSF “Ideas Lab” workshop and the IMMERSE Center for Immersive Computing at Illinois.

“Immersive computing, including XR, has the chance to transform many human endeavors from education to healthcare to arts, but only if we can solve the hard problems in technologies, application domains, and human experience”, said Adve, who is director of the IMMERSE Center. “IMMERSE was established to bring together people across all these areas to enable the interdisciplinary research needed to address these problems and push immersive computing forward. The Agora project is a poster child for the IMMERSE vision.”

Democratizing access to medical expertise and to low latency networking

Medical tech start-up MediView built two of the only FDA-cleared augmented reality-based medical visualization and navigation platforms (XR90 and OmnifyXR) that incorporate live medical imaging for in-human soft tissue and bone procedures. Now, as part of Agora, MediView is contributing proprietary software and custom integrated hardware to enable real time imaging and instrument tracking for remote first-person collaboration during clinical care delivery, prototyped with Agora XR runtime and low latency networking components.

Carle Illinois College of Medicine, which is at the forefront of using XR for education as well as for clinical work, is leveraging the Agora platform to build a medical training application with distributed participants.

We are excited to join with our colleagues to advance the frontiers of XR in health care delivery. This project will create new opportunities that leverage XR technologies to improve remote training of first responders and medical staff in rural and underserved urban areas.

— Mark Cohen, Dean of the College of Medicine and Senior Vice President and Chief Academic Officer for Carle Health

XR-based collaboration can overcome barriers such as high costs, limited availability, and long travel distances that traditionally hinder professional development. Researchers at the College of Medicine’s Jump Simulation Center are integrating a virtual cardiac simulator into a multi‑user remote training environment, aiming to deliver a level of interaction—both between participants and with the simulation—comparable to a physical simulation center.

For both the clinical care and training applications, the goal is to “democratize” access to specialized medical expertise – in the sense of making that expertise more widely available – by bringing experts to wherever they are needed through XR.

But connecting participants in different locations is not the only reason networks are critical for XR. Networks are also important for offloading compute-intensive functions from XR devices.

“XR headsets and glasses today are orders of magnitude away from where they need to be in compute and power usage,” said Adve. “If we can offload compute-intensive parts of the XR software stack to servers, we can make XR devices lighter and lower power while enabling richer experiences for the end user.” Adve’s team is developing these offloading mechanisms, building on their past development of the ILLIXR open-source XR runtime.

Creating a convincing XR experience with distributed participants and distributed compute components is not easy. Immersive XR experiences need to rapidly adapt to changes in the position of the user and the environment as they blend real and virtual content to create an XR experience. Latency introduced into this system, beyond a few tens of milliseconds, can quickly lead to poor user experiences and discomfort.

However, latency can frequently spike into hundreds of milliseconds in deployed wireless networks, which have been mostly optimized for bandwidth rather than latency. Although low latency technologies have been proposed, including as a part of 5G wireless, they were designed primarily for specialized uses with low bandwidth, and are not widely deployed.

“So, to democratize access to medical expertise, we need to democratize low latency,” Godfrey said. “We believe, actually, that low latency network service can be useful not only for XR, but also for most other kinds of interactive applications.”

The key technical challenge will be to deliver low latency service efficiently, so that it’s practical to provide the service to many users and many applications. This means the Agora team has to avoid pulling too much capacity away from the bandwidth-focused wireless service that most people use today.

One technique the team is leveraging is to use multiple types of service – both low latency and high bandwidth – in parallel. Agora will include a new transport protocol to make fine-grained message routing decisions onto these parallel services.

Meanwhile, the project is designing the underlying wireless services needed to support XR applications. This work is led by Peraton Labs, the research and development arm of national security company Peraton and an experienced innovator of new 5G technology.  “By dynamically optimizing radio resource scheduling decisions for concurrent 5G low latency and high bandwidth services, we can make more efficient overall use of the wireless spectrum,” said Zhang, who leads the project’s development of new 5G technology, along with T-Mobile and Cisco. Qualcomm will provide the client (user equipment) side of the overall system design.

From research to the real world

In keeping with TIP’s goal of technology translation, NSF gave the Agora team a difficult charge – to produce not just research innovations but running demos of use cases within two years.

“Having researchers and industry leaders work directly alongside each other makes this project especially exciting,” Godfrey commented. “There are difficult challenges along the way, but if we’re successful, we’ll show how low latency can make a real impact on medical collaboration, with underlying technology that’s viable for broad deployment.”

---

Grainger Engineering Affiliations

Sarita Adve is an Illinois Grainger Engineering professor of computer science and Director of IMMERSE. Sarita Adve is Richard T. Cheng Professor of Computer Science.

Brighten Godfrey is an Illinois Grainger Engineering professor of computer science and is affiliated with electrical and computer engineering and the Coordinated Science Laboratory.