5/16/2024 Jenny Applequist
Written by Jenny Applequist
Counterfactual scenarios will be simulated to predict the outcomes of different courses of action.
Computer Science and Coordinated Science Lab professor Shenlong Wang at The Grainger College of Engineering at the University of Illinois received a National Science Foundation (NSF) CAREER award to support the creation of AI systems that can make digital twins—that is, digital replicas—of the physical world that are capable of simulating counterfactual “what-if” scenarios, enabling users to assess the potential outcomes of actions if they are carried out in the real world.
To do so, he will need to create digital replicas with a greater “understanding” of the world than current systems, making it possible for them to “imagine” unseen scenarios rather than just represent things they’ve already observed.
Wang explained that the developed tools will be able to produce high-quality imagery with applications in the entertainment industry and in virtual and augmented reality—but, more importantly, that they could also have a “very helpful and very profound impact in the real world.”
He said the “twin worlds” will act like the real world, providing realistic observations. “The agent can then use these realistic observations to take multiple different actions and then return it to the twin world, and then the twin world can generate multiple expected outcomes. Then, we can decide which outcome we want. And then we can transfer this insight to help the agent make the right decision in the real world.”
There are multiple reasons why such a capability would be highly desirable. It would allow users to try out dangerous actions—say, new surgical techniques—in a risk-free way. It would make it possible to see what happens in scenarios for which little data are available, such as situations that are rare in real life—for example, what happens to a city’s water distribution infrastructure during a 100-year flood—without having to wait for such scenarios to happen in reality. Further, because virtual time can run much faster than real-time, models can be run forward to predict the long-term consequences of actions, even decades into the future.
“If computers can help us to make more informed predictions, we essentially have a time machine,” said Wang. “We might be able to see what’s going on in the next century!”
This diagram shows how hypothetical scenarios can be run through the world’s digital twin (right side), allowing users to “rehearse” multiple proposed solutions in the virtual world before choosing one to apply in the real world (left side).
Wang will consider two use cases in the project. One of them is autonomous driving, which is familiar territory for him. Before joining Illinois' faculty, he worked as a research scientist at Uber, building simulators to test self-driving vehicles’ safety. His second use case will be climate risk assessment for agriculture. For that, the shorter-term goal is to determine things like carbon emission levels based on ordinary mobile phone photographs of a soybean field; in the longer-term, the more ambitious goal is to gain such insights from satellite imagery.
Wang said that he’s particularly happy about the planned education and outreach components of his project. He anticipates that the strong visual appeal of the work and the excitement of being able to create one’s own virtual world will be attractive to young people. While studying computer vision and machine learning normally starts with a lot of intimidating math, he plans to offer an immersive initial experience to make this area more approachable to K through 12 schoolchildren.
Wang's Faculty Early Career Development (CAREER) Award to Digitize and Simulate the Large Physical World via Knowledge-Grounded Scene Representation is the NSF's most prestigious award in support of early-career faculty who have the potential to serve as academic role models in research and education and to lead advances in the mission of their department or organization.