5/10/2023 Aaron Seidlitz, Illinois CS
For years Illinois CS professor Gagandeep Singh has noticed the way researchers have wrestled with difficulties managing accuracy vs. efficiency or accuracy vs provable robustness tradeoffs in Deep Neural Nets. His new proposal, funded by Google, will develop accurate, robust and efficient deep learning models.
Written by Aaron Seidlitz, Illinois CS
As the development of Deep Neural Networks (DNNs) continue to grow in relevance, Illinois Computer Science professor Gagandeep Singh’s research focus – which bridges formal logic, machine learning, and systems research – positions him perfectly to provide the next step toward “Accurate, Efficient, and Provably Robust Deep Learning.”
Researchers in his area of expertise have wrestled with this exact premise for years but never found a solution to it.
The promise of Singh’s work, though, earned him a Google Research Scholar Award for this proposal. As Singh describes it, researchers pairing with industry partners have had a hard enough time balancing two of these three tradeoffs at once; taking the next step represents a whole new line of thinking.
His abstract from the proposal stated, “Training DNNs that achieve all these objectives simultaneously is a hard optimization problem and is beyond the reach of existing methods that only seek to balance accuracy/efficiency or accuracy/robustness tradeoffs.”
The answer he’s on the verge of developing, formed from necessity.
“If you want to build a practical system, especially in safety critical domains, you need all three priorities met at the same time. You want to have accuracy, you want to have guarantees on network behavior, as well as good efficiency. And this is a hard problem, because it represents a three-way tradeoff between aspects that are all in conflict with each other,” Singh said.
“The way that our project will look at this is as an optimization problem, from which we proposed a general framework based on designing novel formulations, concepts, algorithms, and architectures.”
Singh is also especially excited about this particular project being tied to Google.
Calling it an honor to receive the Research Scholar Award, Singh touted Google’s interest in both being an industry leader while genuinely supporting and engaging with academic research. That combination leads to greater potential for his own work, while satisfying a need that collaborators on the industry side have not met yet.
The result, he said, could make a change for the better in many real time systems.
“For example, systems operating on the edge – like robots, drones or autonomous vehicles – don't only need good accuracy; they also need good response times and robustness. Then, if you can build an efficient system, I think we can actually have a transformative impact on real world systems,” Singh said.
A potential result, he said, would build off work that about half of his FOrmally Certified Automation and Learning (FOCAL) Lab is already conducting in terms of verification technology.
The next steps, while unprecedented, also fit well within the lab’s area of expertise.
As DNNs have developed into larger and larger entities, a growing problem has been the energy fueling the systems.
Current “power-hungry” models have a major impact on the carbon footprint. Singh said his students are excited about the possibility of producing smaller models that meet the need for accuracy and robustness, while also adding a brand-new level of energy efficiency.
“The question now is, can we do something better with smaller models? This is where we are trying to go,” Singh said. “I hope that as these students gain knowledge and expertise in this line of thought, they take it to industry partners to help develop a more sustainable pipeline that further develops these machine learning models.”
Considering the potential of this project, Singh believes its impact expands beyond even the designed result.
“I think, up to now, associated research communities have been working somewhat separately on this issue. People first observed that there is a tradeoff between accuracy of the machine learning model, and its efficiency. Then people noticed the tradeoff between accuracy and robustness,” Singh said. “I see this project’s value as a new kind of research direction, through which these separate lines of thought come together to form the accuracy, robustness and efficiency guarantees long sought after.”